Inference Engineering — Chapter 02: Inference (book Chapter 0)

1. Pre-Reading Briefing

Core Question: Why is serving generative AI models so much harder than serving classic ML, and what layers does a complete inference system need?
Key Points to Watch For:
- Two phases, two eras: how the author splits a model's lifecycle, and why he says inference for classic ML (e.g., XGBoost on CPUs) was easy while generative AI is not.
- Three layers: names the three layers of a complete inference system and the single job of each. Notice the claim that one layer alone is “not enough,” and why.
- The runtime toolkit: a list of six model-performance techniques, each defined in one line. Don't master them yet (Chapter 7 / book Ch 5 covers them). For each, note just what it trades or reuses. Two terms will be new: KV cache and prefill vs. decode.
- Scale changes the problem: infrastructure problems shift as you grow. Note the stages and the rough GPU count where the shift happens.
- Developer-experience spectrum: two extremes and where the author says the right answer sits. Ask how this connects to the Ch1 thread “when is self-hosting wrong?”
Context & Thread from Prior Chapters: The Preface claimed inference engineering makes models faster, cheaper and more reliable. This chapter is the first look at how. Map each layer to the Preface's three dimensions (latency, availability, cost): which layer serves which? Also note the book's stated knowledge cutoff (Jan 2026) and what the author says will last despite it.