Inference Engineering — Chapter 02 Memory: Inference (book Chapter 0)

Stage: Previewed — awaiting reading
Next Step: Wait for the reader to say “done” / /review, then ask 2–3 review questions on the watch-for themes.
Reading Span: PDF pages 17–24 (17–18 are the chapter opener; text starts p. 19)
Source Text: /library/Inference Engineering/Chapter-02-Inference/Chapter-02-source-text.md (the chapter's own words; read instead of the PDF)
Full Record: /library/Inference Engineering/Chapter-02-Inference/Chapter-02-chapter-notes.md (read only if needed)
Last Updated: 2026-10-06

Carried-in Context (from earlier chapters)

Ch1 thesis: open-model explosion (2M+ on HF) turned inference engineering (faster/cheaper/more reliable serving, CUDA→Kubernetes) from frontier-lab niche into strategic need.
Closed vs open = weight availability; open weights can be downloaded and self-run; check license (some restrict commercial use).
Three dimensions open + inference eng. unlock: latency, availability (claimed 4 nines vs 2), cost (claimed ≥80% at scale); fine-tuning adds domain quality.
Capability thresholds: prototype on closed API, switch to open/fine-tuned as they catch up.
Reader: solid on “serving in production” and weights definition; earlier conflated fine-tuning (quality) with the three serving dimensions. Skeptical of “open models are better”; wants claims tested.
Open threads: when is self-hosting the wrong choice? Vendor (Baseten) vantage: uptime/cost numbers lack method/baseline.
Weekly action from Ch1: baseline one AI tool (latency, cost/request, outages) + find a licensed open alternative.
Cross-book: Context Engineering / Spec Driven Development treat the model as an API; this book is the layer below.

Core Question: Why is serving generative AI models so much harder than serving classic ML, and what layers does a complete inference system need?
Watch-For Themes: Training vs inference; classic ML vs generative inference; the three layers (runtime, infrastructure, tooling); six runtime techniques; how infra problems change with scale; the developer-experience spectrum; the book's roadmap + knowledge cutoff.
Core Thesis: Pending
Key Concepts: Pending
Notable Arguments / Evidence Limits: Pending
Action Item: Pending

When is self-hosting the wrong choice? Watch whether the tooling-layer “black box ↔ raw compute” spectrum speaks to it.
Vendor vantage: track evidence for latency/availability/cost claims through Ch 5–9.