# Inference Engineering — Chapter 02 Memory: Inference (book Chapter 0)
- **Stage**: Previewed — awaiting reading
- **Next Step**: Wait for the reader to say "done" / `/review`, then ask 2–3 review questions on the watch-for themes.
- **Reading Span**: PDF pages 17–24 (17–18 are the chapter opener; text starts p. 19)
- **Source Text**: /library/Inference Engineering/Chapter-02-Inference/Chapter-02-source-text.md (the chapter's own words; read instead of the PDF)
- **Full Record**: /library/Inference Engineering/Chapter-02-Inference/Chapter-02-chapter-notes.md (read only if needed)
- **Last Updated**: 2026-10-06

## Carried-in Context (from earlier chapters)
- Ch1 thesis: open-model explosion (2M+ on HF) turned inference engineering (faster/cheaper/more reliable serving, CUDA→Kubernetes) from frontier-lab niche into strategic need.
- Closed vs open = weight availability; open weights can be downloaded and self-run; check license (some restrict commercial use).
- Three dimensions open + inference eng. unlock: latency, availability (claimed 4 nines vs 2), cost (claimed ≥80% at scale); fine-tuning adds domain quality.
- Capability thresholds: prototype on closed API, switch to open/fine-tuned as they catch up.
- Reader: solid on "serving in production" and weights definition; earlier conflated fine-tuning (quality) with the three serving dimensions. Skeptical of "open models are better"; wants claims tested.
- Open threads: when is self-hosting the wrong choice? Vendor (Baseten) vantage: uptime/cost numbers lack method/baseline.
- Weekly action from Ch1: baseline one AI tool (latency, cost/request, outages) + find a licensed open alternative.
- Cross-book: Context Engineering / Spec Driven Development treat the model as an API; this book is the layer below.

## This Chapter
- **Core Question**: Why is serving generative AI models so much harder than serving classic ML, and what layers does a complete inference system need?
- **Watch-For Themes**: Training vs inference; classic ML vs generative inference; the three layers (runtime, infrastructure, tooling); six runtime techniques; how infra problems change with scale; the developer-experience spectrum; the book's roadmap + knowledge cutoff.
- **Core Thesis**: Pending
- **Key Concepts**: Pending
- **Notable Arguments / Evidence Limits**: Pending
- **Action Item**: Pending

## Reader State
- **Pending Questions**: None yet.
- **Reader's Answers (paraphrase)**: None yet.
- **Misconceptions / Feedback Given**: None yet.
- **Personal Threads**: Ch1 baseline action item (not yet reported).

## Open Threads
- When is self-hosting the wrong choice? Watch whether the tooling-layer "black box ↔ raw compute" spectrum speaks to it.
- Vendor vantage: track evidence for latency/availability/cost claims through Ch 5–9.