Inference Engineering — Chapter 01 Memory: Preface

Stage: Complete
Next Step: Preview Chapter 2 (book's Chapter 0: Inference).
Reading Span: PDF pages 11–16
Source Text: /library/Inference Engineering/Chapter-01-Preface/Chapter-01-source-text.md (the chapter's own words; read instead of the PDF)
Full Record: /library/Inference Engineering/Chapter-01-Preface/Chapter-01-chapter-notes.md (read only if needed)
Last Updated: 2026-10-06

Carried-in Context (from earlier chapters)

First chapter — nothing carried in. Started 2026-10-06 from Chapter 1 (Preface); no prior position logged.

Core Question: Why has inference engineering gone from a niche skill at a few labs to something every serious AI product company needs, and what does the job actually involve?
Watch-For Themes: Definition and scope of inference engineering (“CUDA to Kubernetes”); the “alternate future” the author rejects and what changed it; closed vs open models (definition by weights, licensing caveat); the three dimensions inference engineering improves; author's vantage point (Baseten) and how that affects which claims need checking.
Core Thesis: An explosion of capable open models turned inference engineering (faster/cheaper/more reliable serving, CUDA→Kubernetes) from a frontier-lab niche into a strategic need for any differentiated AI product.
Key Concepts: Inference engineering = serving optimization (latency, cost, reliability); closed vs open = weight availability (+license caveat); rejected “rent intelligence” future; capability thresholds (prototype closed → switch open/fine-tuned); three dimensions latency/availability/cost (+ fine-tuning for domain quality).
Notable Arguments / Evidence Limits: HF counts, DeepSeek V3/R1 closing the gap Dec 2024, Baseten customer list. Uptime (2 vs 4 nines) and ≥80% cost claims have no method/baseline; vendor vantage; no “when not to self-host.”
Action Item: Baseline one AI tool/API you use (latency, cost/request, outages) and find one licensed open-model alternative on Hugging Face.

Pending Questions: None. “When is self-hosting wrong?” moved to Open Threads.
Reader's Answers (paraphrase): (1) Serving models in production; making them easier/more useful. (2) Closed = weights private, open = public; open can be fine-tuned on your data. (3) Wants to examine “open models are better.”
Misconceptions / Feedback Given: Q1 goal too vague (it's faster/cheaper/more reliable), missed CUDA→K8s range and the open-model explosion shift. Q2 definition right; fine-tuning = capability, not one of latency/availability/cost. Q3 too broad; sharpened into testable claims (two vs four nines, 80% at scale, Dec 2024 gap).
Personal Threads: Skeptical of “open models are better”. Track evidence for latency/availability/cost claims through Ch 5–9.

When is self-hosting the wrong choice (low traffic, small team, modest uptime needs)? Book hasn't addressed it yet.
Vendor perspective: author leads at an inference provider; check concrete numbers (uptime, cost savings) against independent sources later in the book.