diff --git a/MEMORY.md b/MEMORY.md index fc37587..99ca71c 100644 --- a/MEMORY.md +++ b/MEMORY.md @@ -44,3 +44,12 @@ Status board only. Details live in each chapter's `memory.md`; read the one list - **Source File**: /library/FOCUS Architecture/source-file.pdf - **Structure**: /library/FOCUS Architecture/book-structure.md (24 numbered chapters plus an interlude after Chapter 1) - **Last Updated**: 2026-10-01 + +## Inference Engineering — Philip Kiely +- **Status**: In Progress +- **Current Position**: Intake complete — starting chapter to be confirmed (Chapter 1 of 11 = Preface) +- **Current Chapter Memory**: Not created yet +- **Folder**: /library/Inference Engineering/ +- **Source File**: /library/Inference Engineering/source-file.pdf +- **Structure**: /library/Inference Engineering/book-structure.md (11 sections; reading number = book chapter + 2; Appendices A/B = Chapters 10/11) +- **Last Updated**: 2026-10-05 diff --git a/library/Inference Engineering/Back-Matter/Back-Matter-source-text.md b/library/Inference Engineering/Back-Matter/Back-Matter-source-text.md new file mode 100644 index 0000000..4e1c9e1 --- /dev/null +++ b/library/Inference Engineering/Back-Matter/Back-Matter-source-text.md @@ -0,0 +1,33 @@ +# Inference Engineering — Back-Matter: Acknowledgements +- **Source**: /library/Inference Engineering/source-file.pdf +- **PDF pages**: 259–259 +- **Pages without text**: none + +--- + + +Acknowledgements +This book was made possible by people who invested in me early, without +assurance of success. My thanks to Mike Bilodeau and Dannie Herzberg, +who greenlit this project when I was supposed to be working on a dozen +other priorities. More thanks to Tuhin Srivastava, Amir Haghighat, Phil +Howes, Pankaj Gupta, and Emmiliese von Avis, who hired me four years +ago, even though I had no formal experience in the role. To Megha, for +everything. And to my family, who has bet on me for my entire life. +Thank you to the many reviewers who helped me make this book as +accurate and precise as possible: Raymond Cano, Tianshu Cheng, Tri +Dao, Michael Feil, William Gao, Pankaj Gupta, Amir Haghighat, Mahmoud +Hassan, Nidhi Hiremath, Phil Howes, Tyron Jung, Kaz Kato, Madison +Kanna, Alex Ker, Harry Kim, Colin McGrath, Deepak Nagaraj, Eskil Olsen, +Ed Shrager, Aaryam Sharma, and Joey Zwicker. Any errors that remain +are my sole responsibility. +Thank you to Robin Bourjaily for editing this book, along with just about +everything else I’ve ever written. +Thank you to Luke de Hass for the cover and internal illustrations, to Aaron +Relph for design guidance, to Javier Crocco for internal illustrations, and +to Raúl López for additional design assistance. +Thank you to the many researchers behind the open models, contributors +to the open-source libraries, and authors of the papers mentioned in this +book. You are the giants upon whose shoulders we stand. +And finally, my thanks to you, the reader, without whom I would merely +be creating very elaborate training data. diff --git a/library/Inference Engineering/Chapter-01-Preface/Chapter-01-source-text.md b/library/Inference Engineering/Chapter-01-Preface/Chapter-01-source-text.md new file mode 100644 index 0000000..6052961 --- /dev/null +++ b/library/Inference Engineering/Chapter-01-Preface/Chapter-01-source-text.md @@ -0,0 +1,137 @@ +# Inference Engineering — Chapter-01: Preface +- **Source**: /library/Inference Engineering/source-file.pdf +- **PDF pages**: 11–16 +- **Pages without text**: none + +--- + + +Preface +Inference is the most valuable category in the AI industry. +Inference engineering, on the other hand, is still in its infancy. Inference +engineers work across the stack from CUDA to Kubernetes in pursuit of +faster, less expensive, more reliable serving of generative AI models in +production. +On November 30, 2022 – the day that ChatGPT was launched – there +were perhaps a few hundred inference engineers in the world, though they +didn’t call themselves that at the time. These specialists mostly worked at +frontier labs like OpenAI, Midjourney, and Anthropic or big tech companies +like Google and NVIDIA. +Back then, it looked like this might be the way of the AI industry. Per- +haps training generative AI models would be so hard, and so expensive, +that only a handful of companies would develop closed models and thus +require inference engineering for production serving. In this alternate +future, the rest of the world would be mere consumers of AI via APIs, +renting intelligence a token at a time. +Three years later, it turns out that training generative AI models is hard. +It is expensive. But it is neither so hard nor so expensive that it is limited +to that handful of players. +Instead, a Cambrian explosion of open models – more than two million +and counting on Hugging Face – means that every engineer can now +deploy their own intelligence to power their AI products. Research labs +around the world, from OpenAI and NVIDIA Nemotron in America to +Mistral AI and Black Forest Labs in Europe to Alibaba Qwen, DeepSeek +AI, Z AI, and Moonshot AI in China, regularly release open models of +all modalities. + + +10 Preface +Figure P.1: There are well over two million open models on Hugging Face, 25 times +more models than fi ve years ago. +Despite closed models getting smarter and cheaper, the movement into +open models is accelerating. Open models differ in the availability of their +weights: +• Closed model: A proprietary model where weights are unavailable to +the public, like GPT-5 or Claude Sonnet. +• Open model: A model where weights are publicly available, like Llama +or DeepSeek, usually released under the MIT license or a similar per- +missive license (though some models restrict commercial use, always +double-check license terms). +Until December 2024, there was a meaningful gap in intelligence between +closed and open models. When DeepSeek V3 and R1 were released, that +gap disappeared. + + +Preface 11 +Today, new closed models are matched by open models within months +if not weeks, with occasional open models like Kimi K2 Thinking even +exceeding closed model capabilities for brief windows. +Even if open models are constantly chasing closed models on bench- +marks, they still change the equation for AI product builders. As both types +of models get better, closed and open models cross capability thresholds +and power new classes of products. +Figure P.2: Open and closed models are both improving rapidly, unlocking and +expanding access to new capabilities. +In 2022, it was impossible to build the kinds of AI-native products that +defi ne the industry today. +Over time, closed models got smarter and new categories like customer +service voice agents and AI-powered IDEs became possible. These early +models were slow, expensive, and unreliable, but the capabilities were +there and AI engineers began building companies around these capa- +bilities. +As open models crossed the same capability thresholds, these prod- +uct builders began using them to replace closed models. Many also + + +12 Preface +began fine-tuning open models to cross capability thresholds faster +and even exceed closed model quality for their specific product and +domain. +Figure P.3: Customizing open models unlocks new capabilities while retaining control +over latency, reliability, and economics. +Switching to open models unlocks the opportunity to use inference engi- +neering to make the models powering AI products better in new dimen- +sions: +• Latency: Closed model APIs are built for throughput, but open models +can be optimized for real-time applications. +• Availability: While APIs for GPT and Claude are stuck at two nines +of uptime, it’s possible to achieve four nines or better with dedicated +deployments of open models. +• Cost: Open models are often at least 80 percent less expensive at +scale. +So where three years ago it looked like inference engineering might be a +niche fi eld, today every company aiming to build truly differentiated and +competitive AI products needs an inference strategy. + + +Preface  13 +AI-native startups like Cursor, Clay, Gamma, and Mercor are redefining +hypergrowth building products that rely on open and in-house models. +Leading digital native companies like Notion and Superhuman are thriving +by deeply integrating AI capabilities into products that hundreds of millions +already love. +And a new generation of blended research and engineering teams – World +Labs, Writer, Mirage, and dozens of others – are building enormous busi- +nesses by training and productizing their own foundation models. +Adoption is even strong in enterprise and regulated industries, which +historically have been slow to adapt to new technologies. Companies +like OpenEvidence, Abridge, and Ambience are making generative +AI ubiquitous in healthcare, while at the world’s largest companies, +AI initiatives are moving past the pilot stage into massive user adop- +tion. +I’ve been incredibly fortunate to have a front-row seat to the fastest-mov- +ing market in history over the last four years at Baseten, where we power +mission-critical inference for the best AI products, including every company +listed in the previous paragraphs. +The incredible market-wide demand for inference means that everyone +from developers to executives has the opportunity to learn inference engi- +neering and use it to advance their career and business. +You are early. While the potential and impact of inference are becoming +clear, the space is young. There are relatively few people working on infer- +ence, and newcomers can become experts quickly. There are enormous +opportunities to solve novel, interesting, and deeply technical problems +at all levels of the stack. +Inference Engineering is your guide to becoming an expert in infer- +ence. It contains everything that I’ve learned in four years of working at +Baseten. This book is based on interviews with dozens of experts from +our engineering team; technical talks I’ve delivered at conferences like +NVIDIA GTC, PyTorch Conference, AWS re:Invent, and AI Engineer +World’s Fair; and countless conversations with customers and builders +around the world. + + +14  Preface +Thank you for reading Inference Engineering and welcome to the early +days of inference. +Philip Kiely +San Francisco, CA diff --git a/library/Inference Engineering/Chapter-02-Inference/Chapter-02-source-text.md b/library/Inference Engineering/Chapter-02-Inference/Chapter-02-source-text.md new file mode 100644 index 0000000..8fe34b3 --- /dev/null +++ b/library/Inference Engineering/Chapter-02-Inference/Chapter-02-source-text.md @@ -0,0 +1,158 @@ +# Inference Engineering — Chapter-02: Inference +- **Source**: /library/Inference Engineering/source-file.pdf +- **PDF pages**: 17–24 +- **Pages without text**: 17, 18 + +--- + + +CHAPTER 0 +Inference + + + + + +Inference  17 + +Inference +Inference is the second phase in a generative AI model’s lifecycle: +• Training: The process of learning model weights from data. +• Inference: Serving generative AI models in production. +In last decade’s machine learning (ML) boom, hundreds of thousands of +data scientists and ML engineers became familiar with the full lifecycle, +both training and inference, for ML models. +Inference for classic ML models is relatively straightforward. In the early +days of Baseten, we ran inference for models built with tools like XGBoost +on lightweight CPUs with a simple software stack. +In contrast, inference for generative AI models is complex. You can’t sim- +ply take model weights, get some GPUs, and expect inference to be fast +and reliable enough for large-scale production use. Doing inference well +requires three layers: +• Runtime: Optimizing the performance of a single model on a single +GPU-backed instance. +• Infrastructure: Scaling across clusters, regions, and clouds without +creating silos while maintaining excellent uptime. +• Tooling: Providing engineers working on inference with the right level +of abstraction to balance control with productivity. +These three layers must work together to create a system that can handle +mission-critical inference at scale. + + +18 Chapter 0: Inference +Figure 0.1: A complete inference stack includes both runtime and infrastructure +optimizations. +The runtime layer is responsible for ensuring that an individual model +running on a GPU (or across several GPUs in a single instance) is running +as performantly and efficiently as possible. +This layer depends on a sophisticated software stack, from CUDA to +PyTorch to inference engines like vLLM, SGLang, and TensorRT-LLM. +Low-level optimization is important, with kernels like FlashAttention deliv- +ering signifi cant performance gains. +The runtime layer relies on a number of model performance techniques +that apply new research to the unique challenges of inference on gener- +ative AI models: + + +Inference  19 +• Batching: Run incoming requests in parallel, weaving them together +on a token-by-token basis to increase throughput. +• Caching: Re-use the KV cache – the cached results of the attention +algorithm – between requests that share prefixes. +• Quantization: Lower the precision of select pieces of the model to +access more compute and reduce memory burden. +• Speculation: Generate and validate draft tokens to produce more than +one token per forward pass during decode. +• Parallelism: Efficiently leverage more than one GPU to accelerate large +models without introducing new bottlenecks. +• Disaggregation: Separate the two phases of LLM inference, prefill and +decode, onto independently scaling workers. +These model performance techniques are used for models of all modal- +ities, not just LLMs. Modalities like vision language models, embedding +models, automatic speech recognition, speech synthesis, image gener- +ation, and video generation extend the capabilities of AI systems and +require their own inference optimizations. +These runtime optimizations are not enough. No matter how performant a +single instance of a model server is, it will eventually receive more traffic +than it can handle. +This is not a CUDA problem or a PyTorch problem. This is a systems +problem that needs to be solved at the infrastructure layer. +The nature of infrastructure problems changes with each level of scale. +At first, the problems are around autoscaling: knowing when to add and +remove replicas and figuring out how to do so quickly. +Past a certain scale, generally a few hundred GPUs, infrastructure prob- +lems are defined by capacity. To get access to enough GPUs, inference +engineers begin spreading workloads across multiple regions and multiple +cloud providers. +This quickly leads to silos, where models in one cluster may be starved +for resources while other clusters have unused capacity. The final level of +scale in infrastructure is a global system that treats all available resources +as a single unified pool of compute. + + +20 Chapter 0: Inference +Figure 0.2: Runtime performance optimizations like speculation, diagrammed here, +improve inference latency. +Thoughtful multi-cloud infrastructure also improves reliability, protect- +ing against downtime in any individual region or cloud provider. And for +global applications, running inference near end users improves end-to- +end latency. + + +Inference 21 +Figure 0.3: Serving model inference at scale requires unifying capacity across +multiple cloud service providers. +Once these runtime and infrastructure capabilities are built, they need to +be presented at the appropriate level of abstraction. Both inference pro- +viders like Baseten and internal teams building inference need to consider +what tooling and developer experience to provide as the critical third layer +in a complete inference platform. +Developer experience is subjective. For inference, one extreme is the +black box: give a platform model weights, get back an API. At the other +extreme is providing only basic constructs for compute, network, disk, +and so forth. +The right developer experience is somewhere in the middle, where infer- +ence engineers have enough control to run mission-critical inference con- +fi dently, but enough abstraction to work productively. +Inference Engineering presents a map of the technologies and techniques +that power inference across all three layers of runtime, infrastructure, and +tooling. +Chapter 1, Prerequisites, covers the product thinking and AI engineering +work that need to be done before inference engineering comes into play: +use case defi nition, latency and cost budgeting, and selecting and evalu- +ating which generative AI models to optimize and deploy. + + +22  Chapter 0: Inference +Chapter 2, Models, introduces the technical architecture of AI models – +from large language models to image and video generation models – and +establishes where the bottlenecks exist for inference with a special focus +on optimizing attention. +Chapter 3, Hardware, starts at the spec sheet for modern GPUs and +breaks down compute and memory, then disambiguates architectures and +SKUs within NVIDIA’s datacenter-grade offerings before briefly surveying +other accelerators on the market. +Chapter 4, Software, builds abstractions from CUDA to frameworks like +PyTorch, Transformers, and Diffusers and inference engines like vLLM, +SGLang, and TensorRT-LLM. It also introduces Dynamo, NVIDIA’s latest +system for large-scale distributed model serving. +Chapter 5, Techniques, discusses key model performance optimization +techniques adapted from cutting-edge research and applies them in pro- +duction: quantization, speculative decoding, KV cache re-use, model par- +allelism, and disaggregation. +Chapter 6, Modalities, expands inference engineering beyond LLMs to +voice and visuals. Many types of generative AI models – vision-language +models, embedding models, automatic speech recognition (ASR) models, +and speech synthesis models – adapt LLM architectures, meaning infer- +ence engineers can run them with the same tools and techniques used +with LLMs. Image and video generation models have their own architec- +tures and associated performance optimization techniques. +Chapter 7, Production, concludes the book with a rundown of the important +problems to solve in operating infrastructure for and building performant +applications on optimized model inference services. +Appendices A and B add a glossary of inference engineering terms and +a collection of recommended resources for further reading, respectively. +Like LLMs, books have knowledge cutoffs. This book was finished in Jan- +uary 2026. While details will change, the principles, concepts, and founda- +tional technologies in this book provide a strong background on inference +engineering that will serve you well for years to come. diff --git a/library/Inference Engineering/Chapter-03-Prerequisites/Chapter-03-source-text.md b/library/Inference Engineering/Chapter-03-Prerequisites/Chapter-03-source-text.md new file mode 100644 index 0000000..288e0ea --- /dev/null +++ b/library/Inference Engineering/Chapter-03-Prerequisites/Chapter-03-source-text.md @@ -0,0 +1,443 @@ +# Inference Engineering — Chapter-03: Prerequisites +- **Source**: /library/Inference Engineering/source-file.pdf +- **PDF pages**: 25–40 +- **Pages without text**: 25, 26, 40 + +--- + + +CHAPTER 1 +Prerequisites + + + + + +Prerequisites 25 + +Prerequisites +Inference engineering adds speed and scale to AI products by optimizing +production serving of generative models. +Optimization means identifying the best solution out of a range of options. +Before optimizing model performance and building robust infrastructure, +you need to know what “best” means for your product – many performance +improvements come from making tradeoffs among latency, throughput, +and quality. +In practice, optimization is often about fi nding the right balance rather than +maximizing a single factor. +NFL players are big, fast, and strong. But they’re not as big as sumo wres- +tlers, as fast as Olympic sprinters, or as strong as champion powerlifters. +Their bodies and skills are optimized to fulfi ll the specifi c demands of their +position over the course of a full season. +Figure 1.1: Just like elite athletes, inference services must be specialized for the +demands of their workloads. +Similarly, your inference system must be optimized to fulfi ll the specifi c +demands of your model, your product, and your traffic. The more con- +straints you can introduce, the better outcomes you can achieve. + + +26  Chapter 1: Prerequisites +You should know your: +• Model requirements: Which model(s) do you need to run inference +on? +• Application interface: How will inputs be delivered to the model, and +how is the output expected to be formatted? +• Latency budget: End-to-end, how fast does your product need to +respond to a user action? +• Unit economics: What does it make sense to spend on a per-request, +per-user, or per-month basis? +• Usage patterns: How many concurrent users are you serving, and is +there any pattern to their usage (e.g., more activity during business +hours)? +Early on in building an AI product, the answers to these questions may +not be clear. At this early stage, it’s often better to use off-the-shelf APIs +whenever possible rather than investing in dedicated inference. As the +product scales, requirements become clear and inference engineering +becomes a worthwhile pursuit. +1.1  +Scale and Specialization +There are two ways that you can add AI models to your product: +• Shared inference: Send your traffic to a public API endpoint for a given +model and pay per million tokens or some other consumption-based +metric. +• Dedicated deployments: Rent GPUs and set up an inference service +exclusively for your application, paying per hour of GPU time (or pur- +chase and install GPUs on-premises). +Shared versus dedicated inference is not exactly the same conversation +as closed versus open models – there are plenty of shared endpoints +for open models and many model labs offer large customers some kind +of dedicated setup for their closed models. However, one of the key +motivations for adopting open models is that it unlocks unrestricted ded- +icated inference. + + +1.2  About Your App  27 +Most AI products start with pay-per-token APIs because the tradeoffs +make sense while looking for product-market fit or in the early stages of +growth. +Pros of shared inference +Cons of shared inference +Zero overhead, only pay for +consumption +Cost scales linearly with usage +No cold start times, model is always +available +Provider uptime caps product uptime, +noisy neighbors +Minimal engineering work, just need +an API key +No control over latency, model +quality, or rate limits +Over time, shifting AI products to dedicated deployments is essential for +three reasons: +• Scale: You are processing enough volume of traffic that it’s more eco- +nomical to pay per GPU than per million tokens. +• Specialization: You are running a custom or fine-tuned model, or you +have specific latency or uptime requirements. +• Orchestration: Your product relies on multiple models and multi-step +pipelines and you need to minimize network latency and deployment +complexity. +The switch to dedicated deployments puts you in charge of your own +inference engineering. This gives you flexibility and control but adds to +your engineering surface area and increases the floor of your monthly +spend on inference. Only switch once there is a clear and immediate +business need. +1.2  +About Your App +Every inference engineering decision you make will be downstream of +your use case. +Imagine a sports coach recruiting at a school. Which students would they +want to talk to? It would depend entirely on which sport they’re coaching +– the basketball coach would want the tallest kids in the class, but the +gymnastics coach would select among the shortest. + + +28  Chapter 1: Prerequisites +Similarly, the way your inference system will be used determines how you +go about building it. +There are two cases where inference engineers need to build at the high- +est level of generality: +• Foundation models: You have trained your own model and will sell +consumption directly via a public shared inference API and need to +support many usage patterns. +• Inference platforms: You are building an inference platform, either +internally or as a product, and need to support any model and any use +case. +But most AI-native applications are vertical apps like code editors or +customer service agents, where AI is used to create some novel user +experience. When building inference for vertical AI-native apps, you +want to add as many constraints as possible by getting specific with +your use case. +1.2.1  +AI-Native Applications +Generative AI models have unlocked a new class of applications across +industries and domains. Each category of application relies on different +models and modalities and requires tailored inference. +Category +Example +Considerations +Agents +Prospecting agent for sales +teams +One user action triggers many +inference calls +Chat +Front-line customer support +chat with RAG +Time to first token makes chat +feel fast +Voice +Real-time translation between +languages +End-to-end latency for natural +conversation +Media +Virtual try-on for clothes, +shoes, and jewelry +Balance output quality vs. +speed +Search +Legal document discovery +Offline corpus filling vs. online +user requests + + +1.2.2  Online versus Offline  29 +Category +Example +Considerations +RecSys +E-commerce product +recommendations +Consistent latency with high +request volume +Completion +Tab completion for coding in +an IDE +Full completion chunk at +user’s typing speed +Moderation +Scan user-generated content +for safety +High throughput for cost- +effective checks +This is a tiny sample of the AI-native applications that are being built +today, but it shows the breadth of considerations that inference engineers +encounter. And as models get faster, cheaper, and smarter, new use +cases that haven’t even been imagined today will emerge. +1.2.2  +Online versus Offline +One of the primary tradeoffs in inference engineering is latency versus +throughput. Lower latency makes your application faster, but higher +throughput makes it cheaper at scale because you can use fewer GPUs +for the same number of users. +Most AI applications – code completion, chat, voice agents – are online +applications that run in real time. With an impatient user waiting on the +other end of every API call, these online applications should be optimized +for latency. +However, some applications have offline batch inference needs. Offline +jobs are better served by high-throughput model deployments where each +individual request would be too slow for a good user experience, but the +system as a whole processes far more requests per hour in parallel. +Some example offline workloads include: +• Catalog transcription: Transcribing a back catalog of podcasts, inter- +views, or other audio to make it searchable. +• Document processing: Embedding, converting, or analyzing a set of +documents on a regular cadence. +• Corpus preparation: Cleaning, embedding, or otherwise preparing +massive corpora of data for model training. + + +30  Chapter 1: Prerequisites +It’s possible to have a single model that is used for both online and offline +jobs. Whisper, a speech-to-text model, could be used in both a real-time +dictation app and a batch transcription job. Assuming both use cases +have enough volume, it will be more cost effective to create two separate +deployments for the same model, with one optimized for latency and the +other optimized for throughput. +1.2.3  +Consumer versus B2B +Applications built for consumers and businesses have different inference +needs. +Consumer applications are generally much more cost-sensitive and have +less predictable usage patterns. Many consumer AI apps are designed +for virality, and a single launch or marketing campaign can drive a spike +in usage overnight. +Inference engineers working on consumer apps should prioritize mar- +ginal cost and flexibility while keeping latency and availability at a decent +standard. +Business-to-business products often have better margins and more sta- +ble usage but require high availability and consistently low latency. Mis- +sion-critical software in the revenue path is held to a high standard of +performance and reliability. +Inference engineers building for businesses must favor latency and uptime, +though cost and scale are important secondary concerns. +In both consumer and business applications, compliance can limit infra- +structure options, especially in regulated industries. Some essential con- +siderations include: +• Data sovereignty: Are your GPUs in a geographic region where you +are allowed to send user data? +• User privacy: Are the inputs and outputs of your model kept private +and secure? +• Regulatory compliance: Are you and your underlying providers com- +pliant with all relevant regulations? + + +1.3  Model Selection  31 +Inference engineers must work closely with security and legal experts to +ensure that the infrastructure they are operating is compliant. +1.3  +Model Selection +All else being equal – hardware, runtime, optimizations, architecture – +inference on a smaller model with fewer parameters will be faster and +cheaper than inference on a larger model with more parameters. +That’s why the most important decision in model performance optimization +isn’t the runtime engine or speculation algorithm, it’s which model you +choose to work with in the first place. +AI engineers iterating on early-stage products should just use prebuilt +pay-per-token APIs for powerful frontier models like Kimi and DeepSeek +(or even closed models like GPT and Gemini). Before product-market fit, +it’s not worth spending time or money on doing your own inference. +But when it’s time to scale, the opposite advice applies. Find – or create – +the smallest, easiest-to-run model that’s smart enough to handle the task +at hand. In many cases, this is still going to need to be a trillion-parameter +frontier model. But it’s always worth checking if a smaller, cheaper, faster +model can get the job done. +Which model you pick also affects what inference optimizations are avail- +able to you. Inference engines vary in the depth of support for different +model architectures. Stick with popular model architectures to ensure you’ll +find robust support across the performance tooling landscape. +1.3.1  Model Evaluation +Model evaluation, or evals, is the practice of systematically measuring +model intelligence. +High conviction model evaluation is a prerequisite for inference engineer- +ing. Evals help inference engineers: +• Spend time wisely: Before investing in making a model fast, evals +ensure the model is useful. + + +32  Chapter 1: Prerequisites +• Establish a baseline: Some performance optimization techniques risk +reducing model quality, requiring a baseline to compare against. +Unlike standard intelligence benchmarks, which measure model capabil- +ities against common tasks like MMLU or SWE-bench, evals are tailored +to specific products, domains, and tasks. +Intelligence benchmarks are useful for shortlisting models, but they have +become saturated or even gamed. Goodhart’s Law states that “when a +measure becomes a target, it ceases to be a good measure," and this +applies to frontier labs’ heavy incentive to show new world-record intelli- +gence benchmarks with each model release. +While there are better ways to gauge overall model intelligence, like Elo +rating on head-to-head win rate versus other models, there is no substitute +for directly measuring how a model performs for your application. +A few tips for doing useful model evaluation work: +• Look at your data: Check eval results against your intuition for the +product and problem space. +• Be precise: Have a clear idea of the hardest problems a model needs +to solve and focus evaluation there. +• Use tools: Don’t reinvent the wheel on one of the fundamental problems +in AI engineering. +Appendix B includes recommendations for tooling and further reading +about evals. +1.3.2  +Fine-Tuning for Domain-Specific Quality +Fine-tuning is the practice of taking a pre-trained foundation model and +adapting it to a specific use case by introducing new data. + + +1.3.3 Distillation 33 +Figure 1.2: Fine-tuning a model changes the values of the model weights while +keeping the same overall architecture. +If you can fi ne-tune a small model to pass your evals, you set yourself +up for an easier time hitting your latency and cost targets for inference. +A great example of a domain where fi ne-tuning is effective is translating +English into SQL, a language used to query databases. +General-purpose coding models are good at writing SQL, but these +models are hundreds of billions of parameters. SQL is a relatively +constrained language, so for an application that only needs to generate +SQL queries from natural language prompts, a tiny fi ne-tuned model +of just a few billion parameters can reach equivalent performance on +this specifi c task. +Text-to-SQL is an extreme example – many domains won’t support such a +vast reduction in model size – that illustrates what’s possible with a cleanly +scoped domain, a strong set of evaluation criteria, and high-quality labeled +data to use for fi ne-tuning. + 1.3.3 Distillation +What if you could retain most of the intelligence of a large model at a +fraction of the size? That’s the idea behind distillation. + + +34 Chapter 1: Prerequisites +Figure 1.3: Distillation preserves the behavior of a large model in a smaller, more +efficient model. +Distillation is the process of using a large “teacher” model to train a smaller +“student” model to emulate the larger model’s behavior. Unlike fi ne-tuning +on synthetic data, where the model is trained on input-output pairs, dis- +tillation shows the student model the teacher model’s actual probability +distributions, not just its fi nal answers. +Where fi ne-tuning teaches a model to perform better in a specifi c domain, +distillation teaches the model how to emulate the behavior – good and +bad – of a larger model. +Distillation sees substantially less real-world use than fi ne-tuning. +When a frontier lab releases a family of models of different sizes, the +smaller models are generally not distilled from the larger ones. Instead, +the models are independently trained to prevent the biases of the large +models from artifi cially limiting the smaller models. But if the lab only trains +a large model, distillation can make that model more accessible. +In January 2025, open model research lab DeepSeek released their +then-fl agship reasoning model DeepSeek-R1. As the model was so large +(671B parameters), they also released distilled versions of the model on +top of the most popular open model architectures at the time: Llama 3 +and Qwen 2.5. + + +1.4  Measuring Latency and Throughput  35 +These distilled models showed similar reasoning behavior to the main +DeepSeek-R1 model, albeit with worse intelligence benchmark scores, +but the distilled models were relatively small and could take advantage of +existing performance work for the Llama and Qwen architectures. +At publication, these DeepSeek-R1 distills are still among the most pop- +ular distilled models on Hugging Face, along with distills of models like +Whisper for audio transcription and some image generation models. +1.4  +Measuring Latency and Throughput +The two most common performance metrics for LLMs are TTFT (time to +first token) and TPS (tokens per second). For modality-specific metrics +beyond LLMs, see chapter 6. +Time to first token (TTFT) +Tokens per second (TPS) +With streaming output, how long +does it take for a user to see the first +output token? +How many tokens each second does +the user receive after the first token is +generated? +Based on compute-bound prefill +Based on bandwidth-bound decode +Lower TTFT == better latency +Higher TPS == better latency +While TTFT is a clear term, TPS is less precise. TPS can be a latency +metric (tokens per second per user) or a throughput metric (tokens per +second for the entire inference service). +Most people use TPS to mean a per-user latency metric. When needed, +use more specific terms: +• Perceived TPS: The observed tokens per second per user after the +first token (latency). +• Total TPS: The total number of tokens generated each second by the +inference service (throughput). +• Inter-token latency (ITL): The time between subsequent tokens. An +ITL of 10 milliseconds equates to 100 tokens per second per user. + + +36 Chapter 1: Prerequisites +Figure 1.4: TTFT is the time it takes to generate the fi rst token, while TPS measures +how quickly subsequent tokens are generated. +TTFT and TPS are most commonly used for user-facing LLM systems like +chatbots, where output is streamed to the user. For other requests, like a +tool call for an agent, you instead measure latency as total response time +as the tokens aren’t useful individually. + 1.4.1 +Latency Percentiles +One important distinction when discussing and comparing metrics is what +percentile you are measuring. +Figure 1.5: Mean latency is generally higher than P50 latency due to outliers in the +right-skewed distribution of inference times. + + +1.4.2  End-to-End Metrics  37 +The naive approach is to simply look at an average (mean) TTFT or TPS. +However, this does not tell the whole story. LLM total response time is gen- +erally a right-skewed distribution, where most times concentrate around a +mode, but outliers can take significantly longer. +These outliers can dramatically affect user experience and trust in a prod- +uct. It’s not good enough for most interactions to feel snappy if one in every +ten takes several seconds. +Instead, inference engineers measure latency in percentiles. +P50 +Median latency +1 in every 2 requests is slower +P90 +90th percentile latency +1 in every 10 requests is slower +P95 +95th percentile latency +1 in every 20 requests is slower +P99 +99th percentile latency +1 in every 100 requests is slower +While driving down average latency matters, good performance work also +focuses on reducing P90/P99 latencies for a more reliable user experi- +ence. +1.4.2  End-to-End Metrics +The other important distinction in metrics is whether you’re measuring +solely inference time – the on-GPU time required to generate tokens – or +an end-to-end measurement that accounts for network latency and any +queue time. +Both inference-only and end-to-end metrics are valuable to know. Infer- +ence time tells you how effective your model performance work is, while +end-to-end metrics reveal your users’ perception of how fast your applica- +tion is. When inference time is fast but end-to-end time is slow, turn your +attention to infrastructure rather than model performance optimization. + + + diff --git a/library/Inference Engineering/Chapter-04-Models/Chapter-04-source-text.md b/library/Inference Engineering/Chapter-04-Models/Chapter-04-source-text.md new file mode 100644 index 0000000..84b400f --- /dev/null +++ b/library/Inference Engineering/Chapter-04-Models/Chapter-04-source-text.md @@ -0,0 +1,827 @@ +# Inference Engineering — Chapter-04: Models +- **Source**: /library/Inference Engineering/source-file.pdf +- **PDF pages**: 41–72 +- **Pages without text**: 41, 42, 72 + +--- + + +CHAPTER 2 +Models + + + + + +Models  41 + +Models +Inference engineering is the practice of making generative AI models +faster, less expensive, and more reliable – without sacrificing the quality +that makes them so valuable. Both improving performance and preserving +quality require a strong intuition for how models work under the hood. +Generative AI models are a composition of big, complex neural networks. +The history of neural networks stretches back to the 1950s, when the first +perceptrons for simple binary classification were implemented in hard- +ware. In the following decades, perceptrons were abandoned but then +reinvented from single to multi-layer perceptrons with a new concept, +back-propagation, which introduced hidden states between layers and +a learning procedure that repeatedly adjusts weights within the network. +These neural networks had only a few layers. In the 2000s, research +began into deep neural networks with dozens of layers. In 2012, Alex- +Net became the first deep neural network to show promising real-world +capabilities and the effectiveness of GPUs for deep learning, leading to +new architectures like word embedding models for text and Generative +Adversarial Networks (GANs) for images. +But the story truly starts in 2017, when Vaswani and colleagues published +the seminal paper “Attention Is All You Need,” introducing the transformer. +A transformer is a neural network with an attention mechanism that can +learn relationships between various parts of a sequence. +Transformers are the foundation of generative AI. Transformers aren’t just +for LLMs, they power every modality of model from embedding to voice to +image and video generation. +Across modalities, there are two important styles of transformer-based +models: +• Autoregressive token generation: Start from a tokenized sequence +and predict the most likely next token. +• Iterative denoising: Start from random noise and refine toward the +most likely output via diffusion. + + +42  Chapter 2: Models +This chapter explores the architectural details of LLMs (autoregressive +token generation) and image generation models (iterative denoising). +2.1  +Neural Networks +Generations of research into neural networks form the theoretical foun- +dation for generative AI. +To be a productive inference engineer, you need a basic intuition for +essential concepts in neural networks. This section provides a high-level +introduction; Appendix B offers recommendations for further reading. +The fundamental unit of a neural network is a node (a.k.a., neuron). A +node is a short program that takes an input, multiplies it by some weights, +adds some bias, and returns the result. +A group of nodes forms a layer. Nodes within a layer are independent +of each other – they do their own calculations. The connection between +nodes, or the “network” in a neural network, is between layers, where the +nodes in a layer receive the output of the previous layer. +The neural networks behind LLMs contain dozens to hundreds of layers. +There are three types of layers: +• Input layer: The first layer, which accepts and processes the input to +the neural network. +• Hidden layers: Every layer between the first and last, which iteratively +transform the input to arrive at an output. +• Output layer: The final layer, which returns the prediction from the +network. +Each layer produces an output that the next layer reads as input. For the +hidden layers, these outputs are called hidden states. +Hidden states are one kind of internal representation for data within a neu- +ral network. A key aspect of internal representation is its dimensionality, +or the actual size of the vectors used. + + +2.1 Neural Networks 43 +Figure 2.1: Multi-layer neural networks have one input layer, many hidden layers, and +one output layer. +Internal representations for text input increase the dimensionality, encod- +ing text chunks into vectors of hundreds or thousands of numbers to cap- +ture semantic meaning. But internal representations for image models +reduce the dimensionality from millions of pixels down to a manageable +size. +There are neural networks for creating these internal representations, and +there are neural networks for using them: +• Encoder: Takes an input like text or an image and creates an inter- +nal representation of the input that includes additional information and +semantic meaning. +• Decoder: Uses the internal representation to generate an output like +text or an image. +Neural networks are composable. You can combine multiple neural net- +works together into a single model or use them sequentially to build a +pipeline. + + +44 Chapter 2: Models +Modern LLMs are decoder-only, while encoder-only models are somewhat +rare today, with old-school text embedding models from the BERT family +as a prominent example. +Many models in other modalities use an encoder-decoder architecture. +Whisper, a popular open model for audio transcription, uses an encoder +to process audio input and a decoder to generate text tokens. + 2.1.1 +Linear Layers and Matmul +The most essential operation within a neural network is a matrix multipli- +cation, or matmul. A matmul takes an input vector (a list of numbers) and +a matrix (a grid of numbers) and multiplies the vector through the matrix +to produce an output vector. +Within a neural network, a linear layer is the simplest form of matmul. +Given an input vector, the linear layer applies a weight matrix and adds +a bias vector: +Figure 2.2: In a matmul, the output vector y is the product of an input vector x and a +weights matrix W plus a bias vector b. +The weights of any given linear layer are a small part of a generative AI +model’s total weights, and the individual values within the weights matrix +are set during training. + 2.1.2 +Activation Functions +Matrix multiplication is composable, meaning that multiplying a vector +by two matrices is equivalent to multiplying that vector by the product of +those matrices. + + +2.1.2 Activation Functions 45 +Figure 2.3: Two matmul equations representing separate layers collapse due to +composition of linearity. +This is a problem for multi-layer neural networks because a series of linear +layers, each one a matmul, would collapse into a single layer with all of +the matrices multiplied together. +Deep multi-layer neural networks are useful because more layers use +more parameters effectively and encode more meaning in hidden states. +Neural networks separate layers by breaking linearity with an activation +function. Activation functions are non-linear to prevent composable matmul +from collapsing layers, and are differentiable or mostly-differentiable to +support back propagation. +One of the most basic activation functions in inference is ReLU, which +stands for Rectifi ed Linear Unit. ReLU is a simple function: if X is greater +than zero, return X, else return zero. There are dozens of activation func- +tions – including one named “Swish” thanks to its resemblance to the +Nike logo – but most follow the same general pattern of mapping negative +values to zero or near-zero, while keeping positive values unchanged. +Activation functions like ReLU, SiLU, Swish, and SwiGLU are fast to run, +easy to train on (as they are mostly differentiable, they have a gradient +at least for most values), and break linearity to support multi-layer neural +networks. + + +46 Chapter 2: Models +Figure 2.4: Activation functions like ReLU are used to break linearity in multi-layer +neural networks. + 2.2 +LLM Inference Mechanics +LLMs are autoregressive token generation models. An LLM generates +new tokens one at a time based on every previous token. +These tokens, the atomic units of language models, are numbers that rep- +resent chunks of text. Modern LLMs use subword tokenization, meaning +that each token is a word or a fraction of a word. +Figure 2.5: Subword tokenization uses one token per common word and punctuation +mark, but it splits less common words into multiple tokens. + + +2.2  LLM Inference Mechanics  47 +Converting text into tokens and tokens into text does not require any neural +networks. Instead, a tokenizer is a simple mapping between strings and +their numerical token representation. +A language model’s vocabulary is the complete mapping between tokens +and strings. Vocabularies and tokenizers vary from model to model, with +more recent models employing more efficient tokenization schemes; the +fewer tokens required to generate an output, the faster the end-to-end +inference. +Most models have over 100,000 tokens in their vocabulary. In other modal- +ities, like speech synthesis, the model vocabulary is expanded to let tokens +represent other information like audio waveforms. +Inference involves two or three sequences of tokens: +• Input sequence: The prompt, chat, context, functions, and other input +passed into the LLM. +• Reasoning sequence: Optionally, for reasoning models, an interme- +diate output sequence for thinking. +• Output sequence: The response generated by the LLM. +Combined, these sequences are limited to the model’s context window +(the total number of tokens the model can process and generate per +request). A request may further limit the output sequence length with a +max_tokens argument. +While the input sequence is a single string, LLMs are trained to accept +varied inputs like multi-turn chat sequences with roles, function signatures +for tool calls, and, in some cases, multimodal inputs. These inputs need +to be combined into a single sequence. This is handled by the chat tem- +plate, which differs subtly from model to model and must be implemented +correctly in the inference engine. +Tokenizing the input sequence with the chat template applied is step zero +for inference. Then, there are two primary phases of inference: +• Prefill: Process the input sequence to calculate attention for each input +token and store those values in a KV cache. + + +48 Chapter 2: Models +• Decode: Perform forward passes through the model to generate tokens +autoregressively. +Each forward pass in the decode phase must generate a token. This takes +a few extra steps, as neural networks output vectors, not tokens. +The output layer of the neural network used in LLM decode generates a +vector of logits. The length of this vector equals the model’s vocabulary +size. After normalization, these logits represent the probability of each +potential token in the vocabulary being the correct output. +Figure 2.6: A decode pass generates a logit for each token in the model’s vocabulary, +then normalizes logits to percentages. +The output token is selected via a weighted random number generation +based on the normalized probability vector. You can nudge that process +via inference arguments: +• Temperature: Adjust the logits themselves before normalization. +• Top-k: Select the k most likely tokens after normalization, then re-nor- +malize among them. +• Top-p: Select the smallest set of tokens after normalization whose +probabilities add up to p. + + +2.2.1  LLM Architecture  49 +A lower temperature, top-k, or top-p makes LLM output more predictable +as the model is constrained to selecting highly likely tokens. Setting the +temperature to 0 or top-k to 1 makes token selection deterministic (always +select the highest-probability token). +When generating structured output, where the output conforms to a +schema like JSON, there are additional tools like logit biasing for further +directing the output of an LLM. These apply after each forward pass and +support important LLM abilities like tool use; their correct implementation +is essential for high-quality inference. +This logit generation and token selection process continues until the model +decides that the stop token, a special value signifying the end of the output +sequence, is generated (unless the context window or max tokens limit +is hit first). +The two key pieces of this inference loop are generating the KV cache +during prefill and generating output tokens (or more specifically the logit +vectors that become tokens) during decode. These steps take the over- +whelming majority of the time and resources during inference as they rely +on large neural networks. +2.2.1  LLM Architecture +Every LLM on Hugging Face (the largest repository of open models) +includes a config.json file: a few dozen lines detailing the model’s +architecture. +The architecture of a model is a collection of decisions made during the +training process about the nature and shape of each component of the +model. Within a single architecture, there may be: +• Multiple sizes: Models at different parameter counts, like Llama 8B +and 70B. +• Multiple variants: The “base” and “instruct” variants of a given model +share the same architecture. +• Unlimited fine-tunes: Methods like LoRA (Low-Rank Adaptation) +change behavior, not architecture. + + +50  Chapter 2: Models +These architectures matter because they determine runtime and engine +support. If you have a highly optimized deployment of a given architecture, +you can deploy another variant of the same architecture and enjoy the +same performance improvements. +Model architecture is one of the first lines in most configuration files. To +parse an architecture name like Qwen3MoeForCausalLM: +• Qwen: The model family, or the brand name of the model. +• 3: The major version of the architecture within the family. +• MoE: Indicates a Mixture of Experts model (section 2.2.4). +• CausalLM: Indicates a causal language model. +A causal language model predicts the next token in a sequence based on +previous tokens, as opposed to, for example, a masked language model +which fills in the blank based on surrounding tokens to the left and right. +All generative LLMs today are causal language models. +Beyond the architecture’s name, the config.json file contains informa- +tion about the nature and dimensions of the various layers that form the +underlying neural networks of a model and the vectors that pass through +them during inference. +2.2.2  Transformer Blocks +The main body of an LLM is a series of dozens to hundreds of transformer +blocks. These blocks form the core of a large neural network with three +kinds of layers: +• Embedding layer: The input layer of the neural network takes tokens +and returns embeddings. +• Transformer blocks: The hidden layers within the network are trans- +former blocks that generate a prediction. +• Output layer: Also known as a language modeling head or LMHead, +converts the hidden states from the transformer blocks into a vector of +logits, one for each token in the model’s vocabulary. + + +2.2.2 Transformer Blocks 51 +Within the transformer blocks, there are sublayers for attention, a feed-for- +ward neural network, and normalization. +Figure 2.7: Transformer block diagram, adapted from “Attention Is All You Need” +(Vaswani et al., 2017). +The feed-forward neural network is a multi-layer perceptron. These linear +sublayers make up the majority of the trainable weights within an LLM, +while the attention sublayers are the second-largest component. Other +components like normalization and activation functions are a rounding +error in the model’s size. + + +52 Chapter 2: Models +While linear sublayers are the largest portion of the weights, the more +complex operation for inference is attention. + 2.2.3 Attention +Attention is the mechanism transformers use to relate a given token to +other tokens in the sequence. Humans are good at interpreting the rela- +tionship between words. Attention brings the same capability to LLMs. +Consider the sentence “I decided to write a book because I thought it would +be easy, but it was actually hard.” Attention shows that the word “it” in the +sentence refers to writing a book. +The standard form of attention is scaled dot-product attention, as shown +in this equation. +Figure 2.8: The attention equation, adapted from “FlashAttention: +Fast and Memory-Efficient Exact Attention with IO-Awareness” (Dao et al., 2022). +Attention takes three inputs: +• Q (queries): The embedded representation of the token being gener- +ated or updated. +• K (keys): Representations of all prior tokens. +• V (values): Computed attention values for all prior tokens. +Attention sublayers within models are multi-head, where each head is +one attention operation. If you visualize the architecture of a neural net- +work, these heads are parallel to each other on the same sublayer. Each +head could be responsible for attending to different kinds of relationships +between tokens, like subject-verb agreement and co-reference resolution. + + +2.2.4  Mixture of Experts Models  53 +There are two main types of attention: +• Self-attention: Q, K, and V are all from the same sequence. +• Cross-attention: Q is from a different sequence than K and V, condi- +tioning Q on external information. +LLMs use self-attention (with a causal mask to prevent looking ahead +in the sequence), while image generation and multimodal models also +use cross-attention (e.g., between the image being generated and the +associated text prompt). +Because attention looks for relationships between the current token and +every previous token in the sequence, it’s a quadratic-time equation with +respect to sequence length. As context expands, attention slows. +In practice, attention is linear, not quadratic, thanks to the KV cache. The +KV cache is a standard component in attention implementation and stores +key-value pairs for each previous token. By looking up this information +in the KV cache rather than recomputing it each time, attention runs in +linear time. +The KV cache is built during LLM prefill, used and updated during decode, +and lives on GPU memory by default. Storing, accessing, and re-using +the KV cache, a major topic in inference engineering, is covered in detail +in section 5.3. +2.2.4  Mixture of Experts Models +The density of a neural network is determined by the number of con- +nections between layers. Denser networks retain more information, while +sparser networks take less compute and memory to run. +Mixture of Experts (MoE) is an architecture optimization that adds sparsity +to linear layers. Rather than a single giant matrix, an MoE model has hun- +dreds of smaller matrices (the experts) and routes each input to a small +selection of experts. This is called activating the experts. +For an MoE model like Qwen3-235B-A22B, 22 billion parameters out of the +model’s 235 billion total parameters are activated per request. Thanks to + + +54 Chapter 2: Models +their low number of active parameters, MoE models are highly efficient for +single-request local inference. However, in batched inference on production +servers, different requests activate different experts, and you should expect +almost all of the model parameters to be active at any given time unless +sparsity is achieved in large-scale Expert Parallelism (section 5.4.2). +Expert routing is granular. Each forward pass through a model generates +one token by working through every layer of the model. The router, a tiny +model within the LLM, picks which experts to activate at each layer of +the model. In the Qwen example, with 128 experts, the router picks eight +experts at each of the 94 layers for every token that is generated. +MoE architectures are especially popular for larger models with 100B+ +parameters, though there are MoE models as small as 20 to 30 billion +parameters. Mixture of Experts unlocks a new form of inference parallelism +called Expert Parallelism, which enables high-throughput inference for +large models on multiple GPUs. +Figure 2.9: Mixture of Experts architecture includes both sharding and replicating to +take advantage of multi-GPU inference. +Models under 32B parameters, and especially models under 8B parame- +ters, tend to use traditional dense architectures efficiently. Domain-specifi c +models for tasks like tab completion also don’t gain much benefi t from MoE +as the entire model is effectively one expert. + + +2.3  Image Generation Inference Mechanics  55 +2.3  +Image Generation Inference Mechanics +Image generation models take a text prompt and create an image based +on the prompt. These models slightly predate the public rise of LLMs, with +both closed models from Midjourney and open Stable Diffusion models +first released in the summer of 2022. +Image generation models aren’t monolithic models like LLMs. Instead, +they are pipelines of multiple models working together to generate images. +Within a foundation model for image generation, there are three essential +components: +• Text encoder: Converts the text prompt into instructions that the image +generation model can understand. +• Denoising model: The heart of the model, iterates from noise to an +image based on the prompt. +• Variational Autoencoder (VAE): Converts the model output from latent +space to pixel space. +This pipeline mentality extends through the image model ecosystem. +Beyond the base model, image inference often includes: +• LoRAs: Lightweight fine-tunes to change style and enhance quality. +• ControlNets: Outlines and edges to steer output images to match broad +shapes and colors. +The vast and rich open-source ecosystem around image generation +models includes tools like ComfyUI for building complex pipelines of +foundation models and adaptions, swapping components to produce +unique outputs. +The entire image generation pipeline operates in latent space. An ordinary +HD image may be 1024x1024 pixels; that’s well over a million pixels. As +the denoising model needs to calculate attention over the entire image in +parallel, it would be infeasible to work in pixel space. +Latent space is a low-dimensional representation of an image. A latent +space matrix for an image may be 128x128, or about one percent of the +total values of the pixel space it represents. + + +56 Chapter 2: Models +The latent space is initialized as random values, or noise. The denoising +model refi nes that noise into an image over a series of steps based on +the text prompt. Each step updates the entire latent space, unlike LLMs +which process tokens one at a time. Most image generation models take +30 to 50 steps to create a high-quality image. +Figure 2.10: Diffusion-based models iteratively generate an image from noise, +generally over 30 to 50 steps. +Within each step, the model runs two forward passes: one with condition- +ing (the text prompt) and one without conditioning. These generations are +then combined based on a guidance scale. Because of this two-part step, +a 50-step image generation actually takes 100 forward passes. +This process, and other essential parts of image generation, are con- +trolled on a request-by-request basis via inference arguments. The most +important arguments are: +• Prompt: Describes what the image should look like. +• Negative prompt: Separately describes any styles or objects that +should not be in the image. +• Number of steps: Trades off speed and quality with the number of +denoising steps, 30 to 50 for most models. +• Guidance scale: Controls the balance between creativity and prompt +adherence, integer value generally around 4. +• Image size: Selects from a fi xed menu of resolutions and aspect ratios +for the output image. +While these core mechanisms are common across image generation +models, their architecture has evolved considerably in the past few years. + + +2.3.1 Image Generation Model Architecture 57 + 2.3.1 Image Generation Model Architecture +Image generation models are built on transformers, specifi cally diffusion +transformers. A diffusion transformer is very similar to the transformers +that LLMs use, but instead of processing embedding representations of +discrete tokens, it processes image data. +Diffusion transformers look at images in patches. When training a text-to- +image model, the images in the training data are fed in via overlapping +patches of 2x2 or 4x4 pixels, which are then embedded into latent space. +Inference works in the opposite direction, with latent space transformed +back to pixels once the image generation is fi nal. +Image generation models are pipelines of multiple models, including a text +encoder, denoising model, and variational autoencoder. A clean example +of this pipeline is Stable Diffusion XL (SDXL). SDXL is an old model, but +its architecture remains relevant. +Figure 2.11: SDXL architecture pipeline, adapted from “SDXL: Improving Latent +Diffusion Models for High-Resolution Image Synthesis” (Podell et al., 2023). +SDXL’s pipeline contains two diffusion models for denoising, the base and +refi ner. These models were trained for separate tasks: the base model +goes from pure noise to a coherent image, while the refi ner model adds +details and ensures prompt adherence. +Modern models substantially outperform SDXL with better prompt adher- +ence; accurate faces, hands, and details; legible text rendering; and sup- +port for image-to-image inference. These models, like the Qwen Image + + +58  Chapter 2: Models +family, are broadly similar to the SDXL pipeline, but with larger and more +capable models at every step. +Component +SDXL (2023) +Qwen Image (2025) +Text encoder +CLIP-based model +Qwen 2.5 VL (7B) +Denoiser +<4B parameters +20B parameters +VAE +Single encoding +Dual encoding +The substantial growth in capability in modern image generation models +like Qwen Image comes from larger component models and more com- +plex pipelines. New abilities like legible text rendering and photorealistic +human faces comes from switching from tiny NLP models to full LLMs +for text encoding and increasing parameter counts on denoisers by a +factor of five. +These larger models take more resources to run. Fortunately, as models +have grown larger, GPUs have grown more powerful, though inference +engineers can’t rely on hardware gains alone to run these models effi- +ciently. +The latest research direction in image generation models is blending dif- +fusion transformer architecture with LLM architecture. +Anything that can be tokenized can be modeled as an LLM. LLMs have +the advantage of baked-in text understanding, and they are already an +important component of image generation pipelines. +LLMs solve many of the problems inherent to diffusion models. Where +diffusion models can only produce a fixed size of output, LLMs are autore- +gressive and can produce a variable-length output. And where image mod- +els need up to 100 forward passes to generate an image, LLMs generate +tokens in a single forward pass. Models like HunyuanImage-3.0 use this +LLM-style architecture. +While this is a new frontier in image generation, there are other existing +architectures for accelerating image creation and for extending image +models to video generation. + + +2.3.2  Few-Step Image Generation Models  59 +2.3.2  Few-Step Image Generation Models +The most time-consuming part of image generation is the 30 to 50 denois- +ing steps. Rather than making each step faster, what if there was a way +to optimize image models by simply using fewer steps? +Few-step image generation models are trained to do just that: create +high-resolution images with eight or fewer denoising steps. These models +are 80 to 90 percent faster out of the box than traditional image generation +models, though their output quality is noticeably lower. +There are two primary methods for creating these models: +• Latent consistency: Train a model to predict the target latent image +vector directly and repeat the prediction two to four times to enhance +quality. +• Distillation: Use adversarial distillation and/or progressive distilla- +tion to train a small model to emulate a larger one in fewer inference +steps. +Distillation is more common than latent consistency today. When new +image models like FLUX and Qwen Image are released, members of the +open image model community create distillations in addition to quality and +style-oriented LoRAs. +If you have a latency-sensitive use case where quality is less important, +like real-time generative filters, consider few-step image generation mod- +els. +2.3.3  Video Generation +Video generation models are architecturally similar to image generation +models, just bigger. They have three to five times more parameters and +encode ten to one hundred times more information in latent space. +The naive approach to video generation is to work frame-by-frame. Early +video generation used this framewise approach: first generating a starting +frame, then using that frame to generate the next frame, and so forth. + + +60  Chapter 2: Models +The issue with a framewise approach is error accumulation. Small issues +early on compound with each frame, and the video goes off the rails +quickly. +Instead, modern video models hold the entire video in latent space and +modify it on each denoising step. Each frame attends to each other frame +and is updated on every forward pass. +If latent space for an image model represents two physical dimensions, X +and Y, then latent space for a video model represents three dimensions: +X, Y, and T (time). +The main limitation of this approach is that videos are a fixed number of +frames, just like image generation models have fixed aspect ratios. Modern +video generation models create sequences of a few seconds. +The constraint on video length is compute resources. Even with the lat- +est GPUs, attention over massive latent space is extremely expensive, +taking several seconds of inference for each second of video. Video +generation models are so compute-intensive that they typically run with +a batch size of one, meaning a full node of eight GPUs is working on a +single request. +While attention on each denoising step is expensive, video generation +models have the same total number of steps as image models, generally +around 50 steps. +Video generation is a recent modality compared to LLMs and image gen- +eration. Their limitations line up with the limitations of LLMs two years ago. +LLMs (Late 2023) +Video gen models (Late 2025) +High TTFT, Low TPS +Slow generation times +Frequent hallucinations +Unrealistic physics +Maxed out Ampere GPUs +Max out Blackwell GPUs +Limited context windows +Short video outputs +Today, these limitations are mostly eliminated from LLMs. Removing +them from cinematic video generation, as well as related areas like +world models, 3D object generation (XYZ dimensions rather than XYT), + + +2.4  Calculating Inference Bottlenecks  61 +and other generative AI rendering models is a highly active area of +research. +One important direction in research is coming back around to the idea +of autoregressive generation, like with blending LLM architecture into +image generation models. Rather than pure framewise video gener- +ation with its unusable error accumulation, new techniques like Self +Forcing combine a global view of quality with an iterative approach to +generation. +Adding autoregressive components to video models can partially address +the bottleneck of attention, though it remains the most important and +expensive component of inference. +2.4  +Calculating Inference Bottlenecks +In a perfectly optimized system, every resource is fully utilized at all times. +In GPUs, there are two main resources: +• Compute: The number of floating-point operations per second that the +GPU can achieve. +• Memory bandwidth: The number of bytes that the GPU can move per +second. +Ideally, compute is never sitting idle waiting for information from memory, +and memory bandwidth never goes unused waiting for compute to finish. +In the real world, systems have bottlenecks: imbalances where one +resource is idle while another is saturated. Discovering these bottlenecks +is the first step to improving performance. If a certain operation is bottle- +necked on memory bandwidth, no amount of compute optimization will +make the system faster, and vice versa. +In most cases, inference systems have the following bottlenecks: +• LLM prefill (KV cache construction) is compute bound. +• LLM decode (token generation) is memory bound. +• Image and video generation are compute bound. + + +62 Chapter 2: Models +When optimizing performance on each of these phases, the goal is to +make the bottleneck less limiting to system-wide performance. For + example, batching multiple requests together makes LLM decode less +memory bound because processing a batch of requests uses more com- +pute for the same amount of memory traffic. + 2.4.1 Ops:Byte Ratio and Arithmetic Intensity +Each GPU has a specifi c compute speed (measured in operations per +second) and memory bandwidth (measured in gigabytes or terabytes +per second). Compare these to determine the ops:byte ratio of a given +GPU. +For example, an H100 GPU in FP16 can perform 989 teraFLOPS of dense +computation against 3.35 TB/s of memory bandwidth. This yields an ops:- +byte ratio of about 295. +For inference in FP16 to be perfectly balanced (as all things should be) on +an H100 GPU, the inference system needs to perform 295 fl oating point +operations for every byte of memory it accesses. +To fi gure out that ratio, calculate the arithmetic intensity of the algorithm. +Arithmetic intensity, also known as operational intensity, is the ratio +between work and memory traffic for the calculation at hand. +Figure 2.12: The equation for arithmetic intensity. +Where ops:byte was measured on a per-second scale, arithmetic intensity +is measured across the execution of a single function or algorithm. +Arithmetic intensity is visualized with a roofl ine model, which charts perfor- +mance against the bandwidth ceiling (a diagonal line) and the performance +ceiling (a horizontal line). + + +2.4.2 LLM Inference Bottlenecks 63 +Figure 2.13: A roofl ine chart shows the switch from memory to compute bottleneck +based on arithmetic intensity. +Plotting against the roofl ine model reveals if the algorithm is: +• Compute bound: When the arithmetic intensity is higher than the hard- +ware’s ops:byte ratio and hits the horizontal performance ceiling, it’s +compute bound. +• Memory bound: When the arithmetic intensity is lower than the hard- +ware’s ops:byte ratio and hits the diagonal bandwidth ceiling, it’s mem- +ory bound. +To fi nd a bottleneck, look at arithmetic intensity for the most expensive +calculations in a system. For inference, one such calculation is attention. + 2.4.2 LLM Inference Bottlenecks +LLM inference has two phases: +• Prefi ll: Determines the time to fi rst token (TTFT) and is compute-bound. +• Decode: Determines the tokens per second (TPS) and is memory- +bound. + + +64 Chapter 2: Models +For each phase, you can prove the existence of the bottleneck by compar- +ing the arithmetic intensity of the most important operation to the ops:byte +ratio of available hardware. +In both prefi ll and decode, the most expensive operation is attention. The +exact arithmetic intensity of attention depends on the model architecture +(dimensions, heads, etc), the input sequence length, and the implemen- +tation of the attention algorithm. +The essential difference is that prefi ll processes the entire input sequence +in parallel, while decode generates tokens one at a time. +For prefi ll, the model weights are loaded a single time, then a series of +large matrix multiplication between the matrix of inputs and the attention +matrices occurs. This is a lot of calculations versus a single read from +memory, creating a high arithmetic intensity. +On decode, the model weights are loaded for every token, which is gen- +erated via much less-expensive vector-matrix multiplication. In this case, +relatively few fl oating-point operations are needed compared to loading +the entire model weights, so the arithmetic intensity is low. +As an example of calculating exact arithmetic intensity, consider a decode +step for a model with a 128-dimensional attention head (d=128) on a +sequence of 4096 tokens (N=4096). For this analysis, use the standard +algorithm for attention without any optimizations. +Figure 2.14: Standard attention implementation, adapted from “FlashAttention: +Fast and Memory-Efficient Exact Attention with IO-Awareness” (Dao et al., 2022). +Based on the parameters of this exercise, establish the size of these +matrices: + + +2.4.2 LLM Inference Bottlenecks 65 +• N: The sequence length, established as 4096. +• d: The dimensionality of the attention head, set to 128. +• Q, K, V: Given as Nxd, or 4096x128. +• S, P: Calculated as NxN, or 4096x4096. +• O: Calculated as Nxd, or 4096x128 +Assume FP16 inference, where each value in the matrix is two bytes. For +reference, a 4096x4096 matrix is about 32 MiB, or about the same amount +of data as a high-resolution RAW DSLR photo. +Each of the three lines of the attention algorithm follows the same pat- +tern: load data from memory, perform a calculation, and store the result +to memory. +Figure 2.15: Memory movement (reads and writes) and compute work for the +attention implementation in Figure 2.14. +To calculate the total memory movement, sum the fi rst and third columns, +which track reads from and writes to GPU memory: +Figure 2.16: The total memory movement for a kernel is the sum of all reads and +writes across the three steps. + + +66 Chapter 2: Models +To calculate the total compute, sum the second column: +Figure 2.17: The total compute for the kernel is the sum of operations across the +three steps. +To calculate the arithmetic intensity, compare the work (total compute) to +the memory traffic: +Figure 2.18: The arithmetic intensity of a kernel is the total work (number of compute +operations) divided by the memory movement +For this example, the arithmetic intensity of 62 is much lower than the +H100 GPU’s ops:byte ratio of 295. The exact numbers vary by model, +sequence length, and hardware, but this example illustrates the general +principle that decode is memory bound. +Calculating arithmetic intensity like this is an academic exercise, not a +routine task for inference engineers. But seeing it once is useful for build- +ing intuition. + + +2.4.3  Image Generation Inference Bottlenecks  67 +2.4.3  Image Generation Inference Bottlenecks +Image and video models are relatively small – they have a tenth as many +parameters as frontier language models – but their attention mechanism +is just as computationally demanding. +Image and video generation models use iterative denoising, not autore- +gressive token generation. +Just like attention for LLM prefill processes the entire input sequence at +once, attention for generating media must consider the entire image or +video object as represented in latent space. +Also like LLM prefill, image and video generation model inference is bot- +tlenecked on compute. Specific techniques for optimizing inference for +these modalities are featured in sections 6.5 and 6.6. +2.5  +Optimizing Attention +For LLMs, attention scales quadratically with the length of the input +sequence. Each calculation of attention depends on the K and V values +of each previous token. In practice, attention is a linear-time operation +during decode as the KV cache stores the results of key and value com- +putations for previous tokens. +Even a linearly scaling algorithm gets very expensive. Attention is one of +the most expensive parts of inference across models and architectures. +Naturally, optimizing attention is an important and highly active research +area. +Attention is a sensitive process because each token depends on every +previous token. Small errors in attention can accumulate quickly, making +attention optimization a delicate process. +Figure 2.14 showed that the attention algorithm itself is straightforward. +However, that basic implementation is inefficient. The intermediate matri- +ces S and P are stored at the end of one step, then immediately loaded +in the next step. + + +68  Chapter 2: Models +There are two strategies for optimizing attention: +• Implementation improvements: Write higher-performance kernels +that use memory and compute more efficiently. +• New algorithms: Create algorithms for attention that scale in bet- +ter-than-quadratic time with minimal quality loss. +Implementation improvements are still limited by attention’s quadratic +time complexity, but are lossless (do not affect quality) and make infer- +ence feasible for long sequences on today’s hardware. Other algorithmic +approaches trade off quality for time and space complexity, though training +techniques can minimize the impact. +The most famous implementation of attention is the FlashAttention series +of papers and kernels. Where the basic algorithm can be implemented +in a handful of lines of code, FlashAttention uses tens of thousands +of lines to implement attention in hand-fused kernels built for specific +GPUs – FlashAttention for H100 uses different code than FlashAttention +for B200. +FlashAttention works by eliminating excess reads and writes from memory +and laying out the attention algorithm to precisely fit the GPU’s capabilities. +FlashAttention is especially useful for compute-bound operations like LLM +prefill and video generation. +Another important implementation is PagedAttention. KV caches quickly +grow large, filling GPU memory and taking time to read. PagedAttention +partitions the KV cache into blocks (pages) that can be accessed via a +lookup table. This means the KV cache can be stored across the GPU +with fragmented memory rather than requiring a single contiguous block +of memory. +While FlashAttention and PagedAttention are valuable optimizations, they +don’t change the fact that attention is a quadratic algorithm. New variants +of attention improve the underlying time and space complexity: +• Sliding window attention: Computes attention for a sliding window of +the previous w tokens, turning attention from O(N^2) to O(Nw) where +w is often in the range of 8K to 32K. + + +2.5  Optimizing Attention  69 +• Gated attention: Various types of layers introduced in training allow +for approximating attention for certain chunks of context in linear time +with respect to chunk length. +• Linear attention: Replaces the quadratic softmax equation with a lin- +ear-time algorithm that approximates attention. +• Compressed attention: Periodically compresses context from earlier +in the sequence, attention considers both compressed context and +uncompressed recent tokens. +• Multi-latent attention: Approximates attention in low-dimensional +latent space. +Intuitively, it makes sense that tokens near each other in a sequence +affect each other more than tokens from much earlier. The sentence I am +writing now follows closely from the previous sentence, but less so from +the sentence at the start of this chapter. +This intuition can be extended through training. Algorithms like sliding +window attention, when applied during training, create models that keep +quality high when the same technique is used in inference. +Another avenue of research is avoiding attention altogether by using a +different architecture than transformers. Mamba is a selective state-space +model that replaces self-attention with a recurrent state update, achieving +linear scaling on sequence length. Hybrid models sometimes mix Mam- +ba-style state-space model blocks with transformer blocks. Applications of +state-space models are still limited, though hybrid models are becoming +more popular with open models like NVIDIA Nemotron 3 Nano adopting +hybrid architectures. + + + diff --git a/library/Inference Engineering/Chapter-05-Hardware/Chapter-05-source-text.md b/library/Inference Engineering/Chapter-05-Hardware/Chapter-05-source-text.md new file mode 100644 index 0000000..291b244 --- /dev/null +++ b/library/Inference Engineering/Chapter-05-Hardware/Chapter-05-source-text.md @@ -0,0 +1,651 @@ +# Inference Engineering — Chapter-05: Hardware +- **Source**: /library/Inference Engineering/source-file.pdf +- **PDF pages**: 73–94 +- **Pages without text**: 73, 74, 94 + +--- + + +CHAPTER 3 +Hardware + + + + + +Hardware  73 + +Hardware +Inference engineering relies on accelerators: powerful hardware designed +to load terabytes of data and perform trillions of operations per second. +The most common type of accelerator for inference is the GPU, and the +market leader in GPUs for inference is NVIDIA. This book focuses on +inference engineering for NVIDIA GPUs in the datacenter, but section +3.4 of this chapter covers other vendors of datacenter accelerators, and +section 3.5 covers local inference. +Across vendors, there are three types of GPUs on the market: +• Datacenter GPUs: Racked servers with interconnected high-perfor- +mance GPUs. Example: NVIDIA B200. +• Workstation GPUs: Individual desktop GPUs for professional work- +flows. Example: NVIDIA RTX Pro 6000. +• Personal computing GPUs: Individual desktop GPUs for everyday +use. Example: NVIDIA GeForce RTX 5090. +Inference at scale uses datacenter GPUs mounted on racks: refrigera- +tor-sized chassis with standardized power, networking, and cooling. +Datacenter GPUs like the NVIDIA B200 offer the highest individual per- +formance, but more importantly, include high-bandwidth GPU-to-GPU +interconnects, are installed in highly standardized configurations, and are +available by the millions in datacenters worldwide. +I doubt that you have a B200 GPU running under your desk. If you do, +send me a picture! Instead, inference on datacenter GPUs runs in one +of three modes: +• Cloud: GPUs are rented in someone else’s datacenter, usually hyper- +scalers like AWS and GCP or neoclouds like Coreweave and Nebius. +• On-premise: GPUs are purchased and installed in a datacenter that +you control directly. +• Air-gapped: GPUs are installed on-premise and you need to physically +access the GPUs to run inference. + + +74  Chapter 3: Hardware +Most inference engineers use cloud GPUs. Large enterprises and gov- +ernments run on-premise and air-gapped deployments, but cloud-based +GPUs offer the flexibility and access that fast-growing AI products need +to scale. +Even with these constraints, navigating the hardware landscape is com- +plex. From variations among cloud providers to NVIDIA’s own naming +conventions, there are many nuances in selecting the right accelerator. +3.1  +GPU Architecture +GPUs are throughput machines. Where CPUs are great at complex +sequential execution, GPUs are designed for simple, massively parallel +workloads. +Specifically, GPUs are great at performing one uniform operation on thou- +sands of independent pieces of data. Given that AI model inference is a +series of vector and matrix multiplications, GPUs are a natural fit. +While the principle of highly parallel computation is simple, GPUs them- +selves are extraordinarily complex pieces of technology. Hardware engi- +neering is a fascinating multidisciplinary field, from the physics of powering +and cooling the chips to the impossibly tight tolerances involved in man- +ufacturing each component. +Inference engineers work at a comfortable level of abstraction above the +GPU hardware, but a strong mental model for what’s going on inside the +box is essential for building high-performance systems. +3.1.1  +Compute +If you’re familiar with CPUs, you’ve probably heard about cores, like an +8-core Intel i9 CPU in a high-end gaming computer. +In GPUs, cores have a different meaning. GPUs have Streaming Multipro- +cessors (SMs), while each SM contains multiple cores. There are three +types of compute in GPUs: +• CUDA Core: Operates on individual numbers (scalars). + + +3.1.1 Compute 75 +• Tensor Core: Operates on vectors and matrices. +• Special Function Unit (SFU): Accelerates certain mathematical oper- +ations like sin, cos, and log. +When measuring GPU compute for inference, measure in terms of Tensor +Core compute. SFUs are essential for softmax, but Tensor Cores are +responsible for Matrix Multiply and Accumulate (MMA) instructions, which +are foundational to inference. +The “accumulate” step in MMA means adding the product of two matrices +to a base matrix to produce the output, as shown in Figure 3.1. +Figure 3.1: Matrix Multiply and Accumulate (MMA) multiples matrix A by matrix B, +adds matrix C, and stores the result as matrix D. +Unlike cores, the concept of a thread is similar between CPUs and +GPUs. Where a CPU has dozens to hundreds of threads, GPUs have +tens to hundreds of thousands of threads that can work concurrently, +switch tasks in a single clock cycle, and execute simple instructions +in parallel. +Compute is measured in FLOPS (fl oating point operations per second) +and datacenter GPUs are capable of trillions or quadrillions of FLOPS +(teraFLOPS and petaFLOPS, respectively). However, when you read a +spec sheet, you’ll see two measurements for Tensor Core compute: +• Dense: The raw fl oating-point operations per second if every element +of the tensor is used. +• Sparse: In tensors with 2:4 structured sparsity, where 50 percent of the +values are 0, Tensor Cores can skip multiplication by 0. +A GPU’s FLOPS at a given precision with sparsity are often, but not always, +double that of dense operations at the same precision. By default, infer- +ence is dense, so ensure that you’re looking at FLOPS without sparsity. + + +76  Chapter 3: Hardware +FLOPS generally double with each halving of precision. A GPU capable of +one petaFLOPS on 16-bit numbers will be able to do two petaFLOPS on +8-bit numbers. This is relevant for inference – be sure to compare FLOPS +across GPUs at identical precisions. +Compute is the bottleneck for LLM prefill and for image and video gener- +ation. If you’re selecting hardware with one of these use cases in mind, +pick the accelerator with more FLOPS. +3.1.2  +Memory and Caches +GPUs contain high-speed onboard memory called VRAM. Just like the “G” +in GPU stands for graphics, the “V” in VRAM stands for video – a callback +to these accelerators’ original purpose. +Today, VRAM is added to GPUs in the form of HBM3, HBM3e, or HBM4, +all various generations of high-bandwidth memory. GPUs have dozens or +hundreds of gigabytes of VRAM. +There are two types of memory on any chip, CPU or GPU: +• DRAM (Dynamic RAM): General-purpose off-chip memory denomi- +nated in gigabytes. +• SRAM (Static RAM): Faster, more expensive, on-chip memory denom- +inated in kilobytes or megabytes. +VRAM is a type of DRAM. GPUs also feature SRAM on-chip in the form +of caches. GPUs have three levels of cache: +• L0: Instruction cache for a single Tensor Core. +• L1: Shared memory per Streaming Multiprocessor. +• L2: Global cache across Streaming Multiprocessors. +An H100 GPU has 256 KB of L1 cache per Streaming Multiprocessor and +50 MB total L2 cache on chip. +VRAM bandwidth measures the peak transfer rate between GPU cores +and VRAM via the memory bus. In practice, this determines how quickly +VRAM can supply data into the GPU’s cache hierarchy. + + +3.2 GPU Architecture Generations 77 +Figure 3.2: A GPU has multiple SMs, each of which has multiple Tensor Cores. L1 +cache sits within SMs and L2 cache is shared. +The total amount of VRAM on a GPU limits the size of the model you can +load onto it. The VRAM should hold the model weights, plus at least 50 +percent headroom for KV cache (more for long context, high batch sizes, +or video generation models). +If there isn’t enough VRAM available for the weights, loading the model +will fail with an OOM (out of memory) error. And if there isn’t enough +headroom, inference will be slow or crash with an OOM. +Memory bandwidth is the bottleneck for LLM decode at low to medium +batch sizes. High-end GPUs have terabytes per second of memory band- +width. If you are picking a GPU and want to generate more tokens per +second, select the accelerator with a higher memory bandwidth, like the +H200 instead of the H100. + 3.2 +GPU Architecture Generations +Hardware iteration cycles are slow. There are years of lead time between +fi nalizing the architecture design and shipping GPUs. +Given the speed of the AI industry, this tapeout and testing process means +that even next-generation GPUs were designed at a time when AI model + + +78 Chapter 3: Hardware +capabilities looked nothing like they do today. Designing a GPU archi- +tecture with staying power on the market requires foresight into how use +cases will evolve over the expected lifetime of the GPU. +For years, training AI models was the primary use for GPUs. Now, with +inference rising as the dominant use case, the latest architectures are +introducing inference-focused features. +GPU names, like B200, have two parts: +• Letter: Signifi es the architecture generation used in the chip. +• Number: Identifi es individual models within the generation. +For example, the H100 directly replaces the previous generation A100, +while the H200 is a larger GPU within the same generation (and is in turn +supplanted by the B200). +The numbering appears somewhat arbitrary, with different sets of numbers +used from generation to generation, but the general rule is that a bigger +number means a larger, more powerful, more expensive GPU. On the +other hand, the lettering for architecture is very meaningful. +Every one to two years, NVIDIA releases a new GPU architecture, which +powers all of their products from the datacenter to personal computers. +Each architecture generation introduces both improved base speeds on +compute and memory and new features for more efficient inference. +Since 1998, NVIDIA has named their GPU architectures for prominent +scientists. +Figure 3.1: NVIDIA GPU architecture names from 2017 through all announced future +architectures. + + +3.2.1  Hopper GPUs  79 +While GPU architectures go back decades, inference engineers generally +work within the three to five most recent generations of GPUs. Even for +cost-sensitive workloads, modern architectures’ efficiency often makes +them more cost-effective for large scale traffic, and of course newer archi- +tectures offer better performance. +You may still see Turing (T4) and Ampere (A10, A100) GPUs from time +to time in low-traffic or legacy systems, but most deployments today use +Lovelace (L4, L40), Hopper (H100, H200), or Blackwell (B200, B300) +GPUs. +The Hopper and Blackwell architectures offer low-precision Tensor Cores, +high-bandwidth memory, and inference-focused features, while Lovelace +GPUs are used for low-cost inference on small models. The upcoming +Rubin and Feynman architectures promise even greater performance +when released in 2026 and 2028, respectively. +3.2.1  Hopper GPUs +GPU +FP8 compute (dense) +Memory +Bandwidth +H100 +1,979 teraFLOPS +80 GB +3.35 TB/s +H200 +1,979 teraFLOPS +141 GB +4.8 TB/s +The Hopper architecture, named for Rear Admiral Grace Hopper, was first +released in March 2022 with the H100 GPU. +Hopper introduces support for FP8, a floating-point number format with +8 bits of precision. FP8 Tensor Cores are twice as fast as FP16 Ten- +sor Cores, and moving FP8 values around takes half as much memory +bandwidth. As section 5.1 discusses, this does not linearly translate to +double the performance, but for workloads that can be run in FP8, it’s a +significant gain. +The Hopper architecture adds fourth-generation Tensor Cores within +more and faster Streaming Multiprocessors than previous generations. +Alongside dynamic programming instructions, thread block clusters, and +distributed shared memory, Hopper GPUs give CUDA engineers more +tools for writing high-performance kernels. + + +80  Chapter 3: Hardware +One such kernel is FlashAttention 3, which improves performance and +memory efficiency for attention on Hopper GPUs. FlashAttention 3 takes +advantage of the new asynchronous data transfer and execution features +introduced with Hopper. +The H100 and H200 GPUs are among the most widely used inference +accelerators, and for good reason. The Hopper architecture is new enough +to be performant but established enough for industry-wide support and +highly optimized kernels, and H100 and H200 GPUs are right-sized for +common workloads across all modalities. +3.2.2  Ada Lovelace GPUs +GPU +FP8 compute (dense) +Memory +Bandwidth +L4 +242 teraFLOPS +24 GB +300 GB/s +L40 +362 teraFLOPS +48 GB +864 GB/s +The Ada Lovelace architecture, named for the first computer programmer, +was first released just six months after Hopper. +The two architectures are similar, with Lovelace acting as more of a coun- +terpart than a successor. Lovelace also supports FP8 inference. +Where Hopper is focused on AI applications, Lovelace GPUs are more +graphics oriented. Lovelace GPUs do not support NVLink interconnect. +This is a major limitation. Nodes with eight Hopper or Blackwell GPUs +use these high-bandwidth interconnects for efficient parallelism. Lovelace +GPUs must be used individually or via inefficient parallelism methods like +Pipeline Parallelism. +L4 GPUs can be a cheap and convenient way to run small models for +modalities like text embeddings and computer vision. +But L40 GPUs generally aren’t a great choice for inference. For the same +memory footprint, multi-instance GPUs (section 3.3.2) offer much higher +compute and memory bandwidth on fractional H100s. + + +3.2.3  Blackwell GPUs  81 +3.2.3  Blackwell GPUs +GPU +FP8 compute (dense) +Memory +Bandwidth +B200 +~5 petaFLOPS +192 GB +Up to 8 TB/s +B300 +~5 petaFLOPS +288 GB +Up to 8 TB/s +The Blackwell architecture, named for mathematician David Blackwell, +was first released in November 2024 with the B200 GPU, followed by the +B300. While the B100 does exist, it’s not common for inference. +Where Hopper introduced FP8, Blackwell goes further in low-precision +computing with FP4, a 4-bit floating point format, plus a set of microscal- +ing formats (MXFP8, MXFP4, NVFP4) for better quality retention during +inference. Section 5.1 explains these formats in detail. +Blackwell builds on Hopper’s asynchronous programming paradigm with +more features for loading and storing between tensor and global memory. +The updated FlashAttention 4 kernel relies heavily on tiling loads, compu- +tations, and writes in asynchronous pipelines. +The B200 and B300 are the new gold standard for inference, offering the +highest performance for large language models and demanding workloads +like video generation. Software support, optimized kernels, and general +availability for Blackwell GPUs have all come online in recent months, +marking an important transition in the inference industry. +3.2.4  Rubin GPUs +The Rubin architecture, named for astronomer Vera Rubin, will launch in +2026 as NVIDIA’s next-generation GPU architecture. +When evaluating new architectures, it’s important to reserve judgement +until you can run real-world performance benchmarks. New architecture +rollouts, followed by industry-wide software support, take a year or so to +fully saturate. +At the time of publication, there are some concrete details about Rubin. +The Rubin architecture uses HBM4, an upgrade from the HBM3 and +HBM3e that has powered the last few generations of GPUs. Inference + + +82 Chapter 3: Hardware +tasks like LLM decode that are bound on memory bandwidth will benefi t +from this higher-throughput VRAM. +Rubin also introduces the new CPX, a separate chip that’s built for com- +pute-bound tasks like LLM prefi ll. The CPX will be part of NVIDIA’s rack- +scale systems for high-volume inference. +After Rubin, NVIDIA will release Feynman in 2028. Few details are known +about Feynman, but it is likely to support larger and more powerful chips +with a faster memory architecture. + 3.2.5 Grace and Vera CPUs +NVIDIA also offers its own ARM-based CPUs, which are integrated with +their GPUs on superchips like the GH200 and GB200. The “G” stands for +“Grace,” as in Grace Hopper (to match Hopper GPUs). +NVIDIA Grace GPUs have overall strong compute performance, but what +matters for inference is that they have a much higher-bandwidth connection +between the CPU and GPU. Grace CPUs use NVIDIA NVLink Chip to Chip for +up to 900 GB/s bi-directional bandwidth between the CPU and GPU memory. +Figure 3.3: Grace CPUs have a higher-bandwidth interconnect to the GPU than +standard CPUs do. + + +3.3  Instances  83 +This CPU-to-GPU interconnect is several times faster than PCIe or other +standard connections between CPUs and GPUs. +Some inference setups call for offloading important information like LoRA +fine-tune weights and KV caches from previous inference calls to CPU +memory, which is far larger than GPU memory. With Grace CPUs, this +information can be retrieved much faster. +For the Rubin architecture, the Vera CPU (named for Vera Rubin) replaces +the Grace CPU that was used on Hopper and Blackwell systems. +3.3  +Instances +The atomic unit of GPU allocation on the cloud is an instance. An instance +is a virtual machine that includes: +• GPUs (device): One or more GPUs for inference tasks. +• CPUs (host): General-purpose compute for any tasks that don’t run on +the GPU. +• Memory (host memory): General-purpose memory for CPU operations +(traditional RAM). +• Storage: Disk memory for loading and storing large files. +• Networking: Physical network connections to the datacenter and even- +tually the public internet. +• Interconnect: Physical GPU-to-GPU and node-to-node connections +for running on multiple GPUs at once. +Not all GPUs, and not all instances, are created equal. Depending on +your cloud provider, instances vary in compute, memory, storage, and +interconnect. While NVIDIA offers its own reference architectures, each +cloud provider ultimately builds out systems according to their own pref- +erences. +Even the GPU itself can differ from instance to instance. For example, +NVIDIA A100 GPUs have two form factors: PCIe and SXM. But most +inference on A100 runs on SXM GPUs as they have five percent higher +memory bandwidth than the PCIe variant. + + +84  Chapter 3: Hardware +When provisioning instances, it’s essential to understand exactly what +you’re getting. Any component of the instance, not just the GPU, could +present a bottleneck or failure point during inference. +3.3.1  Multi-GPU Instances +Often, a model is too big to run on a single GPU, or inference engineers +want to use multiple GPUs together to improve performance. It’s common +to need two, four, eight, or even more GPUs to run large models like +DeepSeek or demanding modalities like video generation. +The standard unit for GPUs is a node, which contains eight individual +GPUs. For example, a B200 node contains eight B200 GPUs. +These GPUs are connected together via two systems: +• NVLink: A one-to-one communication layer between GPUs, up to 1800 +GB/s on Blackwell and 900 GB/s on Hopper. +• NVSwitch: An all-to-all communication layer on top of NVLink for coor- +dination among all GPUs in a node. +These high-bandwidth interconnects make it possible to spread inference +on a single model across multiple GPUs, up to a full 8-GPU node. +But sometimes one node isn’t enough. Running extremely demanding +inference workloads on more than eight GPUs requires a high-bandwidth +interconnect between nodes. +The standard in node-to-node interconnect for NVIDIA GPUs is InfiniBand. +InfiniBand competes with networking technologies like Ethernet, and in +2019 NVIDIA bought Mellanox, which manufactures InfiniBand. +InfiniBand is much slower than NVLink, with specs up to 400 Gb/s per +Network Interface Controller (NIC). But it’s the fastest node-to-node inter- +connect on the market – Ethernet maxes out at 100 Gb/s per NIC. + + +3.3.1 Multi-GPU Instances 85 +Figure 3.4: NVLink, NVSwitch, and Infi niband work together to enable GPU-to-GPU +communication. +Not every cloud provider uses Infi niBand. Some offer their own intercon- +nects, while others have Infi niBand on some GPUs but not all. When +provisioning GPUs, double-check what interconnect is provided and what +bandwidth that interconnect delivers. +In addition to Infi niBand, NVIDIA offers a high-end solution for NVLink +connections among more than eight GPUs. Their NVL72 GB200 system +combines 72 Blackwell GPUs and 36 Grace CPUs on a full-rack system. +These systems provide massive throughput for serving the world’s largest +models with intense traffic. +The NVIDIA Vera Rubin NVL 144 CPX is the next generation of this mas- +sive system, with updated Vera CPUs and Rubin GPUs alongside the +new Rubin CPX. + + +86 Chapter 3: Hardware +When working with multi-GPU and multi-node systems, keep relative +bandwidths in mind. When an interconnect like NVLink is an order of mag- +nitude faster than Infi niBand, it can handle far more data before becom- +ing a bottleneck. Parallelism and disaggregation techniques discussed in +sections 5.4 and 5.5 navigate this topology to deliver performant inference +across multiple GPUs. + 3.3.2 Multi-Instance GPUs +Sometimes, inference engineers run into the opposite problem: the GPU +is too big for the model. +Using newer high-performance architectures like Hopper and Blackwell +requires GPUs like the H100 or B200, with relatively large compute and +memory allocations. For models with a couple of billion parameters or +fewer, it’s hard to utilize these GPUs effectively. Even with large batch +sizes, valuable GPU resources are wasted. +Rather than running small models on older, lower-performance GPUs, +there’s a way to run these lightweight workloads on fractions of newer, +high-performance GPUs. +Multi-instance GPU (MIG) is a hardware-level capability in larger GPUs +including the A100, H100, H200, and B200. These GPUs can be split into +as many as seven pieces. These fractional GPUs also receive a slice of +CPU, RAM, storage, and other resources required to form an instance. +Figure 3.5: An H100 has eight memory slices and seven compute slices for +assembling multi-instance GPU instances. + + +3.4  Other Datacenter Accelerator Options  87 +For example, an H100 MIG with three slices has about 3/7 of the available +compute and can access as much as half of the total VRAM, or 40 GB. +It also includes about half of the CPU cores, CPU memory, storage, and +network bandwidth allocated to the underlying H100 to form an instance. +Software engineering generally works in multiples of two, so seeing +seven compute slices may appear strange. Compute slices are made up +of Streaming Multiprocessors in a GPU. GPUs generally do not have a +clean multiple-of-two SM count, for example, an SXM H100 GPU has 132 +SMs. Accordingly, seven evenly-sized compute slices are created, and +the leftover SMs are left idle. +For small models like Orpheus TTS, a 3B parameter model, using two MIG +instances is sometimes a more efficient use of resources than allocating +the full GPU to a single instance. +3.4  +Other Datacenter Accelerator Options +NVIDIA’s leadership in the AI hardware market has made it the world’s +most valuable company. But it is far from the only company to make +hardware that can run AI inference. +From fellow industry giants like Amazon and Google to a massive crop of +startups, competitors are pouring billions of dollars into developing and +manufacturing alternatives to NVIDIA GPUs. +While this book focuses on optimizing inference on NVIDIA GPUs, here’s +a short list of the other notable hardware options for running model infer- +ence. +Company +Stage +Flagship Product +AMD +Public +MI350 GPU: A datacenter GPU with competitive +specs on AMD’s own software stack. +AWS +Public +Inferentia and Trainium: A pair of chips +purpose-built for inference and training, +respectively, on AWS. +Cerebras +Startup +WSE-3: A wafer-scale chip with extremely +high memory bandwidth to remove decode +bottlenecks. + + +88  Chapter 3: Hardware +Company +Stage +Flagship Product +Etched +Startup +Sohu: An Application-Specific Integrated Circuit +(ASIC) for the transformer architecture. +Furiosa +Startup +RNGD: A power-efficient accelerator designed for +tensor contraction operations. +Google +Public +TPU: A Tensor Processing Unit is an AI-specific +ASIC built for inference and training. +Groq +Startup +LPU: A composable language processing unit +relies on SRAM for high memory bandwidth +Qualcomm +Public +Cloud AI 100 Ultra: A full-sized GPU composed +of multiple power-efficient mobile GPUs. +Sambanova +Startup +RDU: A Reconfigurable Dataflow Unit with large +memory allocation for trillion-parameter models. +GPUs are fairly general-purpose accelerators. Every hardware company +competing to win inference workloads from NVIDIA is betting on a specific +edge where their product can win: +• Memory bandwidth: Startups like Cerebras and Groq achieve high +token per second scores for LLMs by accelerating decode on ultra- +high-bandwidth memory. +• Power efficiency: Companies like Furiosa and Qualcomm design +chips for lower power consumption, which leads to cheaper operating +costs. +• Platform integration: Enterprises like Amazon and Google build deep +integrations with their cloud service platforms and proprietary closed +models. +While each of these accelerator options have their winning use cases, +they share common challenges: +• Software: Without CUDA, hardware providers have to rebuild the entire +inference stack for their accelerators. +• Manufacturing: Companies need to assemble the most complex object +humankind has ever created. +• Distribution: After manufacturing a chip, providers need it installed and +brought online to put capacity on the market. + + +3.5  Local Inference  89 +Competition accelerates innovation. A market with more hardware options +in the datacenter is good for every inference engineer. Competition in this +space will only grow more robust as inference workloads become more +valuable, as will exploration of options outside the datacenter such as +local inference. +3.5  +Local Inference +Local inference, also called edge inference, client-side inference, or on-de- +vice inference, means running AI model inference directly on the end +user’s device rather than on a centralized server. +Client-side inference has four massive advantages over server-side infer- +ence: +• Zero network latency: There’s no communication overhead, saving +tens or even hundreds of milliseconds. +• Independence: There’s no dependency on internet connection or +impact from high server traffic or downtime. +• Improved privacy: The end user’s data never leaves their device. +• Cost: Datacenter GPUs are expensive, while edge inference is free for +the developer, unlocking new business models. +Local inference sounds perfect in theory. In practice, there’s a reason +most inference happens in the cloud. There are four weaknesses to local +inference that limit its applications: +• Hardware capabilities: Even high-end prosumer desktops offer a frac- +tion of the speed and power of datacenter GPUs. +• Thermal constraints: Local devices have worse cooling than datacen- +ters, further limiting their speed and power. +• Fragmented support matrix: Endless combinations of hardware and +software make standardization challenging. +• Battery life: Inference is a demanding workload that quickly drains the +batteries of laptops and smartphones. +When building with local inference, it’s important to keep your audience +in mind. An AI enthusiast may have the latest and greatest phone and a + + +90  Chapter 3: Hardware +powerful computer, but a median user is more likely to have older devices +with less powerful components. +Local inference is turning the corner from experimentation to production, +with a strong ecosystem across hardware and software and a vibrant +community closely affiliated with the world of open models. +3.5.1  Desktop Inference +The classic local device is a workstation or gaming PC equipped with one +or two high-end consumer GPUs from NVIDIA or AMD. While researchers +and enthusiasts do use these setups, they’re a small portion of the desktop +inference market. +Increasingly, Apple is the leader in the desktop inference market. Apple’s +custom M-series CPUs and GPUs draw from a single unified memory, +giving inference on GPUs access to far more memory, albeit at slower +speeds. +The highest-end options from Apple and NVIDIA currently available on +the market illustrate the tradeoff between memory capacity and speed. +Hardware +NVIDIA RTX 5090 +Apple M3 Ultra +Memory +32 GB +512 GB +Bandwidth +1,792 GB/s +819 GB/s +Cost (full computer) +$5,000 +$10,000 +This trend continues through midrange hardware at more reasonable price +points. Low-end computers like Chromebooks are not equipped to run any +meaningful local inference. +The enthusiast-led open-source ecosystem focuses on running frontier +open models on desktops and laptops. Today, running aggressively quan- +tized 100B+ parameter models on high-end personal hardware is possible +using tools like Ollama and llama.cpp. +The increased popularity of Mixture of Experts is also a tailwind for desk- +top inference. These models have fewer active parameters, meaning that + + +3.5.2  Mobile Inference  91 +an individual user’s single request only touches a fraction of the model’s +total weights. +Image generation is also quite popular on personal computers, especially +via ComfyUI, a tool for assembling multiple image model components +together into a single pipeline. +Smaller language models and other modalities like speech are also feasi- +ble to run on midrange computers, and the industry is rapidly developing +browser inference libraries like WebLLM and other cross-platform stan- +dards to bring these capabilities from early adopters to the mainstream. +3.5.2  Mobile Inference +Local inference on mobile devices represents the majority of on-device +workloads today. Both major operating systems offer tooling for developers +to add edge inference to applications: +• Android: Google’s AI Edge SDK and ML Kit GenAI APIs interface with +Gemini Nano and OSS Gemma models. +• iOS: Apple’s Foundation Models and Core ML frameworks provide APIs +for models across modalities. +Mobile devices have extremely limited hardware capabilities and bat- +tery capacities, making inference even more challenging. Even high-end +phones struggle to run models with more than one or two billion param- +eters. +Still, some modalities are well-suited for edge inference on phones. For +example, transcription and speech synthesis models are latency-sensitive, +and some models are small enough to run in real time on modern phones. +Other discrete tasks, like translation, can be handled on edge devices by +small fine-tuned models. +Like any other software, the future of inference isn’t local or cloud, it’s +both working together to power fast and seamless user experiences. +Small models and quick queries will run on end-user devices, while more +demanding workloads will remain on datacenter GPUs in the cloud. + + + diff --git a/library/Inference Engineering/Chapter-06-Software/Chapter-06-source-text.md b/library/Inference Engineering/Chapter-06-Software/Chapter-06-source-text.md new file mode 100644 index 0000000..041cfd8 --- /dev/null +++ b/library/Inference Engineering/Chapter-06-Software/Chapter-06-source-text.md @@ -0,0 +1,688 @@ +# Inference Engineering — Chapter-06: Software +- **Source**: /library/Inference Engineering/source-file.pdf +- **PDF pages**: 95–118 +- **Pages without text**: 95, 96, 118 + +--- + + +CHAPTER 4 +Software + + + + + +Software  95 + +Software +NVIDIA’s market dominance in the inference space is in no small part +due to the robust and mature software ecosystem around its hardware. +Hardware iteration cycles are slow. Best-in-class hardware companies +like Apple and NVIDIA release new architectures and generations at most +yearly, with two-year release cycles being more common. +But software iteration is fast. Often, to run a newly released open model on +day zero, you need to install a nightly build or other prerelease version of +each of your software dependencies just to get support for the new model. +Software’s fast iteration cycle and lower barrier to entry dramatically +expands the landscape of inference engineering. While hardware centers +on NVIDIA and a few competitors, there are countless companies building +software at various levels of the inference stack. +For inference engineers, these are some key players: +• NVIDIA: Invests heavily in its own sometimes-proprietary software eco- +system, from CUDA up to Dynamo. +• Hugging Face: Maintains a model registry for all open models plus +transformers and diffusers. +• The Linux Foundation: Maintains hardware-agnostic projects like +PyTorch and vLLM. +• LMSYS Org: Develops essential tools for inference and evaluation, +most notably SGLang. +There are thousands more companies, universities, and research institu- +tions making essential open-source contributions to inference. +The software space is too big and too fast to exhaustively document in +any book, much less in a single chapter. Instead, this chapter presents +foundational technologies with long-term relevance. +Throughout the chapter, technologies are presented with increasing levels +of abstraction: + + +96  Chapter 4: Software +• CUDA: Direct communication to the GPU for explicit control over com- +putations and memory (section 4.1). +• Deep learning frameworks: Abstractions over CUDA for training, +exporting, and running neural networks in Python (section 4.2). +• Inference engines: Highly configurable PyTorch-backed inference for +common architectures (section 4.3). +• NVIDIA Dynamo: Sits on top of inference engines to power large-scale +deployments (section 4.4). +Most inference engineering today happens at the higher levels of abstrac- +tion, configuring and deploying inference engines and orchestrating infer- +ence across multiple GPUs. No matter what level of the stack you work +at, it’s essential to have a strong mental model for the adjacent levels of +abstraction to guide your work. +4.1  +CUDA +CUDA is how you write code to run on NVIDIA GPUs. +More formally, CUDA is NVIDIA’s proprietary computing platform and pro- +gramming model for executing parallel tasks on GPUs. Both “platform” and +“programming model” are broad definitions; it’s easier to look at CUDA in +its component parts: +• CUDA kernel: A user-defined function that executes parallelized code +on the GPU. +• CUDA graph: A directed acyclic graph (DAG) of kernels and other GPU +operations for optimizing repeated workflows. +• CUDA driver: A low-level interface between the application and the +GPU hardware to manage memory and execution. +• CUDA runtime: A developer-facing API for launching kernels and man- +aging memory. +CUDA – which stands for Compute Unified Device Architecture, though +the acronym is rarely expanded today – is the foundation for the entire +generative AI ecosystem on NVIDIA GPUs. + + +4.1 CUDA 97 +CUDA is not a programming language. Instead, CUDA programs are spec- +ifi ed in a programming language, most often C++, then compiled into +separate CPU and GPU code by a compiler like nvcc. +A kernel is just a function that does some parallel computation. Whenever +you see the phrase “CUDA kernel” you can replace it with “a piece of code +written for NVIDIA GPUs.” +For a “Hello, World!” kernel, consider these six lines of C++ code which +take in an array of length n and double each element in the array. +Figure 4.1: Example CUDA kernel doubles each value in an array. +Ordinarily, on a CPU, this function would run in linear time, with each ele- +ment in the array being doubled sequentially. But on a GPU, thousands +of elements can be processed simultaneously, making this function much +more efficient. +Writing CUDA kernels shifts inference engineering from thinking about +algorithms to thinking about implementations. For example, the traditional +attention algorithm that is central to generative AI can be expressed in +a few dozen lines of code. However, FlashAttention, which is the same +mathematical operation, takes tens of thousands of lines of code to +implement the algorithm in a more memory-efficient manner for a spe- +cifi c GPU. + + +98  Chapter 4: Software +4.1.1  +CUDA Kernels for Inference +Writing CUDA kernels does not mean building from scratch. +The prior art to build upon predates CUDA by decades. BLAS (Basic +Linear Algebra Subprograms), first implemented for Fortran in the 1970s, +is a specification for common linear algebra operations from dot products +to matrix multiplication. +cuBLAS, the CUDA implementation for BLAS, brings this specification to +CUDA in the form of pre-built kernels for essential linear algebra opera- +tions. Similarly, cuDNN (CUDA Deep Neural Network) provides primitives +for neural networks. +Within BLAS, the most frequently used operation in inference is GEMM +(General Matrix-Matrix Multiplication). Every linear layer in a model uses +matrix multiplication, and cuBLAS provides a strong starting point. +But you aren’t limited to the cuBLAS implementation. As operations like +GEMM are so essential for inference, you may need more fine-grained +control. You might write different GEMM kernels for matrices of different +shapes or to better run on specific GPU architectures. +CUTLASS is a template library that provides building blocks for writing +high-performance kernels. For example, FlashAttention 3 uses CUTLASS. +CuTe, another template library, introduces abstractions for tiled tensor +operations for recent architectures. With tools like CUTLASS and CuTe, +kernels can be written at a higher level of abstraction while retaining strong +performance. +Another source for kernels is FlashInfer, a library with high-performance +implementations of kernels for LLM inference, including many optimized +attention kernels and fused sampling functions. +4.1.2  +CUDA Kernel Selection +Most inference engineers will never need to write their own kernels. How- +ever, kernel selection – choosing the best kernel from a range of options +– is an important part of inference optimization. + + +4.1.2  CUDA Kernel Selection  99 +Kernel implementations are highly specialized. CUDA exposes low-level +APIs for memory management and parallel computing, and kernel engi- +neers make implementation decisions based on the exact specifications +of the hardware they’re writing for. +It’s important to understand how closely tied kernel implementations are +to specific hardware details. Kernels often have hard-coded values based +on the memory bandwidth or the number or layout of Tensor Cores on a +given GPU. +A kernel written for an H100 will likely fail to take advantage of the archi- +tecture and extra memory of a B200, while a kernel written for that B200 +could be backwards incompatible with the previous-generation Hopper +architecture. With each generation of GPUs, porting handwritten kernels to +run optimally on the new architecture takes substantial engineering work. +Most kernel selection is automatic. Deep learning frameworks and infer- +ence engines have pre-configured kernels for various architectures, while +PyTorch and TensorRT-LLM include automatic kernel selection in their +compilation steps. +However, you might manually choose a few kernels for essential algo- +rithms to speed up inference. +For example, most production-ready GEMM kernels come from cuBLAS. +But when the DeepSeek lab released their updated version of Deep- +Seek-V3, they also released DeepGEMM, which provides more efficient +GEMM kernels for running matrix multiplication in FP8 on the Hopper +GPU architecture. +Manual kernel selection lets you insert a DeepGEMM kernel as a plugin +to speed up a specific step in inference, like multiplying two matrices of +precise dimensions. Just be careful to ensure compatibility. For example, if +you were to upgrade to a B200 GPU, you’d have to either swap the kernel +back, wait for Blackwell support in DeepGEMM (now supported as of the +time of publication), or port the kernel yourself. + + +100  Chapter 4: Software +4.1.3  +Reducing Memory Accesses with Kernel Fusion +Running two different kernels back-to-back on the same data results in +wasted reads from and writes to memory. +As a simple example, imagine two kernels: multiply_by_2 and +multiply_by_3. If these kernels were run back-to-back, the sequence +would be: +1. Read the input vector [1, 2, 3] from memory. +2. Run multiply_by_2 on the vector. +3. Save the output vector [2, 4, 6] to memory. +4. Read the new input vector [2, 4, 6] from memory. +5. Run multiply_by_3 on the vector. +6. Save the final output vector [6, 12, 18] to memory. +The inefficiency is clear: steps three and four comprise an unnecessary +round trip to memory. During decode, the bandwidth-bound phase of LLM +inference, an inference engine can’t afford unnecessary reads from or +writes to memory. +Kernel fusion is the process of taking two or more kernels and re-imple- +menting them into a single kernel that handles both operations. In this +example, the fused kernel would be multiply_by_6 and the new order +of operations would be: +1. Read the input vector [1, 2, 3] from memory. +2. Run multiply_by_6 on the vector. +3. Save the final output vector [6, 12, 18] to memory. +In practice, kernel fusion is far more complicated – functions are more +complex, and data overlap isn’t as clean. But there are common patterns +in kernel fusion for inference, like combining matrix multiplication, bias +adding, and activation. + + +4.2 Deep Learning Frameworks and Libraries 101 +Figure 4.2: Kernel fusion reduces reads and writes between memory and compute +within a GPU. +Kernel fusion can be an automatic or a manual process. Compilers can +identify straightforward fusion opportunities and automatically create fused +kernels. But more sophisticated algorithms, like FlashAttention, require +handwritten fused kernels, which are used via plugins during inference. + 4.2 +Deep Learning Frameworks and Libraries +Deep learning frameworks and libraries are the bridge between working +directly in CUDA and using off-the-shelf inference engines like vLLM. +These libraries are used in both training and inference. +Over the past few years, PyTorch has emerged as the clear leader at this +level of the stack. There are two other frameworks that, for concision, I +will only mention briefl y: +• TensorFlow: An end-to-end machine learning platform officially sup- +ported by Google, TensorFlow was prominent in the 2010s ML era but +has fallen out of favor today. +• JAX: A research project unofficially associated with Google, JAX pres- +ents a simpler interface without as many legacy features and opera- +tions. However, as the documentation warns, expect sharp edges. + + +102 Chapter 4: Software +The remainder of this section focuses on PyTorch and the technologies +around and above it in the stack. + 4.2.1 PyTorch +PyTorch is a Python package for describing tensor operations. Originally +created at Meta and now a part of the Linux Foundation, PyTorch is the +industry standard technology underlying both training and inference for +generative AI models. +Personally, I’ve been a Python programmer for my entire career, and I +fi nd writing low-level C++ difficult. With PyTorch, I can write highly per- +formant inference code in Python for both CPUs and GPUs, but I always +have the option to dip down into CUDA when necessary by plugging in +specifi c kernels. +PyTorch can train any kind of neural network. The PyTorch documentation +shows a basic neural network example: +Figure 4.3: A basic neural network in PyTorch, adapted from the PyTorch +documentation. +While this is a very simple example, you may recognize the linear layers +and ReLU activation functions discussed in chapter 2 as key pieces of +neural networks. + + +4.2.2  Model File Formats  103 +PyTorch automatically computes gradients for any differentiable function +via its autograd module. This is what makes PyTorch so powerful for +training – you define a computation graph, and you get a gradient to train +against. +But PyTorch is special because it isn’t just great for training, it’s also +powerful for inference. PyTorch balances built-in functions and automatic +performance optimizations with manual control where you need it. +The step that transforms a model from training to inference is compila- +tion. PyTorch compilation (torch.compile) targets a specific GPU and +performs automatic kernel selection and kernel fusion to ensure optimal +performance. +torch.compile can’t fuse plugin kernels like DeepGEMM, FlashAtten- +tion, or custom kernels. This limits its utility for LLM inference, where most +kernels are custom. However, PyTorch compilation is useful for optimizing +less common model architectures and compiling long sequences of light- +weight kernels. When you are optimizing a model that has a custom or +rare architecture, you may have to rewrite functions to be more abstract +– especially with respect to Python-specific language features – for com- +pilation to succeed. +PyTorch alone is a powerful and flexible tool for building high-performance +inference services. But there is a rich ecosystem on top of PyTorch that +makes it faster to implement, compile, and execute optimized code for +common model architectures. +4.2.2  Model File Formats +The dominant file format for serializing model weights is safetensors, cre- +ated by Hugging Face. +Safetensors is a replacement for generic formats like bin designed specifi- +cally for holding model weights. The safety in safetensors comes from the +fact that unlike a general format that can execute arbitrary Python code +during deserialization, safetensors only hold tensor data, not executable +code. + + +104  Chapter 4: Software +Generative AI models have hundreds of gigabytes of weights. These +weights are split across dozens of safetensors files. The safetensors for- +mat uses memory mapping to ensure that the files are loadable without +allocating the full memory, which makes loading model weights faster +and safer. +Another leading format, ONNX (Open Neural Network Exchange), stores +weights along with an execution graph for the model. Where the safeten- +sors format separates the weights from the architecture, ONNX bundles +them together. +ONNX files are highly portable. With deep integration into PyTorch and +support for multiple hardware options, ONNX is a great alternative to +safetensors for when you want to store model graphs, not just weights. +4.2.3  ONNX Runtime and TensorRT +ONNX Runtime and TensorRT are high-performance inference runtimes. +PyTorch models can be exported to the ONNX format, which ONNX Run- +time can execute directly or TensorRT can compile into a highly optimized +engine. +ONNX Runtime +TensorRT +Open source, associated with the +Linux Foundation +Mix of proprietary and open +components, built by NVIDIA +First-class exporter in the PyTorch +ecosystem +Integrated with PyTorch via Torch- +TensorRT +Supports many types of GPUs +NVIDIA GPUs only +ONNX Runtime is an open community standard, while TensorRT is spe- +cific to NVIDIA GPUs. +The export process looks somewhat like Torch compilation. However, +these standards do not support every data structure, type, and operation +within PyTorch. The export process can identify these issues in PyTorch +code, but this gets tricky with more complex models. +An example is DeepSeek V3, which introduces Multi-Latent Attention +(MLA). MLA, as implemented in PyTorch, is difficult to export, but the + + +4.2.4  Transformers and Diffusers  105 +transformers architecture overall is simple enough that hand-fusing ker- +nels is feasible. +Today, it’s increasingly popular to skip directly from PyTorch to an inference +engine like vLLM or TensorRT-LLM for models that these engines support, +bypassing the intermediate representation step and exporting weights only +as safetensors. ONNX Runtime and TensorRT are still widely used – Ten- +sorRT especially for its strong out-of-the-box runtime for image and video +models – but the industry is bifurcating between the control of handwritten +PyTorch code or the convenience of prebuilt inference engines. +4.2.4  Transformers and Diffusers +The transformers and diffusers libraries by Hugging Face are +built on PyTorch but are not designed to run large-scale production +inference. Instead, these libraries offer reference implementations of +models for inference engineers to learn from and adapt. +While these libraries are toolboxes for tinkering, they do include essential +information about models and useful utilities for building model inference +servers. The config.json file that ships with models implemented for +transformers and diffusers contains essential information, and the +libraries’ utilities for Hugging Face operations like downloading model +weights are widely used. +You’ll find transformers or diffusers sample code in the model card +for most popular open models on Hugging Face. This sample code is +great for understanding the exact input and output spec of a model or for +running local inference and notebooks. But for production, you’ll want to +either write and compile PyTorch code directly or use a production-ready +inference engine. +4.3  +Inference Engines +There are three competitive inference engines on the market: vLLM, +SGLang, and TensorRT-LLM. +These frameworks offer good out-of-the-box performance for LLMs and +other modalities with similar architectures (chapter 6). + + +106  Chapter 4: Software +In late 2025, vLLM and SGLang also began supporting some image and +video generation models via vLLM Omni and SGLang Diffusion, respec- +tively. TensorRT-LLM does not support image or video generation models. +Inference engineers can also use TensorRT or PyTorch directly to run +these models (section 6.5). +Inference engines are powerful because they are configurable. Working +with pre-optimized components at a higher level of abstraction, inference +engineers can spend their time testing combinations of techniques rather +than repeating routine implementations. +At a very high level, vLLM and SGLang are more general tools that are +easier to adopt and have day zero support for more models, while Ten- +sorRT-LLM has a steeper learning curve but usually achieves the best +performance. +Engine +vLLM +SGLang +TensorRT-LLM +Performance +Good +Good +Best +Ease of use +Easy +Easy +Hard +Model support +Most +Most +Some +Hardware +GPU, TPU +NVIDIA, AMD +NVIDIA only +License +Apache 2.0 +Apache 2.0 +Apache 2.0 +Each framework runs out of the box with core features like continuous +batching and supports the main performance optimization techniques – +post-training quantization, speculative decoding, prefix caching, parallel- +ism, disaggregation. +At Baseten, we use all three frameworks, though we use TensorRT-LLM +the most. Inference engineers should be familiar with all three and select +on a deployment-by-deployment basis. +4.3.1  vLLM +vLLM has the largest market share among inference engines. GitHub stars +are a rough measure for popularity, but at the time of publication vLLM has +twice as many stars as SGLang and TensorRT-LLM combined. + + +4.3.1 vLLM 107 +First released in the summer of 2023, vLLM is the oldest of these inference +engines by a few months. Originally created at UC Berkeley, vLLM is now +hosted by The PyTorch Project within The Linux Foundation. +vLLM’s best selling point is its broad support. It supports the most hard- +ware options – NVIDIA, AMD, and Intel GPUs along with Google TPUs +– as well as the most models and architectures. Just about every open +LLM out there integrates with vLLM from day zero. vLLM also supports +multimodal inference via vLLM Omni, which extends the engine to support +image, audio, and video inputs and outputs. +One of the core principles of inference engineering is that the more +constraints you can introduce, the better performance you can achieve. +vLLM’s broad platform can achieve impressive performance results when +properly confi gured, but in my experience it falls short of the highest-end +performance possible with narrow frameworks like TensorRT-LLM. +vLLM’s developer experience is built around the vllm serve command, +with server confi guration passed in as fl ags. +Figure 4.4: vLLM inference example on eight GPUs. +vLLM is pip-installable and provides official Docker images with pre-bun- +dled dependencies and support for various hardware architectures. +You should use vLLM when: +• You want to quickly stand up a model server that will offer solid perfor- +mance out of the box for almost any open model. +• You want to run an “Omni” model with multiple input and output modal- +ities. + + +108 Chapter 4: Software +• You are using a smaller GPU or older architecture where TensorRT-LLM +offers few performance benefi ts. + 4.3.2 SGLang +SGLang is the other major community-driven fast inference framework. +First released in December 2023, SGLang has risen to prominence along- +side Chinese open models like DeepSeek and Qwen and is the engine of +choice for inference at xAI. +SGLang’s unique angle on the problem of model serving is expressed in +its developer experience, which pairs a fast backend runtime with a fl exi- +ble frontend language. In practice, that means you can choose individual +components of your engine for deep customization without needing to +rewrite everything else from scratch. +SGLang supports both NVIDIA and AMD GPUs, and has strong day-zero +support for a wide range of models. SGLang works closely with labs like +DeepSeek, Qwen, Kimi, and Z AI to release optimized implementations of +new architectural features like DeepSeek’s Multi-Latent Attention. +SGLang has invested heavily in supporting large-scale deployments of +MoE LLMs, specifi cally multi-node deployments on systems like GB200 +NVL72 for high throughput. These systems offer extremely cost-efficient +inference for large models with signifi cant traffic. +SGLang’s +developer +experience +is +built +around +the +sglang.launch_server command, with server confi guration passed +in as fl ags. +Figure 4.5: SGLang inference example on eight GPUs. + + +4.3.3  TensorRT-LLM  109 +SGLang also supports image and video generation model inference via +SGLang Diffusion. +SGLang Diffusion introduces a pipeline abstraction which orchestrates +a number of stages. This flexible approach maps closely to the architec- +ture of image and video generation models. For performance, SGLang +Diffusion adds support for various diffusion-specific parallelism methods +and re-uses the scheduler and optimized kernels from the main SGLang +package. +You should use SGLang when: +• You want excellent out-of-the-box throughput with decent latency on +large MoE models like DeepSeek and Kimi. +• You want the inference engine experience for image and video gener- +ation models. +• You want control and customization and are excited to participate in the +SGLang community. +4.3.3  TensorRT-LLM +TensorRT-LLM is NVIDIA’s open-source inference engine. Of the three +main options, TensorRT-LLM offers the highest performance and the most +flexibility to expert users. +A note on naming: There are two major versions of TensorRT-LLM. Only +the older version is actually related to TensorRT: +• TensorRT-LLM V0 (0.X.Y): Major versions starting with zero are a +plugin for NVIDIA TensorRT. +• TensorRT-LLM V1 (1.X.Y): Major versions starting with one are a +standalone package based on PyTorch with no dependency on Ten- +sorRT. +Originally, TensorRT-LLM built a TensorRT engine for serving lan- +guage models. With the modern PyTorch-based version, TensorRT-LLM +bypasses the intermediate representation of TensorRT and uses PyTorch +directly. + + +110 Chapter 4: Software +TensorRT-LLM V1 was released in the summer of 2025. Deployments of +the previous major version are still common – always be sure to check +which version you are using. +TensorRT-LLM achieves the best performance in large part because it has +access to kernels written by NVIDIA engineers, including some closed- +source kernels. These handwritten and manually fused kernels offer excel- +lent support for the latest hardware architectures like Hopper and Blackwell +and for NVIDIA-specifi c number formats like NVFP4. +TensorRT-LLM offers a robust implementation of in-fl ight batching (token- +level continuous batching), which helps with throughput. It also supports +just about every model performance optimization setting you could ask +for, including quantization, speculation algorithms, prefi x caching, chunked +prefi ll, fl exible parallelism, and disaggregation. +With V1, TensorRT-LLM introduces a developer experience that looks a +lot like vLLM and SGLang. However, in addition to fl ag arguments on the +trtllm-serve command, it expects a config.yaml fi le for deeper +customization. +Figure 4.6: TensorRT-LLM inference example on eight GPUs. +The best way to install TensorRT-LLM is by running it via one of NVIDIA’s +official Docker containers + + +4.4  NVIDIA Dynamo  111 +Use TensorRT-LLM when: +• You are running a well-supported model architecture on a Hopper or +later GPU. +• You are willing to do extra engineering work to get the best possible +performance. +• Optionally, you are planning to use NVIDIA Dynamo for serving and +want the most deeply integrated engine. +4.4  +NVIDIA Dynamo +NVIDIA Dynamo is a distributed system for model serving first announced +at NVIDIA GTC in March 2025. +Dynamo works with every inference engine – vLLM, SGLang, and Ten- +sorRT-LLM – as backends, with Dynamo itself providing an orchestration +layer for large-scale deployments. +Dynamo provides support for essential model performance techniques: +• KV cache re-use: Retaining KV information between requests and +routing requests based on prefix match. +• Disaggregation: Separating prefill and decode onto individually opti- +mized engines with independent scaling. +• Multi-node parallelism: Optionally using two or more nodes of GPUs +in a single replica for a model, usually with Expert Parallelism. +Each of these techniques will be detailed in chapter 5. As with the infer- +ence engines, there is a lot of work for inference engineers to do to con- +figure Dynamo for their use case and achieve maximum performance. +Dynamo’s thoughtful abstractions for distributed KV routing, disaggrega- +tion, and multi-node model parallelism provide high-performance aggre- +gation of information during runtime, allowing for real-time adjustments +to configuration as traffic fluctuates. For example, you can automatically +scale up and down prefill and decode workers with an SLA-based planner +operating on user-defined TTFT and TPS constraints. + + +112  Chapter 4: Software +As a general principle, the more scale you have, the more tools and tech- +niques there are available to you for inference optimization. +Dynamo is built for scale: big models, big traffic. It excels at serving foun- +dation models like the trillion-parameter Kimi family to a large number of +concurrent users. For smaller models, Dynamo can still offer moderate +performance improvements on large-scale deployments. +If you’re building an inference API for a built-from scratch foundation model +or serving an open model in a high-usage product, Dynamo is a great +choice. +But many deployments don’t need the additional complexity of Dynamo. +Unless you’re operating with enough volume for disaggregation and +KV-aware routing to matter, Dynamo will be unnecessary work and excess +overhead. In these cases, you can use inference engines directly. +Dynamo is the newest project covered in this chapter, and features are +still being built out. Dynamo is open source under the Apache 2.0 license. +The community around Dynamo is active, and the project welcomes con- +tributions with a public CI and support from NVIDIA engineers. +4.5  +Performance Benchmarking and Load Testing +Benchmarking is an essential part of model performance optimization. +Without precise, accurate performance benchmarks, there’s no way of +knowing if your optimizations are actually working. +A high-quality benchmark simulates real life as closely as possible. The +best benchmark is to shadow real-world production traffic onto the system +you are testing. Shadowing is the process of copying incoming requests +onto the test system so that you can benchmark its performance without +affecting the original request. +If you can’t shadow real usage, you’ll need to simulate it. LLM performance +is affected by a number of factors. When simulating traffic, you need to +match your expected production workload on multiple dimensions: + + +4.5.1  Performance Benchmarking Tooling  113 +• Sequence lengths: Time to first token and memory usage rely on the +input sequence length (ISL) and output sequence length (OSL), mean- +ing the number of tokens in the prompt and response. +• Volume and pattern of traffic: Batching and server load depend on +the number of concurrent requests. Jitter traffic to mimic real usage. +• Request contents: The actual prompt within each request affects per- +formance factors like cache hit rate and draft token acceptance. +• Input parameters: Settings like temperature and reasoning effort that +affect inference should be set to their anticipated production values. +Remember, optimization is about tradeoffs and constraints. If you’re max- +imizing benchmark performance against bad inputs, performance in pro- +duction won’t match expectations. +4.5.1  Performance Benchmarking Tooling +As a performance benchmark should closely reflect production traffic, +everyone’s benchmarking setup should look a bit different. But there are +a few common tools: +• SGLang Genai-bench: A CLI and dashboard by the SGLang team for +benchmarking models deployed with any inference framework. +• NVIDIA GenAI-Perf: A client-side tool by NVIDIA for measuring latency +and throughput on varied traffic. +• Locust: An open-source load-testing tool, not specific to generative AI +systems, that simulates as many as millions of simultaneous users. +Another great tool for benchmarking is open-source evals datasets – from +general evals like MMLU and gsm8k to domain-specific evals like SWE- +bench. +While the purpose of benchmarking work is to measure performance, not +model output quality, these eval datasets serve two purposes: acting as +a set of varied and realistic inputs, and spot checking that performance +optimizations haven’t impacted model output quality. +When possible, choose an eval dataset that matches the expected use +of your production system, like HumanEval when reducing latency for a +code completion system. + + +114 Chapter 4: Software + 4.5.2 Performance Benchmarking Tips +Along with being realistic, great benchmarks are also consistent. Make +sure your benchmarks send enough traffic to get a good read on perfor- +mance without being swayed by outliers. When in doubt, run a benchmark +multiple times and average the results. +Before you do any performance optimization work, start with a solid base- +line benchmark. As you test optimizations, keep a consistent confi guration +in your benchmarking setup, and test each optimization individually as well +as collectively to fully understand what is driving performance improve- +ment. In some cases, optimizations can work against each other, like trying +to run speculative decoding with large batch sizes. +The principle of changing one thing at a time applies to your benchmarking +confi guration as well. It’s common to need to test various traffic patterns or +sequence shapes, but as with any experiment only change one variable +at a time to ensure that you are getting clear results. + 4.5.3 Profi ling Performance +Profi ling is one click deeper than benchmarking. Where a benchmark +gives a single fi gure (e.g., the P90 TTFT is 350 ms), a profi ling tool shows +where each of those milliseconds was spent in the inference process. +Benchmarking tells you how your system is performing; profi ling tells you +why it’s performing that way. +Figure 4.7: A kernel profi ler shows you how long each operation within a kernel takes +to execute, revealing bottlenecks. +Most inference engineers won’t need to do profi ling as part of their daily +work. When using an already high-performance tool like the inference + + +4.5.3  Profiling Performance  115 +engine TensorRT-LLM, your workflow is a cycle of configuration and +benchmarking – profiling would be extraneous. +However, if you’re contributing to an inference framework like vLLM or +SGLang, writing your own inference service in PyTorch, or operating at +the cutting edge of a new modality like video generation, performance +profiling should be part of your toolkit. +The most popular profiling tools for inference are: +• PyTorch Profiler: An easy-to-use profiling library for capturing step-by- +step performance metrics (CPU time, GPU time, memory usage) during +inference. +• NVIDIA Nsight Systems (NSys): A featureful but complex tool for +GPU and CPU sampling and tracing that provides system-wide analy- +sis across multiple GPUs and their interconnects. +• NVIDIA Nsight Compute (NCU): A profiling utility and CLI for in-depth +analysis of individual CUDA kernels on both compute and memory +usage. +In addition, frameworks like TensorFlow and TensorRT ship with their +own built-in profilers. +Profilers are valuable because they give you granular information about +compute and memory usage, which guides your optimization work toward +improving the most expensive steps in your inference pipeline. +For example, using PyTorch Profiler you might find that activation functions +are taking an unusually long time due to excess memory reads, and figure +out how to write a fused kernel that runs activations alongside attention +to prevent the excess reads. Then, you would insert that new kernel into +your PyTorch code and re-run system-level benchmarks to see if you’ve +achieved your latency targets. +Together, profiling and benchmarking give you the information you need +to improve system performance and, eventually, the confidence to deploy +your optimizations in production. + + + diff --git a/library/Inference Engineering/Chapter-07-Techniques/Chapter-07-source-text.md b/library/Inference Engineering/Chapter-07-Techniques/Chapter-07-source-text.md new file mode 100644 index 0000000..9364360 --- /dev/null +++ b/library/Inference Engineering/Chapter-07-Techniques/Chapter-07-source-text.md @@ -0,0 +1,974 @@ +# Inference Engineering — Chapter-07: Techniques +- **Source**: /library/Inference Engineering/source-file.pdf +- **PDF pages**: 119–154 +- **Pages without text**: 119, 120, 154 + +--- + + +CHAPTER 5 +Techniques + + + + + +Techniques  119 + +Techniques +One of the coolest parts about working in inference engineering is that +unlike many industries where new academic research takes years or +decades to be adopted by industry, techniques from new papers are live +in production within months or even weeks. +There is a gap to cross between research and production, and some of +the most visible inference engineering work in the industry comes from +bridging that gap. +A core principle of inference engineering is that the more constraints you +can introduce in your inference system, the better performance you’ll +achieve. This principle continues to apply throughout this chapter, with +techniques like disaggregation, which allows you to constrain individual +engines to prefill and decode. +With these model performance techniques, there’s a new principle to keep +in mind: the more traffic you have, the more performance optimizations +you can make (while keeping unit economics reasonable). Higher model +parallelism across more GPUs, KV-aware routing, and dynamic disaggre- +gation only make sense when you have a large number of GPUs, often +multiple nodes, serving the same model with vertical scale and horizontal +replication. +Real-world traffic defies constraints. But with volume, you can adapt your +systems over time to match the changing nature of usage. Tuning the +parameters of inference engines, speculation algorithms, and model serv- +ers isn’t a one-time task. Instead, either through iterative deployments or +dynamic runtime adjustments, you can continuously improve the perfor- +mance of your inference system. +Finding the right combination of techniques and configurations takes +patient experimentation. I remember an internal hackathon during which +one of Baseten’s inference engineers was working on an autocomplete +model for code and ended up trying 77 different configurations via a hand- +written script before finding a non-obvious solution that doubled TPS for +a customer’s model. + + +120  Chapter 5: Techniques +To make inference optimization even more complex, sometimes tech- +niques are symbiotic and sometimes they are incompatible. For example, +quantizing the KV cache alleviates a bottleneck in disaggregation, but +increasing batch sizing reduces the compute available for speculation. +An inference engineer’s goal is always to create a balanced set of optimi- +zations that delivers more than the sum of its parts. +This chapter introduces five key categories of applied research for infer- +ence acceleration: quantization, speculation, caching, parallelism, and +disaggregation. In each section, pay special attention to the recommended +circumstances for using each technique and the potential bottlenecks or +tradeoffs each introduces. +5.1  +Quantization +Quantization improves latency (both TTFT and TPS), increases system +throughput, and opens up headroom for other optimizations like disag- +gregation, speculation, and prefix caching to be even more effective. But +when it goes wrong, quantization can materially reduce the model’s output +quality. +Models are trained with weights, activations, and other components rep- +resented in a certain native number format. Usually, this is BF16 or FP16, +though 8-bit and 4-bit native precisions are becoming more popular in +training. +Post-training quantization works by changing those model weights and +other values from their native number format to a lower-precision format. +Cutting precision in half improves performance in both phases of inference: +• Prefill: Compute-bound prefill now runs on lower-precision Tensor +Cores with twice the FLOPS. +• Decode: Memory-bound decode now loads half as much data per +value, effectively doubling memory bandwidth. +Working with quantized data does introduce overhead, so it’s not linearly +twice as fast to go from 16 to 8 bits. In practice, quantization down a single +level of precision generally offers 30 to 50 percent better performance for +LLMs. + + +5.1.1  Number Formats  121 +The catch with quantization is that it runs the risk of reducing the model’s +output quality. Quantization has the potential to introduce precision errors +throughout the calculations that power inference. +Precision errors compound over time. Consider what happens when you +square and cube different precisions of Pi: +Pi precision +Pi squared +Pi cubed +3.14159 +9.869588 +31.006198 +3.14 +9.8596 +30.959144 +3 +9 +27 +Most of the work in quantization is around both preventing precision errors +and minimizing their impact on the final model output. +5.1.1  Number Formats +Quantization introduces a new collection of essential terms and abbrevia- +tions. The most important ones to know are the common number formats: +Name +Abbr +First architecture +64-bit Floating Point +FP64 +Fermi (2010) +32-bit Floating Point +FP32 +Kepler (2012) +16-bit Floating Point +FP16 +Pascal (2016) +Brain Floating Point 16 +BF16 +Ampere (2020) +8-bit Floating Point +FP8 +Hopper (2022) +Mixed-Precision FP8 +MXFP8 +Blackwell (2024) +8-bit Integer +INT8 +Pascal (2016) +6-bit Floating Point +FP6 +Blackwell (2024, experimental) +4-bit Floating Point +FP4 +Blackwell (2024) +Mixed-Precision FP4 +MXFP4 +Blackwell (2024) +NVIDIA FP4 +NVFP4 +Blackwell (2024, proprietary) +4-bit Integer +INT4 +Turing (2018) + + +122  Chapter 5: Techniques +The largest number format, FP64 or “double precision,” is only used for +high-precision scientific computing, not AI training or inference. FP32 is +sometimes used for training, but almost never for inference. FP6 is more +experimental at the time of publication, though AMD GPUs are rapidly +adopting the format. +That leaves 16, 8, and 4-bit precisions as the primary formats for inference. +Number formats have a: +• Precision: The number of bits used to express a single value in the +format. For example, FP16 uses 16 bits. +• Type: Whether these bits are interpreted to represent an integer (no +decimal) or a floating-point number (a decimal). +• Scale factor: A multiplier used to map values from a low-precision +format back to the higher-precision format. +Together, these attributes determine the two factors behind how well a +number format represents values used in inference: +• Dynamic range: The difference between the lowest and highest value +that can be represented in the format. +• Granularity: The number of parameters or other values that are quan- +tized along a single scale factor. +Dynamic range is essential to low-precision inference without quality loss. +16 bits can represent 65,536 distinct values, while 8 bits can only repre- +sent 256 different values. The dynamic range is the distribution of these +values – the difference between the smallest and largest available value. +Dynamic range explains why floating-point formats are better than integer +formats for inference. Floating-point formats have three properties: +• Sign: A single bit that represents whether the number is positive or +negative. +• Exponent: A set of bits that, taken together, represent an exponent +factor. +• Mantissa: A set of bits that, taken together, represent the base value +multiplied by two to the exponent. + + +5.1.1 Number Formats 123 +An FP8 number in a E4M3 data format means it has a 4-bit exponent and +a 3-bit mantissa, with the remaining bit for the sign. Integer formats only +have sign and value bits. +Figure 5.1: Floating-point number formats have both exponent and mantissa bits +along with the sign bit. +The exponent in fl oating-point numbers gives it a higher dynamic range, +meaning it can better express very large and very small numbers. This +is important because outlier values are signifi cant in inference, and fl oat- +ing-point number formats better represent outliers after quantization. +Within fl oating-point formats, there are multiple options at each precision, +like FP4, MXFP4, and NVFP4. These formats differ in granularity, or the +number of values that are quantized by a single scale factor. +Quantization can be applied at the: +• Tensor level: Calculate a single scale factor for the entire QKV tensor. +• Channel level: Calculate a different scale factor for each feature vector +within the tensor. +• Block level: Within each feature vector, divide the vector into blocks +of N values and calculate a scale factor for each block. + + +124  Chapter 5: Techniques +More granular quantization has a lower chance of smoothing over outliers, +preserving quality. However, more granularity introduces more overhead +for storing and applying scale factors. +MXFP8 and MXFP4, the new number formats supported by Blackwell, are +“microscaling” formats that compute a blockwise scale factor on every 32 +parameters, reducing the impact of these number formats’ lower dynamic +range. +NVFP4, a 4-bit format by NVIDIA, offers even higher granularity than the +MX formats with a block size of 16 and a secondary 32-bit global scale +factor to further combat the quality loss that 4-bit formats introduce. +The tradeoff to a microscaling format is that the small-block scale factor +must also be stored in memory, slightly reducing the performance gains +from quantization. Additionally, both the tensor and block scale factors +need to be applied, introducing a bit of compute overhead. Blackwell GPUs +offset this overhead via scale factor application in Tensor Cores. +While the focus of this book is on inference engineering in the data- +center, quantization is an essential topic for local and edge inference, +especially for large models. GGUF, a binary format for storing models, +is the most popular choice for distributing highly quantized models on +Hugging Face, with individual researchers and companies squeezing +huge models like DeepSeek onto consumer hardware like Apple com- +puters. +These quantization strategies combat quality loss through dynamic quan- +tization, where certain layers or other components of the model are left in +their original precision, while others are quantized to integers with as little +as one bit of precision. Dynamic formats represent their average preci- +sion, which is why you might see something like fine-tuning lab Unsloth’s +popular 1.58-bit quantization. +While these dynamic quantizations are impressive feats of engineering +and are great for local inference, inference engineers working on pro- +duction systems should stick with floating point number formats – integer +formats are not suitable for quality-sensitive workloads due to their lack +of dynamic range. + + +5.1.2  Quantization Approaches  125 +Instead, 8-bit floating-point formats (FP8, MXFP8) are generally the sweet +spot for improving performance without sacrificing quality. FP4 is prom- +ising, especially the NVFP4 format which introduces a higher level of +granularity for improved accuracy, but FP8 and MXFP8 provide the most +flexibility, especially when quantizing the KV cache. +5.1.2  +Quantization Approaches +The more parameters a model has, the less sensitive it is to quantization +as each individual parameter is less important. However, even for very +large models, it is essential to quantize carefully. +Quantization can happen during or after training: +• Quantization-aware training: Training weights and computing scale +factors together to ensure that the final converged weights are accurate +at a given precision. +• Post-training quantization: Converting finished model weights to a +new precision by computing scale factors and preserving accuracy via +calibration. +While some labs release models created with quantization-aware training, +like GPT-OSS in MXFP4 and Kimi K2 Thinking in INT4, inference engi- +neers working with open models only have the ability to perform post-train- +ing quantization as they are working with finished weights. +A leading tool for post-training quantization is NVIDIA TensorRT Model +Optimizer (ModelOpt), an open-source library that also supports pruning, +distillation, and sparsity. ModelOpt outputs are compatible with all infer- +ence engines (vLLM, SGLang, TensorRT-LLM). +After picking a precision, there are two decisions to make before doing +post-training quantization: +1. What parts of the model (weights, activations, KV cache, attention) +should be quantized? +2. What number format offers the appropriate dynamic range and granu- +larity? + + +126  Chapter 5: Techniques +These decisions turn quantization from a binary choice into a spectrum of +tradeoffs around performance and quality. +Components of a model have varying sensitivity to quantization. Reducing +the precision of more sensitive components runs a higher risk of quality +degradation. From least to most sensitive: +1. Weights: Specifically the linear layers are least sensitive to quantiza- +tion. +2. Activations: The intermediate output of activation functions are only +somewhat sensitive to quantization. Note that the activation functions +themselves are rarely quantized as they are such a tiny fraction of the +model’s weights. +3. KV cache: The cached values from the attention calculation are mod- +erately sensitive to quantization. +4. Attention: The attention layers of a model are highly sensitive to quan- +tization, especially equations like softmax. +Within each of these components, you can get more selective about quan- +tization. +Even in linear layers and activations, which are generally the least sensi- +tive to quantization thanks to their size, early and late layers like the input +and output layer of the neural network may be left in their original precision +as these layers are more sensitive. +While quantizing weights and activations helps performance directly, KV +cache quantization gives an additional boost to techniques like prefix cach- +ing and disaggregation. The KV cache is a valuable resource. Quantizing +it allows inference engines to store more of it in memory and read it more +quickly. +However, the KV cache for each token is used by each subsequent token. +This means precision errors introduced by quantization can compound +from token to token. +Compounding errors is exactly the reason why attention layers are the +riskiest to quantize. Not only is attention very sensitive to dynamic range, +but each attention calculation relies on the results of each previous atten- + + +5.1.2 Quantization Approaches 127 +tion calculation. Over a sequence of thousands of tokens, these errors +accumulate quickly. +All but the most aggressive quantization schemes run functions like soft- +max in their original precision. +Figure 5.2: Quantization risk is low for weights and activations, moderate for KV +cache, and high for attention. +A moderate approach to low-precision inference uses a format like FP8 +with high dynamic range – if possible, a microscaling format like MXFP8 – +to carefully quantize select linear layers, activations, and often KV cache +values. Even with these high dynamic range formats, components of the +attention layer are rarely quantized. + + +128  Chapter 5: Techniques +5.1.3  +Measuring Quality Impact +The standard for production-ready quantization is zero perceptible quality +loss. After quantizing a model, it’s essential to thoroughly test its output +quality versus the original precision. +There are three methods for checking model quality after quantization: +1. Perplexity: Calculate the perplexity score for the quantized model and +compare with the original. +2. Intelligence benchmarks: Run a standard intelligence benchmark like +MMLU or SWE-bench and compare to original scores. +3. Custom evals: Run a product-specific evaluation suite on the quantized +model and compare to the original weights. +In every case, you’re looking for a difference in scores that’s indistinguish- +able from noise. LLMs are non-deterministic, so scores vary slightly from +run to run. +The simplest check on quality is perplexity. Rather than asking the model +to generate output, perplexity gives the model expected output sequences +and calculates the likelihood of the model predicting those tokens. +A higher perplexity means a model is more “surprised” by the sequences +– not what you want from a model that’s supposed to predict tokens. After +quantization, you’re looking for a minimal increase in perplexity. +A more comprehensive quality check relies on a public intelligence bench- +mark or, better yet, a domain-specific eval that matches your expected +real-world usage. On evals, you’re looking for a minimal reduction in the +quality score. +The best way to get the full picture of the impact of quantization is to run +all three types of checks and make apples-to-apples comparisons to the +original model weights. +Remember that quantization is a scale, not a binary decision. You can +still get some performance improvements with lower risk of quality loss +by quantizing to FP8 instead of FP4, or by quantizing fewer components +of the model, like weights-only quantization. + + +5.2  Speculative Decoding  129 +If you’re working in a highly sensitive domain and can’t risk model quality, +no worries: every other technique in this chapter is lossless in terms of +quality. +5.2  +Speculative Decoding +The decode phase of LLM inference is an autoregressive process in which +tokens are generated one at a time. The bottleneck on decode is memory +bandwidth, with compute sitting idle at low to moderate batch sizes as +weights are read from memory. +Speculative decoding takes advantage of that spare compute to try to +generate multiple tokens per forward pass through the target model. If an +inference engine could generate two, three, or even more tokens for each +round-trip of weights through memory, it would generate far more tokens +per second. Speculative decoding only improves TPS/ITL, not TTFT. +There are multiple algorithms for speculative decoding. They share a com- +mon mechanism: +1. The speculator generates one or more draft tokens. +2. The target model, or the underlying model that you’re trying to accel- +erate, performs validation on these tokens to check if they match what +the model would have generated. +3. The target model accepts any valid draft tokens and generates an addi- +tional token itself, completing the forward pass. +This generates N+1 tokens per forward pass, or iteration through the +decode loop, where N is the number of accepted draft tokens. +Generating draft tokens is not free, it takes both compute and memory. +However, it is much faster for a target model to validate a draft token than +to generate an original token. If you imagine a sudoku puzzle, solving it +is hard, but checking if the solution is correct is very easy. For the target +model, generating a token is like solving a sudoku, while validating a draft +token is like checking a finished sudoku. +The performance uplift from any speculative decoding strategy depends +on three factors: + + +130 Chapter 5: Techniques +1. Draft token cost: The time it takes to generate a draft token. +2. Draft sequence length: The number of draft tokens that are generated +per forward pass. +3. Token acceptance rate: The percentage of draft tokens that are +accepted by the target model. +Token acceptance rate is high early in the draft sequence, but draft tokens +get less reliable deeper in the sequence. +Figure 5.3: Speculative decoding from draft token generation and validation to prefi x +acceptance with subsequent token generation. + + +5.2.1  Draft-Target Speculative Decoding  131 +Aim for short, high-percentage sequences because generating and validat- +ing tokens, while inexpensive relative to generating tokens in the original +model, still comes with meaningful overhead. Additionally, once a single +draft token is rejected as wrong, all subsequent tokens in the sequence +are also rejected. +Working with speculation is interesting because so many factors affect +token acceptance rate. The big one is temperature – higher temperatures +yield token distributions that are harder to predict, reducing the effective- +ness of speculative decoding. But even factors as simple as subject matter +can make a difference on acceptance rate if the draft model or additional +head used for speculation is better versed in, say, math than history. +Another limitation on speculative decoding is that it’s most useful at low +batch sizes where there are spare compute cycles. At higher batch sizes, +speculative decoding must be dynamically disabled as compute is too +saturated to afford verification. +Each speculation algorithm navigates these tradeoffs differently, and care- +ful implementation of the right algorithm for the situation can lead to major +improvements in TPS. +5.2.1  Draft-Target Speculative Decoding +The original method of speculative decoding uses two models: +• Draft model: An additional model that generates speculative draft +tokens. +• Target model: The original model, which now verifies draft tokens in +addition to performing ordinary decode. +The most important decision when configuring draft-target speculative +decoding is which draft model to use. A good draft model has a high token +acceptance rate while requiring minimal resources to run. +Draft models are most often smaller members of the same family as the +target model, as they share tokenizers and behaviors. As a rule of thumb, +the draft model should be at least ten times smaller by parameter count +than the target model. Fine-tuning or distillation can improve the token + + +132  Chapter 5: Techniques +acceptance rate of these small models by teaching them to behave more +like the target model. +Draft-target speculative decoding is a good choice when you want some- +thing that is quick to set up out of the box without doing any training or +fine-tuning. +However, other speculation algorithms generally offer better performance. +Draft-target introduces the most overhead of any speculative decoding +method. While the draft model is small, the inference engine must store the +draft model’s weights, activations, and KV cache in memory, and dedicate +compute cycles to draft model prefill. Additionally, the draft and target +model must run in coordination so that they don’t compete for resources, +though inference engines like TensorRT-LLM handle that model orches- +tration. +5.2.2  Medusa +Medusa was one of the first alternatives to draft-target speculation. +Medusa addresses the complexity and overhead of operating a draft model +to perform speculation by instead fine-tuning the target model to generate +additional tokens per forward pass. +Fine-tuning a model for Medusa means grafting additional decoder heads +onto the model. The ordinary architecture of an LLM contains a single +decoder head, but Medusa adds an additional two to four heads that +generate sequential draft tokens. +Like with draft-target speculation, draft tokens are validated on the next +forward pass. +Medusa is still limited on draft token count and draft token acceptance rate, +and it is not widely used in production today. However, Medusa inspired +more popular techniques like EAGLE. + + +5.2.3 EAGLE 133 +Figure 5.4: Medusa heads each generate a draft token on top of the token generated +by the target model. + 5.2.3 EAGLE +The main problem with using an off-the-shelf pretrained model as a draft +model is that a model like Qwen 0.5B is designed to be a good standalone +LLM on cheap hardware, not to speculate draft tokens on a B200. These +draft models are inefficient to run and offer a relatively low acceptance rate. +EAGLE offers an alternative: a purpose-built draft model trained from +scratch to generate sequences of up to eight draft tokens (two times more +than Medusa) with a very high acceptance rate. +During inference, LLMs accumulate a lot of context about the predicted +tokens in the form of hidden states between layers. Traditional draft mod- +els don’t have access to this information. +EAGLE is a draft model trained to accept hidden states as input and +generate speculative tokens as output. Specifi cally, it is trained on a set + + +134 Chapter 5: Techniques +of three hidden states: one from an early layer, one from a middle layer, +and one from a late layer. EAGLE is often less than one billion parameters +and scales well when given additional training data. +Figure 5.5: EAGLE speculative models take hidden states as input and produce draft +tokens as output. +In practice, when using EAGLE with an inference engine like Tensor- +RT-LLM, implementations tend to be straightforward, with post-training +EAGLE creation and single-sequence speculation. +EAGLE can be attached to the same module (PyTorch class) as the target +model, so each forward pass runs inference on both the target model and +the EAGLE speculator. This unifi ed pipeline solves the other problem with +draft-target decoding, where multiple round trips to the CPU were needed +to orchestrate the draft and target models. +EAGLE is the go-to speculation algorithm for general use among inference +engineers with the knowledge and means to train EAGLE heads and is +well-supported by inference engines. Like other speculation techniques, +adopting EAGLE for improved latency requires reduced batch sizes, low- +ering throughput and increasing cost. + 5.2.4 N-gram Speculation and Lookahead Decoding +N-gram speculation uses a different mechanism than other types of spec- +ulation. There is no draft model. +Instead, in parallel with generating the KV cache, the inference engine +constructs an n-gram dictionary. The n-gram dictionary maps a single +starting token to an observed sequence of N tokens (the n-gram). + + +5.2.4 N-gram Speculation and Lookahead Decoding 135 +Figure 5.6: An n-gram dictionary matches prefi xes to likely suffixes and is especially +useful for constrained languages like code. +This n-gram dictionary contains common sequences from the input text +and is fi rst constructed during prefi ll. During decode, the generated token +is fed into the dictionary, and any available suffix is selected as draft +tokens. In the next forward pass, the target model verifi es these draft +tokens as normal. +The advantage of n-gram speculation versus EAGLE is that the sequences +can be much longer. While EAGLE can generate eight or so draft tokens +with a decent acceptance rate, n-gram sequences can exceed ten tokens. +However, the acceptance rate for n-grams is only high when the contents +of the model output are similar to the model input. N-gram speculation is +mostly used for code completion and code revision, where syntax is pre- +dictable and output closely matches input. However, within this specifi c +domain, it easily outperforms EAGLE. +A similar method to n-gram speculation is Lookahead Decoding, which +generates n-grams during inference to fi ll the dictionary. Lookahead +Decoding is more general than n-gram speculation as it doesn’t rely as +much on highly repetitive context, but it requires extra compute to generate +the n-grams. +Every speculative decoding algorithm aims to reduce the total number of +forward passes needed to generate a complete output sequence, improv- +ing overall latency and specifi cally tokens per second per user during + + +136 Chapter 5: Techniques +decode. N-gram speculation excels at code completion and similar tasks, +while Lookahead Decoding generalizes in systems with excess compute. + 5.3 +Caching +During prefi ll, the inference engine builds a KV cache (a store of keys +and values for each token) on the input sequence. It then updates the KV +cache for each token during decode. As inference is autoregressive, the +value for each new token depends on the value of every previous token +in the sequence. +Every inference engine uses KV caching by default on a request-by-re- +quest basis. Without KV caching, LLM inference would be unbearably +slow as each previous value in the entire sequence would need to be +re-calculated for each subsequent token. +However, engineers can get even more utility from the KV cache by re-us- +ing it between requests rather than solely within each inference sequence. + 5.3.1 Prefi x Caching and KV Cache Re-Use +Consider the following two prompts, each with four tokens on most tokeniz- +ers, in Figure 5.7. +Figure 5.7: A pair of four-token sequences with two-token matching prefi xes. + + +5.3.1  Prefix Caching and KV Cache Re-Use  137 +By default, the inference engine has to run prefill on all four tokens of each +prompt. But the first tokens of each prompt – “Weather in” – form a shared +prefix between the pair. +With prefix caching, you can re-use the KV cache from the first request +to improve TTFT on the second request by skipping prefill on the first two +tokens and reading in the existing KV cache instead. +When you see pay-per-token APIs charge less for “cache hit” input tokens +than “cache miss” tokens, this is why – re-using cached tokens takes very +little compute power or time. As an inference engineer, you can apply the +same principle to reduce latency and improve throughput (thus saving +money) on your own deployments. +Saving two tokens won’t make a big impact on TTFT, but prefix caching +can skip prefill on thousands of tokens in certain domains: +• Complex system prompts: Agents, customer-facing chatbots, RAG +scaffolds, and tool calls often feature long, complex system prompts on +every call. +• Code completion: Code completion, code generation, and other coding +functions require passing the same thousands of lines of code as shared +context. +• Documents and retrieval: Document summarization, question answer- +ing, and retrieval all add repeated context ahead of user prompts. +• Multi-turn conversations: Ordinary conversations repeat back every +message in a chat template, increasing the savings from prefix caching +with every turn. +Prefix caching works from the start of the input sequence until the first +non-repeated token. The fourth token in the weather example, a question +mark, is shared between the two input sequences. However, the prefix +ends at the first non-repeated token, so the fourth token isn’t read from +cache. +Because prefixes end at the first unique token, your context engineering +determines your TTFT savings. Consider a different approach to the same +prompt: + + +138 Chapter 5: Techniques +Figure 5.8: A pair of four-token sequences with no prefi x match, the fi rst tokens are +different so it doesn’t matter that the next three are the same. +Here, there is no savings from prefi x caching as the very fi rst token differs +between the two sequences, even though every subsequent token is the +same. +To take advantage of prefi x caching, ensure that novel tokens are as late +in your context as possible. +Prefi x caching is the dominant form of KV cache re-use because LLMs +are autoregressive. Each token infl uences every subsequent token, so a +single novel token changes the way the model represents the rest of the +sequence internally, even if the sequences look the same to a human +reader. +However, there is active research around other kinds of KV cache re-use +to overcome this limitation. Caching arbitrary sequences from the middle +of prompts requires correcting both positional embeddings and selectively +recomputing KV entries to maintain output quality. Tools like CacheBlend +and LMCache support non-prefi x sequences, expanding the possibilities +for KV cache re-use. + + +5.3.2 Where to Store the KV Cache 139 + 5.3.2 Where to Store the KV Cache +The KV cache is very valuable. But KV caches take up a lot of memory, +and GPUs only have limited VRAM. +You can confi gure how much memory your inference engine allocates to +KV cache. For example, in TensorRT-LLM, you would set: +Figure 5.9: Allocating free GPU memory to the KV cache is an essential confi guration +decision when running inference engines. +If you’re working on a B200 GPU with 180 GB of VRAM and used 100 +GB for model weights and buffers, this would allocate 80 percent of the +remaining VRAM, or 64 GB, to KV cache. +Once this allocation fi lls – and it will fi ll quickly – you’ll have to start delet- +ing saved KV caches, increasing the chance of a cache miss on future +requests. +To get more room for KV cache, offload from VRAM to other nearby stor- +age. There are four places where you can store KV cache, in descending +order of bandwidth to the GPU: +Level +Memory type +Approximate +speed +Approximate size +G1 +Device Memory +(GPU VRAM) +Terabytes per +second +10s to 100s of +gigabytes +G2 +Host Memory +(CPU RAM) +10s to 100s of GB +per second +100s of gigabytes to +terabytes +G3 +Local SSD +5-10 GB per second +Terabytes +G4 +Networked SSD +Gigabytes per +second +10s of terabytes + + +140 Chapter 5: Techniques +Certain SKUs, like the GB200, come equipped with CPUs and intercon- +nects offering much faster G2 storage making them great for KV cache +offloading. +NVIDIA Dynamo provides support for KV cache offloading via KVBM (KV +Block Manager). KVBM provides APIs for moving KV cache blocks among +different levels of memory. As a general rule, you want to keep the most +frequently used blocks in higher-bandwidth memory, while less-often-used +blocks can be relegated to slower storage until needed. + 5.3.3 Cache-Aware Routing +In a production environment, there will be multiple replicas of your infer- +ence server, with incoming traffic split across the replicas. Usually, traffic +is routed based on how busy each replica is. +If your inference server makes heavy use of prefi x caching, your routing +logic needs to be updated to account for that. A user in a long conversation +with a chatbot or asking multiple questions about a codebase should have +their request routed to the same replica whenever possible so that they +get a cache hit for a faster, less expensive request. +Figure 5.10: Cache-aware routing allocates traffic based on KV cache rather than +simply dividing requests evenly across replicas. +Another option is using the G4 networked storage to build a global KV +cache across replicas. Routing still matters here – a replica with a hot G1 + + +5.3.4  Long Context Handling  141 +cache will serve the request faster than a replica reading from G4 – but a +global cache ensures that all replicas can eventually access any pre-com- +puted sequence and that cached sequences are not lost when nodes cycle +or are spun down during autoscaling. +5.3.4  Long Context Handling +“Long context” is a bit of a tautological definition: a sequence becomes +“long context” when it generates a KV cache large enough to cause prob- +lems during inference. +Depending on the model, hardware, engine, and traffic, these problems +can start to emerge past common cutoffs like 32K, 64K, or 128K tokens. +In your performance benchmarking, be sure to send very large input +sequences to test your inference service against long context requests. +Foundation model labs have been using scaling techniques like RoPE to +unlock longer and more accurate context windows. But supporting these +upgraded context windows introduces new challenges in inference. +Accounting for the KV cache, the attention equation scales linearly with +sequence length. With long sequences, attention can become the main +consumer of VRAM – the very resource decode is limited by. +While approaches like sliding window attention, compressed attention, +and sparse attention offer solutions on a model-by-model basis, there +are general approaches to optimizing the standard attention algorithm: +• FlashAttention: A series of optimized attention kernels to compute +attention with reduced numbers of reads from and writes to memory. +• PagedAttention: A memory management technique that stores KV +cache in fixed-size pages, reducing fragmentation and duplication. +• Chunked Prefill: A strategy of splitting large input sequences into +chunks, which can be run alongside decode as resources allow to avoid +overwhelming the inference engine with a long sequence. +But what if, after these optimizations, you still need more VRAM than a +single GPU offers to store KV cache? You’ll need to parallelize inference +across multiple GPUs. + + +142 Chapter 5: Techniques + 5.4 +Model Parallelism +Every frontier LLM on the market today is too big to fi t on a single GPU +for batch inference. While GPUs have gotten bigger, so too have models, +a trend that does not show signs of reversing. +In FP8, loading a billion parameters of model weights takes roughly a +gigabyte of VRAM. For a model like DeepSeek-V3.1, with 671 billion +parameters, the model weights alone would cause a single B200 GPU to +immediately throw an out-of-memory (OOM) error. +It’s not enough to just barely squeeze the model weights into VRAM. On +4xB200 GPUs, with 720 GB of VRAM, you could theoretically load Deep- +Seek’s weights. But with no room left over for a KV cache, which often +takes up 80 percent or more of the remaining VRAM after weights, four +B200 GPUs would not be able to serve DeepSeek with any reasonable +sequence length or batch size. +Instead, a full node of eight B200 GPUs is needed to serve real production +traffic on a model the size of DeepSeek. You can estimate the minimum +number of GPUs required for a model by multiplying the precision, param- +eter count, and expected KV cache allocation together. +Figure 5.11: After fi guring out how much VRAM inference requires, round up to the +next available instance size to determine minimum GPU count. + + +5.4  Model Parallelism  143 +In many cases, even for midsize models like GPT OSS, you want to use +more than the minimum number of GPUs required to enable larger KV +caches and unlock better per-user latency. +However, all of this requires that inference scales efficiently from one +GPU to multiple GPUs. The limitation in scaling parallel inference is the +communication overhead between GPUs. +Chapter 3 details the different interconnects between GPUs: NVLink and +NVSwitch within nodes, InfiniBand between nodes. +While NVLink and InfiniBand offer high bandwidth, they are a fraction of +the speed of VRAM. With decode bound on memory bandwidth, multi-GPU +inference needs to be carefully designed to avoid bottlenecks in inter-GPU +communication. This field of study is called topology-aware parallelism. +There are three primary forms of model parallelism in inference: +• Pipeline Parallelism (PP): Splits the layers of the model across GPUs. +• Tensor Parallelism (TP): Splits the tensors within each layer across +GPUs. +• Expert Parallelism (EP): Shards entire experts from MoE models +across different GPUs. +Each form of parallelism has its own tradeoffs: +Method +Mechanism +Drawback +PP +Each GPU handles a stage +of the forward and backward +pass. +Not recommended due to poor +latency and utilization from +step-by-step pipeline. +TP +Compute-heavy operations +like matmuls are split across +GPUs. +Requires synchronization +across GPUs, not suitable for +multi-node. +EP +Each expert lives within a +single GPU, making in-expert +inference fast. +Requires routing between +GPUs to reach multiple +experts, better for throughput. +Tensor Parallelism is generally best for low-latency model inference within +a single node, while Expert Parallelism improves throughput for MoE +LLMs. Pipeline Parallelism is only used for multi-node inference. + + +144 Chapter 5: Techniques +Additionally, data parallelism strategies like Context Parallelism split com- +putation across devices. These strategies are rare in LLM inference but +essential for video generation (section 6.6). + 5.4.1 Tensor Parallelism for Lower Latency +Tensor Parallelism should be your default strategy for multi-GPU model +inference. It supports both dense models like Llama 405B and the MoE +models that currently dominate the open model landscape. +Figure 5.12: Tensor Parallelism splits weights across GPUs, effectively sharing VRAM +resources to run large models fast. +TP works by splitting apart each layer of the model (as opposed to Pipeline Par- +allelism, which keeps layers intact) and distributing the layer fragments across +the allocated GPUs. For each layer, the expense of reading from weights +memory and executing matrix multiplication is shared across the GPUs. +Figure 5.13: For Mixture of Experts models, each expert runs across multiple GPUs +with Tensor Parallelism. + + +5.4.2 Expert Parallelism for Higher Throughput 145 +However, the results of each layer need to be communicated in an all-re- +duce fashion into a single output before the next layer can be computed. +In nodes with high-bandwidth intra-node NVLink and NVSwitch, this com- +munication overhead is minimized. +Increasing Tensor Parallelism improves TPS on a per-user basis (assum- +ing the model is large enough and the sequences are long enough that +the communication overhead doesn’t outweigh the faster forward pass, +which is the case for most frontier models). + 5.4.2 Expert Parallelism for Higher Throughput +Expert Parallelism neatly divides experts across GPUs. In a model with +128 experts served in EP8 across eight GPUs, each GPU will host 16 +full experts. +Figure 5.14: Expert Parallelism runs each expert within a single GPU, with GPUs +each hosting multiple experts. +EP improves total system throughput, making inference more scalable +and less expensive. With individual experts processing tokens separately, +each token takes just as long, but the system as a whole can handle more +simultaneous tokens. +Many deployments use a mix of TP and EP to achieve both benefi ts. + + +146 Chapter 5: Techniques +Figure 5.15: This deployment uses TP for attention and EP for the sparse MoE layer. +Expert Parallelism requires less inter-GPU communication than Tensor Par- +allelism. The Expert Router, which determines which experts each token acti- +vates, is replicated onto each GPU as it is a relatively small component of the +model. Inter-GPU communication is necessary for passing tokens from expert +to expert, but unlike TP, it is not required to collect the results of each layer. +Thanks to this lower communication overhead, EP scales well to multi- +node deployments and systems with limited interconnect bandwidth. + 5.4.3 Multi-Node Inference +If you’re serving a huge model at high precision, supporting multi-mil- +lion-token input sequences, or just trying to run inference as fast as pos- +sible, you might need more than eight GPUs. +GPUs are designed to work together across nodes, and multi-node training +has been the standard for years to develop frontier models. But multi-node +inference introduces new challenges: +• Infrastructure: How do you reliably provision two or more interconnected +GPU nodes and build abstractions across cloud providers (chapter 7)? +• Parallelism: How do you effectively communicate over Infi niBand, +which is much slower than NVLink? + + +5.4.3 Multi-Node Inference 147 +Figure 5.16: Infi niBand enables multi-node inference across more than eight GPUs +via high-bandwidth node-to-node interconnect. +Infi niBand introduces a new wrinkle to topology-aware parallelism. Tensor +Parallelism generally requires too much communication across GPUs to +be a good fi t for multi-node inference. Instead, you have two options that +work well over Infi niBand: +1. For dense models, use Tensor Parallelism within each node and Pipe- +line Parallelism between nodes (e.g., TP8PP2). +2. For MoE models, you can also try Expert Parallelism (e.g., EP16) as it +has a lower communication overhead than Tensor Parallelism. +For MoE models, TP8PP2 will generally offer lower latency per user and +EP16 will yield higher overall system throughput. +Unless your model and KV cache are so large as to require multi-node +inference, it probably isn’t the best use of the extra hardware. You’re often + + +148  Chapter 5: Techniques +better off using the extra nodes for horizontal scaling across replicas, or +for disaggregated serving. +5.5  +Disaggregation +Disaggregation combines three important ideas in inference engineering: +1. Prefill is a compute-bound process that determines your TTFT, while +decode is a memory-bound process that determines your TPS. +2. Specialization improves performance in everything from kernel selection +to inference engine parameter tuning. +3. You can effectively parallelize model serving over multiple GPUs, or +even multiple nodes, if you can avoid bottlenecks from lower-bandwidth +interconnects. +When prefill and decode run on the same node under heavy traffic, they +have a higher chance of interfering with one another. Ideally, prefill uses +more compute resources, while decode uses more memory, and the two +can co-exist efficiently. However, with larger batches and more compute-in- +tensive optimizations, prefill and decode start competing for resources. +5.5.1  How Disaggregation Works +Disaggregation, or disaggregated serving, is the idea of separating prefill +and decode into separate engines on separate GPUs or nodes. +Disaggregation turns LLM inference into a three-step process: +1. The prefill engine takes the input sequence and generates a KV cache +while computing the first token. +2. The prefill engine sends the KV cache over the hardware interconnect +to the decode engine. +3. The decode engine computes all subsequent tokens. +In conditional disaggregation, the request is first sent to the decode engine, +which checks if the input sequence is already cached or is short enough +to handle locally: + + +5.5.2 When to Use Disaggregation 149 +Figure 5.17: Disaggregation assigns prefi ll workers to generate the fi rst token and +decode workers to generate subsequent tokens. +1. If it is, the decode engine handles prefi ll locally, skipping disaggregation. +2. If it is not, the decode engine transfers the request to the prefi ll engine +for disaggregated serving. +Conditional disaggregation is better for real-world traffic. +Another benefi t of disaggregation is that with separate prefi ll and decode +engines, you can optimize each engine individually and the system as a +whole. For example, the compute-bound prefi ll engine requires a lower +TP than the memory-bound decode engine. + 5.5.2 When to Use Disaggregation +Disaggregation is very powerful but requires multiple GPUs and extra +engineering work. You should reach for disaggregation only when: +1. You are serving a large volume of traffic, starting at one hundred million +to one billion tokens per day depending on model size. +2. You are serving a larger model, at least a hundred billion parameters. +3. Your traffic is prefi ll-heavy with long input sequences. + + +150  Chapter 5: Techniques +If either point one or two is not true, you’re likely wasting money on extra +hardware for minimal performance gains. If point three is not true, you may +be better off using the extra GPUs to scale replicas horizontally, as decode +engines will be more efficient for short sequences or prefix cache hits. +A great use case for disaggregation is serving a frontier LLM in a code +editor, where many developers are simultaneously passing in large and +varied chunks of code as context. Tons of tokens, mostly prefill, on a tril- +lion-parameter LLM is the textbook workload for disaggregation. +5.5.3  Dynamic Disaggregation with NVIDIA Dynamo +Dynamo provides production-ready support for disaggregation, with flex- +ibility to handle heterogeneous real-world traffic. +Dynamo provides developer tools and pre-built optimizations to enable +disaggregation: +• A prefill queue to hold requests when all prefill engines are saturated. +• Robust support for conditional disaggregation, with prefill routing based +on configurable thresholds for ISL after prefix cache and prefill queue +size. +• Efficient NIXL-based KV transfer from prefill to decode engines with a +kernel to transpose KV blocks between layouts when the engines have +different TP configurations. +Combined, these features enable dynamic disaggregation, where the +number of prefill and decode engines is configurable at runtime and can +be adjusted over time to match the changing nature of incoming traffic. +Disaggregation does not need to be a one-to-one ratio between prefill +and decode engines. While it’s simple to explain disaggregation in terms +of a single prefill engine and a single decode engine, real systems have +multiple of each. +The number of prefill and decode engines is written as xPyD, for example, +5P3D means five prefill and three decode engines working together to +serve a single model deployment. + + +5.5.3  Dynamic Disaggregation with NVIDIA Dynamo  151 +As systems grow more complicated, more potential bottlenecks appear. +With disaggregation, the new bottleneck is prefill queue size. It’s important +to not let the queue grow too large, both by setting a reasonable threshold +for local prefill on the decode engine and by reconfiguring xPyD at runtime +to allocate more resources to prefill if needed. +The other potential bottleneck in disaggregation is running out of KV cache +on the decode engines under high load. Increase KV cache availability +with quantization and KV cache offloading. + + + diff --git a/library/Inference Engineering/Chapter-08-Modalities/Chapter-08-source-text.md b/library/Inference Engineering/Chapter-08-Modalities/Chapter-08-source-text.md new file mode 100644 index 0000000..88ab728 --- /dev/null +++ b/library/Inference Engineering/Chapter-08-Modalities/Chapter-08-source-text.md @@ -0,0 +1,691 @@ +# Inference Engineering — Chapter-08: Modalities +- **Source**: /library/Inference Engineering/source-file.pdf +- **PDF pages**: 155–178 +- **Pages without text**: 155, 156 + +--- + + +CHAPTER 6 +Modalities + + + + + +Modalities  155 + +Modalities +The modality of a model describes what types of input it accepts and +what types of output it creates. Chapters 1 through 5 focus on inference +engineering for LLMs, which take text as input and produce text as output. +This chapter expands the discussion to more modalities. +Generative AI models offer a rich array of modalities, including: +Input +Output +Category +Text and image/video +Text +Vision language +Text or image/video +Vector +Embedding +Audio (voice) +Text +Transcription +Text +Audio (voice) +Speech synthesis +Text +Audio (music) +Music generation +Audio (voice) +Audio (voice) +Speech-to-speech +Text and/or image +3D model +Generative CAD +Text and/or image +Image/Video +Image/video generation +Image/video +Text +Captioning +Image/video +Mask +Segmentation +Text and image +Image +Image editing +Fortunately, while there are many modalities, there are just two broad +archetypes of generative AI models as outlined in chapter 2: +• Autoregressive token generation: Start from a tokenized sequence +and predict the most likely next token. +• Iterative denoising: Start from random noise and refine toward the +most likely output. +LLMs are the most famous autoregressive transformers models for token +generation, but far from the only ones. Vision language models, text and +multimedia embedding models, automatic speech recognition (ASR) +models, text-to-speech (TTS) models, and many others rely on similar +architectures. + + +156 Chapter 6: Modalities +Many of the same inference engines and techniques used for LLMs also +apply to these related modalities. +Image and video generation models instead rely on iterative denoising, +though increasingly hybrid diffusion transformer models are setting the +frontier in quality. While a number of the same philosophies from kernel +selection to parameter tuning also apply to image model optimization, the +details end up quite different. +For each new modality, you also need to adjust the way you think about +and measure latency, throughput, and quality. For example, a single token +of audio output from a TTS model isn’t particularly useful; instead of TTFT, +measure the time to fi rst word or time to fi rst sentence. +This chapter discusses inference engineering for six common modalities +beyond LLMs, with special attention to the different considerations for +each modality. + 6.1 +Vision Language Models +Vision language models (VLMs) take one or more images or videos as +input along with a text prompt and generate a text response. +Figure 6.1: VLMs add image and video understanding to LLMs. +A vision language model usually consists of two modules: +• LLM: A standard large language model. +• Vision encoder: A small model that takes raw images and videos as +input and converts them into image tokens. + + +6.1  Vision Language Models  157 +The language model is much larger than the vision encoder. For example, +in Mistral Large 3, the vision encoder is just two billion parameters com- +pared to the 673B-parameter LLM. +While the vision encoder is small by parameter count, it is critical for +inference. VLMs use varied architectures and implementations for vision +encoders, so runtime support is somewhat more fragmented for vision +language models. This fragmentation increases the importance of vLLM +and SGLang for serving vision language models. +As a rule of thumb, sending a high-resolution input image to a VLM +adds about a thousand visual tokens to the input sequence. While at +a very high level image tokens are similar to regular tokens, they add +up quickly. +Across VLMs, the primary challenge in inference optimization is handling +the longer input sequence and larger KV cache. This adds wrinkles at +both phases of inference: +• Prefill: Images are patched, embedded, tokenized, and fed into prefill +as part of the input sequence. +• Decode: Same mechanics, longer context, and some models add atten- +tion variants for the image tokens. +Every technique from the previous chapter is useful in addressing this +challenge: +• Quantization: KV cache quantization reduces the memory bandwidth +and storage overhead for longer sequences. +• Speculation: Decode for VLMs matches LLMs and can be accelerated +with speculation, especially EAGLE. +• Prefix caching: Re-use KV cache for images in multi-turn chats and +repeated queries. +• Parallelism: Use Tensor Parallelism for fast inference while accessing +more VRAM for large models and long contexts. +• Disaggregation: Move prefill to specialized and independently scaling +workers to handle long sequences. + + +158  Chapter 6: Modalities +In addition to these techniques, VLMs introduce a new quality-speed +tradeoff: downsampling. Images and videos can be converted into visual +tokens at various resolutions. A high-resolution representation takes about +four times more tokens than a low-resolution image, but provides more +detailed information. Downsampling generally isn’t needed for single-im- +age inputs, but it may be needed when passing in multiple images or +video clips. +6.1.1  +Video Processing for Vision Language Models +A video is more than the sum of its frames. Videos may contain audio +(though many VLMs cannot process audio, which must be transcribed +separately and added into the prompt) and their frames express motion +of objects through space that is lost when looking at static images. +VLMs are trained on video clips to understand that time dimension. +High-quality inference requires processing the entire video clip in a sin- +gle call to the model. +One second of cinematic video contains 24 frames. Each frame is an +image. If a high-definition input image takes about 1,000 tokens to repre- +sent, then a four-second video clip produces an input sequence of nearly +100,000 tokens. +In reality, video inputs don’t generate quite this long of an input sequence +– downsampling is practically obligatory. +Reducing the resolution and frame rate makes it possible to evaluate +an entire clip in a single inference request, though video understanding +models are still only capable of taking very short clips. +After the video is tokenized and encoded, inference is similar to working +with images, just with much longer context. Prefix caching, KV cache +offloading, and optimized attention implementations are of even greater +importance for these input sequences of tens of thousands of tokens. + + +6.1.2 Omni-Modal Models 159 + 6.1.2 +Omni-Modal Models +Vision language models are an important part of a trend toward “omni” +models that accept multiple types of input and produce multiple types of +output. There are pros and cons to omni models – their blend of modalities +provides unique capabilities, but smaller specialized models are often +faster and more accurate within specifi c domains. +For example, many VLMs have text recognition capabilities trained into +their image input processing. However, these capabilities lag behind ded- +icated optical character recognition (OCR) models that are generally a +fraction of the size. +Running production inference on VLMs often involves coordinating a +pipeline of multiple models and pre-processing steps. You might have +individual preprocessors for extracting data from PDFs, reading text from +images via OCR, or transcribing audio from a video. +Each component in the pipeline must be individually optimized for speed +and should scale independently to avoid bottlenecks. + 6.2 +Embedding Models +An embedding model transforms a variable-length chunk of text – or +another modality of input like an image – into a fi xed-length vector repre- +sentation that captures the semantic meaning of the input. +Figure 6.2: Embedding models convert unstructured input data into vectors that +encode semantic meaning. + + +160  Chapter 6: Modalities +By encoding content into this shared semantic vector space, you can +compare distance between items with simple math. Embedding models +(along with vector databases) are used to build agent memory, RAG, +search, and recommendation systems. +To support these use cases, embedding model inference workloads have +two different traffic profiles: +1. High-throughput backfills: Bulk operations like indexing millions of +documents, updating product catalogs, or even preparing data for LLM +pre-training. +2. Low-latency lookups: Individual user-facing queries for search, +retrieval, or recommendation, where every millisecond affects user +experience. +Inference engineering for embedding models starts with clarifying which +profile you need to serve. If you need to do both and have enough traffic to +justify the cost, it’s better to build a separate system for each type of usage. +6.2.1  Embedding Model Architecture +There are tens of thousands of embedding models on Hugging Face, but +they all use one of two transformers-based architectures: +• BERT-style models: Encoder-only neural networks, usually <1B +parameters, originally built for masked token prediction. +• LLM-based models: Modern language models, generally <=8B param- +eters, repurposed to generate embeddings. +Today, LLM-based embedding models offer substantially greater capabil- +ities, though BERT-style models are still used for simple latency-sensitive +tasks like classification. +Embedding models introduce their own speed/quality tradeoff in embed- +ding dimensionality, or the size of their output vectors. An embedding vec- +tor contains a few hundred to a few thousand values, with longer vectors +encoding more information. + + +6.2.2 Embedding Model Inference 161 +Modern embedding models use Matryoshka representations to unlock +dynamic tradeoffs between embedding dimensionality and quality while +retaining more information on shorter vectors. Dimensionality doesn’t +materially affect inference time but does affect the storage, retrieval, and +similarity computation time within a system. +In most cases, vectors from one embedding model cannot be meaningfully +compared to vectors from another embedding model, even if they are the +same length, as they encode inputs into different semantic spaces. + 6.2.2 Embedding Model Inference +For embedding models with LLM backbones, like Qwen 3 Embed 8B, +inference optimization shares common tools and techniques with other +high-volume, low-latency deployments of smaller LLMs. +There are multiple runtimes for text embedding models: vLLM, SGLang, +Infi nity, TEI (Text Embedding Inference by Hugging Face). But the best +performance comes from adapting TensorRT-LLM to run these models. +Figure 6.3: A high-performance embedding inference pipeline adds parallel +tokenization and batch management in front of an optimized inference engine. +TensorRT-LLM brings an optimized XQA kernel for fast attention and ker- +nel fusion techniques to reduce memory access overhead. For supported +models, TensorRT-LLM is the most performant inference engine for both +latency and throughput. +Further gains come from quantization. While smaller models are more +likely to lose quality from quantization, FP8 quantization for embedding +model weights offers improved performance with minimal quality loss. + + +162 Chapter 6: Modalities +The easiest way to check embedding model quality post-quantization is +to run the same inputs through both the original and the quantized model, +then check the cosine similarity of the output vectors. A cosine similarity +of one hundred percent means the vectors are identical; you’ll want to +see at least 99 percent similarity to have confi dence in the quantization. +As embedding models process tokens in parallel, prefi x caching and dis- +aggregation aren’t relevant optimizations. And given these models’ small +size, parallelism across multiple GPUs is not effective. Instead, high-traffic +deployments should scale horizontally, with each GPU as its own replica. +In high-traffic deployments of embedding models, batching and queueing +play an important role in performance. Embedding models offer much +higher batch sizes than other models. A single request may batch dozens +or hundreds of text inputs together in a list, and many requests can run in +parallel on a single GPU as even the most demanding embedding models +are relatively small and fast. +Whether you’re performing a large backfi ll or handling a surge in usage, +traffic can exceed even the large batch sizes offered by embedding mod- +els. In these cases, a robust queuing system is essential infrastructure for +supporting embedding model inference. + 6.3 +ASR Models +Automatic speech recognition (ASR) models take audio as input and pro- +duce text as output, powering transcription and dictation apps. The most +popular open ASR model is Whisper, which was released by OpenAI. +Whisper supports dozens of languages with accurate transcription. +Figure 6.4: ASR models transcribe input audio into text. + + +6.3.1  Single-Chunk Latency Optimization  163 +Whisper comes in various sizes, but the largest and highest-quality Whis- +per model is just 1.55B parameters. Whisper runs extremely fast on frac- +tions of large GPUs like H100 via Multi-Instance GPUs (MIGs). While +various other sizes, variants, distillations, and quantizations exist, in prac- +tice it’s possible to satisfy most latency budgets with the highest-quality +models: Whisper 3 Large and Whisper 3 Turbo. +Whisper is an encoder-decoder model: +• Encoder: Takes a processed audio waveform (log-Mel spectrogram) +as input and encodes it into audio features. +• Decoder: Takes these encoded audio features and converts them into +text tokens. +The overwhelming majority of inference time is spent on the decoder, +which is an autoregressive transformer model very similar in architecture +to an LLM. Fortunately, there are excellent tools for optimizing the main +bottleneck. +The main tool for performance optimization on the decoder side is Ten- +sorRT-LLM. With TensorRT-LLM, you can get in-flight batching for the +decoder and an optimized C++ runtime with highly efficient CUDA kernels. +TensorRT-LLM works especially well with recent architectures like Hopper +and Blackwell, making MIGs an even better option for ASR inference. +6.3.1  Single-Chunk Latency Optimization +One use case for Whisper is real-time transcription, like in a dictation app +or voice agent. +For live Whisper, look at round-trip time for a single chunk of audio to +be transcribed. A great target to aim for is 200 milliseconds, which is the +average human reaction time. +With Whisper running on an optimized TensorRT-LLM inference engine, +there isn’t a lot of work to do on the runtime level to improve performance. +Instead, most gains for real-time Whisper come from orchestration and +infrastructure. + + +164  Chapter 6: Modalities +The biggest upgrade in product experience for ASR is streaming, which is +implemented at the API server layer rather than the model runtime layer. +By establishing a WebSocket connection (section 7.5.3) and streaming +audio continuously in and text continuously out, products can transcribe +in real time rather than transcribing pre-recorded audio. +At the ASR runtime layer, nothing changes. Instead, a streaming imple- +mentation for transcription uses a Voice Activity Detection (VAD) model +to monitor the incoming stream and segment it into discrete chunks for +the ASR model to process. Inference is run on the chunks as normal, and +the text results are streamed back via the WebSocket. +This setup can handle several concurrent streams, and it has the advan- +tage of keeping transcription sequential. When each chunk is processed +on the same GPU, you can use the output sequence of the previous chunk +as the prefix for the next chunk, improving transcription quality. +6.3.2  Long File Latency Optimization +One limitation of the Whisper model is that it can only support 30-second +chunks. Transcribing long files, like hour-long podcasts, requires a differ- +ent set of optimizations. +Measure the performance of long file transcription with the confusingly +named Real-Time Factor (RTF). If the world’s fastest typist could manually +transcribe an hour of audio in 30 minutes, they would have an RTF of 2X. +With a Whisper deployment optimized for long files, you can transcribe an +hour of audio in less than four seconds, for an RTF of 1000X. +Fast transcription for long files requires a multi-step pipeline. The first step +is again a VAD model, this time running on its own dedicated hardware. +The model is used to remove silence and chunk out meaningful audio +segments rather than splitting by time intervals, which runs the risk of +cutting words in half. +Then, the chunks can be processed in parallel. Ideally, you use multiple +GPUs (or multiple MIGs) to process more audio chunks at once. RTF +improves roughly linearly with the number of GPUs used. Each GPU pro- +cesses multiple chunks at once with in-flight batching for high utilization. + + +6.3.2 Long File Latency Optimization 165 +Finally, the chunked transcripts are stitched back together by timestamp. +Figure 6.5: A two-stage pipeline for long audio fi le transcription parallelizes chunk +transcription to improve end-to-end request time. +Parallelizing chunk transcription removes the ability to use the previous +sequence as a prefi x for the next sequence. But there are other quality +improvement techniques that more than make up for this. +With ASR output, you can automatically detect hallucinations like repeated +words and phrases by measuring the compression ratio and words per +minute of the output. When a chunk has an issue, you can: +1. Re-run the chunk with a higher temperature. This is counterintuitive – +higher temperatures generally produce more hallucinations – but the +intention is to break cycles of repeated words and generate a different +output. +2. Re-chunk the entire audio, or a segment of the audio, into smaller +chunks and re-run the transcription. +In practice, these techniques obviate the need for passing a previous +sequence as a prefi x, unlocking highly efficient and accurate parallel tran- +scription for long fi les. + + +166 Chapter 6: Modalities + 6.3.3 Diarization +Diarization, or annotating a transcript with who is speaking when, is an +adjacent problem to transcription. Diarization models categorize audio +by voice feature, then segment and cluster across the fi le to timestamp +changes in speaker. +Diarization models are a completely different class of model. Where Whis- +per is an encoder-decoder transformers model, diarization systems like +pyannote audio are pipelines of classic ML models. +A diarization pipeline contains models for segmentation, embedding, and +clustering. To optimize diarization, you have to run each model fast and +orchestrate the whole pipeline efficiently. +As diarization is an ML pipeline, you can use tools like PyTorch and pyan- +note along with optimizations like Torch compilation to improve its perfor- +mance. In practice, even highly optimized implementations of diarization +take at least twice as long to process an audio fi le versus transcription. + 6.4 +TTS Models +Text-to-speech (TTS) models, also called speech synthesis models, +take text as input and produce audio as output, specifi cally generating +speech. In 2025, open models like Orpheus TTS introduced extremely +lifelike speech synthesis to the open model ecosystem. Many companies +fi ne-tuned Orpheus for increased vocal quality and product-specifi c voices, +leading to high adoption of open models in the voice AI space. +Figure 6.6: Text to speech models synthesize input text into audio. + + +6.4  TTS Models  167 +Modern TTS models are fine-tuned LLMs. Orpheus TTS, for example, is +derived from Llama 3.2 3B. This means that many of the same runtime +and performance optimizations developed for LLMs apply to speech syn- +thesis models. +TTS models have a small parameter count – Orpheus TTS at three billion +is on the larger end – meaning that like ASR models, MIGs on H100s are +highly efficient and performant options for inference. +Unlike ASR models, which generally run in FP16, TTS model weights and +KV cache can be quantized to FP8 for better performance in addition to the +optimized kernels and in-flight batching introduced by the TensorRT-LLM +inference engine. +TTS models with LLM backbones are trained by expanding the vocabulary +size of the LLM with tens of thousands of encoded audio tokens. Then, the +models are trained on pairs of text inputs with tokenized audio outputs. This +means that to use TTS models in practice, you also need an audio decoder +that takes the audio output tokens and converts them into a waveform. +This audio decoding process adds a potential bottleneck to inference. The +audio decoder should be implemented using PyTorch and compiled for +efficient operation on the target GPU and should use dynamic batching +with a short timeout (e.g., 15 milliseconds). In-flight batching is not possible +for the audio decoder. +TTS model performance is measured with somewhat different metrics than +LLMs. The key metrics are: +• TTFB: Time to first byte (TTFB) is the equivalent of TTFT for speech +synthesis. +• Time to first sentence: Instead of TTFB, a more user-oriented latency +metric is the time to generate the first meaningful phrase or sentence. +• TPS: Like an LLM, the TTS model generates tokens, so decode speed +can be measured in tokens per second. +Like with TTFT on LLMs, the goal is to minimize TTFB for speech syn- +thesis. For Orpheus, it’s possible to get as low as 150 milliseconds on a +single H100. + + +168  Chapter 6: Modalities +However, there are different goals for TPS on speech synthesis models. +The tokens that the model generates are converted to audio waveforms. +Depending on the model, it might take 80 to 100 tokens per second to +generate audio in real time. Beyond that level, there isn’t any benefit to +generating additional tokens per second. +Instead, performance enhancements are used to scale throughput in terms +of the number of concurrent real-time outputs the model can create. If +a single GPU can support many concurrent users, the per-user cost of +speech synthesis drops dramatically. +6.4.1  Streaming Real-Time Text to Speech +Most TTS tasks call for real-time speech synthesis. Like with ASR models, +the performance gains for real-time systems come less from the runtime +layer – which has already been optimized with TensorRT-LLM, quanti- +zation, and a compiled SNAC decoder – and instead from infrastructure. +Again, streaming over WebSockets is the biggest unlock for performance +versus sending text and receiving audio in discrete chunks. After testing +the inference engine to determine how many concurrent real-time streams +can be generated, set the same batch size and active WebSocket count +to keep usage high but stable. +TTS models are rarely used outside of real-time applications. However, +if you do end up with a batch use case like backfilling a large corpus of +documents to audio for improved accessibility, note that TTS models don’t +do well with long inputs, speech starts to degrade after 30 seconds or so. +6.4.2  Speech-to-Speech Models +One exciting area of research is speech-to-speech models, or models that +take audio as input and generate audio as output. +Today, most voice systems use a cascading approach, where an ASR +model, LLM, and TTS model work in a pipeline to listen, think, and respond +to users. These pipelines also employ auxiliary components like VAD and +embedding models to facilitate natural conversation and add context. + + +6.5 Image Generation Models 169 +Figure 6.7: Most voice-based applications use a cascading approach with a multi- +model pipeline. +Speech-to-speech models, like OpenAI’s gpt-realtime, augment a core +LLM with audio consumption and production capabilities, effectively +unifying the pipeline in a single model. This is possible thanks to ASR, +LLM, and TTS sharing such similar architectures, especially on the +decoder. +At the time of publication, there are no commercially viable open speech- +to-speech models, and closed options like gpt-realtime are signifi cantly +less capable and more expensive than cascading multi-model setups. +However, research in this space is robust, and this emerging modality will +soon require its own fl avor of inference engineering. + 6.5 +Image Generation Models +Figure 6.8: Image generation models may accept both text and reference images to +create new output images. + + +170  Chapter 6: Modalities +Working with image and video generation models is entirely different from +working with large language models on a few axes. +The first is architecture. While some recent models like HunyuanImage-3.0 +more closely resemble LLMs, most image and video generation models +are iterative denoisers, not autoregressive token generators. Image gen- +eration models are pipelines with multiple small models working together +in latent space rather than the uniform decoder architecture of an LLM. +As such, the tooling is different. At the time of publication, SGLang Dif- +fusion and vLLM Omni are brand new. Most image and video generation +model inference is implemented lower in the stack, working with PyTorch +or TensorRT directly. +The constraints are different too. Image generation models are ten to +twenty times smaller than frontier language models, and inference is con- +strained on compute, not bandwidth. +But perhaps the most significant difference is that image and video gen- +eration models offer more direct quality to speed tradeoffs. +Evaluating image model output quality is difficult to do programmatically. +Automatic pipelines using vision language models give directional signal +at best and may diverge from human preferences. The human eye is +mysterious, and most image quality evals work by asking humans to pick +among thousands of images to aggregate vibes and preferences into +quality benchmarks. +6.5.1  Image Generation Kernel Optimization +When you read a model card for an image generation model from its +repository, inference examples generally use the diffusers library with +very few optimizations. +In fact, while image generation is theoretically compute bound, you often +need to select memory-efficient kernels and use kernel fusion to even +reach that bottleneck. +High-performance image model inference uses one of three libraries: + + +6.5.1  Image Generation Kernel Optimization  171 +• SGLang Diffusion: Performant inference engines built for popular +image and video generation architectures. +• TensorRT: High-quality black-box implementations of popular models +with NVIDIA’s in-house kernels. +• PyTorch: Careful kernel selection and fusion yields control, flexibility, +and improved high-end performance. +If you want something that works well and you want it now, just use the +SGLang Diffusion or TensorRT implementation of a model. But with +PyTorch, there’s an opportunity for advanced inference engineers to do +deep customization. +The most essential kernel is the attention kernel. Many image generation +models use FlashAttention 2 out of the box, but FlashAttention 3 and 4 +yield better performance on Hopper and Blackwell GPUs, respectively. +There is a whole barrage of smaller kernels, especially normalization +functions like RMSNorm, that are good candidates for fusion to ensure +efficient memory usage. +Then, GEMM kernels matter for compute-bound inference. GEMM kernels +apply to linear layers, and are generally safe to quantize into 8-bit floating +point formats to access two times higher FLOPS on Tensor Cores. Ker- +nels from CuTe, CUTLASS, or DeepGEMM may prove most efficient on +a model-by-model basis. +Torch compilation includes automatic kernel fusion with a plugin system +for inserting manually selected kernels, and the resulting engine can be +cached for faster load times on node startup (which is important because +compilation takes several minutes). +Like most high-performance engines, Torch compilation targets the spe- +cific GPU model and architecture performing the compilation – if you want +to run the model on a B200, do the compilation on a B200. + + +172 Chapter 6: Modalities + 6.5.2 One Weird Trick for Faster Image Generation +Kernel selection and Torch compilation are all bona fi de inference optimi- +zation techniques. But the world of inference optimization has fun hacks +as well, and here’s one of them. +Figure 6.9: Recall that diffusion is a step-by-step process and that the general outline +of the image is established in early steps. +Image generation time tracks linearly with step count. That’s why few- +step models and latent consistency models are so much faster than full +50-step models. But reducing step count may reduce image quality below +an acceptable threshold. +Each pass through the denoising model is run at a batch size of two as +each step includes a pass with and without prompt guidance. +As a refresher, the guidance parameter controls how much the prompt- +guided image is weighted when combining the two iterations generated +on each step. If the guidance is zero, the prompt-guided image does not +need to be generated. +After the fi rst few steps, the basic outline of the image is in place, and the +rest of the steps are for fi lling in the details. Thus, prompt adherence is +more important in early steps that affect the broad strokes of the image – +the model is not going to change its mind on later steps and generate a +dog when it is in the middle of generating a cat. +If you turn off guidance partway through the image generation, you save +passes through the denoiser without reducing step count. If guidance is +skipped for the last 20 steps of a 50-step run, there are only 80 passes +through the model instead of 100, and quality generally remains high. + + +6.6 Video Generation Models 173 + 6.6 +Video Generation Models +Figure 6.10: Video generation models take text prompts and may take keyframes or +other image, audio, and video input. +Video generation is the most demanding modality. Whenever possible, +these models should be run on Blackwell GPUs (or Rubin, once available). +These GPUs offer a high memory capacity for Context Parallelism, fast +Tensor Cores for attention computation, and microscaling data formats +for more precise quantization. +Architecturally, video generation is similar to image generation, just ren- +dering a full video rather than a single frame from latent space. Following +the principle that greater scale unlocks more techniques, video genera- +tion uses all the same techniques as image generation plus additional +optimizations. +Like image generation, video generation is compute bound and works via +iterative denoising over latent space. Video generation models generally +take about the same number of denoising steps as image generation +models (~50), but each step processes much more data. +As video generation models are compute bound, batching isn’t useful like +it is for text generation. Video generation models usually run on full nodes +of eight GPUs with a batch size of one: all eight GPUs work together to +create one video at a time. +Unlike with batched workloads, where latency-throughput tradeoffs are +possible by adjusting the batch size, the only way to improve the through- +put and cost of video generation is to make the model itself faster. + + +174  Chapter 6: Modalities +Early video generation models were framewise. They generated frames +one at a time. This reduced the quality and coherence of the video output. +Today, video generation models run denoising steps on the video as a +whole in latent space. Where latent space for image generation represents +two dimensions (width, height), for video models it represents three (width, +height, time). +This means passing huge amounts of data through each attention calcula- +tion. For video models, attention is 70 to 80 percent of the compute time, +making attention the most important thing to optimize. +6.6.1  Attention Optimization and Quantization +Attention optimization starts with kernel selection. Test FlashAttention, +DeepGemm, CuTe, and CUTLASS kernels to see which ones perform +best for your model. +Where language models use the KV cache to accelerate attention, video +generation models use other caching patterns to attempt to reuse model +outputs. Re-using parts of the attention computation can make video gen- +eration 30 to 40 percent faster in practice. +Precise methods and algorithms are continuously changing with new +research, but there are two fundamental approaches to caching: +• Timestep-based caching: Caching and re-using the outputs of certain +timesteps to skip entire steps. +• Transformer-based caching: Caching and re-using hidden states to +skip layers within the transformer itself. +Algorithms and implementations range from negligible quality degradation +to unusable output – test these strategies carefully before using them in +production. +Beyond kernels and caching, the main tool for speeding up attention is +quantization. +For bandwidth-constrained language model inference, the benefit of quan- +tization is that it means you have less data to load through memory. For + + +6.6.1  Attention Optimization and Quantization  175 +video models, it means you access double the FLOPS by switching to +lower-precision Tensor Cores. +However, language model quantization focuses on weights – large linear +layers where the impact of quantization is negligible. For video models, +quantizing weights still helps, but while these layers take the majority of +the memory bandwidth, constraining language models, they’re only a small +fraction of the compute time for video models. +Instead, quantization on video models focuses on attention. Attention is +the riskiest part of any model to quantize, as errors accumulate over the +course of inference. For video models, where there are ~50 steps instead +of the thousands of autoregressive iterations in token generation, the risk +is slightly lower but still important. +The first method for reducing the quality impact is to use a blockwise +quantization and a microscaling data format (MXFP8), both available on +Hopper and Blackwell. Microscaling data formats do a better job of pre- +serving outlier values, which have a major impact on attention accuracy. +The most sophisticated approach to attention quantization is selectively +quantizing within the model by: +• Step: Keep early steps in FP16 and quantize later steps. +• Layer: Keep first and last layers and quantize hidden layers. +Quantization by step follows the same insight as the classifier-free guid- +ance trick from image generation models: early steps establish the outline +of the image, while later steps refine the details. These early steps are +more important for prompt adherence and accuracy. +For layers, the first and last layers are more important as they take the +input and produce the final output. The hidden layers only perform inter- +mediate calculations which don’t suffer as much from approximation. +By only quantizing less important parts of the video generation process, +quality is preserved. These tactics are found in kernels like SageAttention, +an 8-bit attention kernel that you can use for quality low-precision attention +on video generation models. + + +176 Chapter 6: Modalities + 6.6.2 Context Parallelism +While video generation models generally run on a full node of eight GPUs, +they use Context Parallelism rather than Tensor Parallelism. +Context Parallelism copies the weights onto every GPU. Video models are +small enough that replicating the weights eight times takes a meaningful +amount of memory but is feasible on B200. +Instead of splitting the model across GPUs, Context Parallelism works by +splitting the attention calculation across the GPUs. This is coordinated +via a mechanism like ring attention, where each GPU holds a piece of +the context and passes intermediate results to the next GPU in the ring. +Figure 6.11: Context Parallelism replicates model weights but shares latent space to +compute attention during video model inference. +Attention for transformer models is multi-head, usually with eight or more +heads. Attention heads are independent, so they can be run separately +with the results combined afterward. +Attention isn’t the only thing that can be parallelized. For example, the +latent decoding step using the variational autoencoder takes three to fi ve +percent of the total inference time and can be run across GPUs. +These parallelism techniques make AI video feasible. As video sequences +get longer and video models get larger, parallelism will continue to be the +most critical technique for video generation model inference. diff --git a/library/Inference Engineering/Chapter-09-Production/Chapter-09-source-text.md b/library/Inference Engineering/Chapter-09-Production/Chapter-09-source-text.md new file mode 100644 index 0000000..a372770 --- /dev/null +++ b/library/Inference Engineering/Chapter-09-Production/Chapter-09-source-text.md @@ -0,0 +1,814 @@ +# Inference Engineering — Chapter-09: Production +- **Source**: /library/Inference Engineering/source-file.pdf +- **PDF pages**: 179–210 +- **Pages without text**: 179, 180 + +--- + + +CHAPTER 7 +Production + + + + + +Production  179 + +Production +The purpose of inference engineering is to make generative AI models +faster, less expensive, and more reliable to operate, which in turn lets you +build better products. +This promise is only fulfilled if your inference engineering work makes it +to production and scales alongside the hypergrowth and viral spikes that +successful AI products generate. +When you scale production traffic, your assumptions are rigorously tested. +Everything from sequence shapes to traffic patterns to what topic a user +decides to chat about impacts your observed performance in production. +And maintaining secure, robust infrastructure is an entirely different skillset +from optimizing model inference on the GPU. +No matter how fast and efficiently a single instance can serve a model, +with enough traffic, the service will be overwhelmed. That is not a PyTorch +problem or a CUDA problem, it’s an infrastructure problem, and requires +a different mindset and different technologies. +Scaling in production introduces new complexities about where and how to +get GPUs, balance traffic across them, and prevent downtime. Plus, cost +accounting gets messy in the transition from paying per million tokens to +paying directly for infrastructure. +Latency in production comes from more than just prefill and decode. You +need to evaluate your system end-to-end and eliminate inefficiencies in +the server, the network, and even the client in situations where you can +own or influence client code. +This chapter introduces the essential considerations for scaling low-la- +tency, high-throughput inference in production. And at the end, I’ll invite +you to try Baseten for deploying mission-critical inference workloads. +7.1  +Containerization +Containerization is the practice of packaging an application together with +its dependencies to standardize deployment in production. Containers + + +180  Chapter 7: Production +turn a program into a packaged artifact that can run anywhere – no more +“it works on my machine.” +Containers are lightweight because they share the underlying host operating +system kernel (in this context, a kernel refers to a Linux kernel, not a CUDA +kernel). This makes containers well-suited for packaging inference services. +For most developers, containerization is synonymous with Docker. Work- +ing with containers introduces more specific terminology: +• Container: An actively running environment that isolates an application +and its dependencies. +• Image: An executable package that contains everything you need to +run a piece of software. +• Dockerfile: A human-readable file with well-specified, machine-inter- +pretable instructions for creating an image. +• Registry: A central repository for managing, storing, sharing, and dis- +tributing images. +NVIDIA, several cloud providers, and Docker themselves all operate con- +tainer registries. A popular registry for AI is Docker Hub – Docker Hub +is to images as Hugging Face is to model weights or PyPi is to Python +packages. +Docker containers are composed of layers. You can take a pre-configured +base image and add other layers with additional filesystem changes on +top of it. +There are three types of layers: +• Base image: Either an operating system distribution like Ubuntu or a +more complex image taken from a registry. The base image itself is +composed of multiple layers. +• Additional layers: Filesystem changes including dependencies, applica- +tion code, and configuration files as specified by Dockerfile instructions. +• Container layer: A thin, ephemeral layer created at runtime. Any +changes to the running container, like creating, updating, or deleting +files, are written to this layer and are lost when the container terminates. + + +7.1.1 Dependency Management 181 +Figure 7.1: Docker containers are composed of layers, from a base image up to an +ephemeral, writable top layer. +Inference engines like vLLM and SGLang offer official base images for +active releases. It’s generally a good idea to start from one of these proven +images, rather than building your own from scratch. + 7.1.1 +Dependency Management +Dependency chains for inference are long and fragile. Getting to a work- +ing build is hard, which makes containerization essential for preserving +a known good build in an ecosystem where breaking changes are all too +common. +Images are built for a specifi c GPU architecture and model. A container +includes many runtime components: +• CUDA toolkit version: The specifi c versions of CUDA, cuDNN, and +drivers compatible with the rest of your stack. +• Python packages: Dependencies like torch, transformers, and +diffusers. + + +182 Chapter 7: Production +• Inference engine: The version of vLLM, SGLang, TensorRT-LLM, or +any other inference engine used. +• System packages: Linux packages like ffmpeg, especially common +when working with audio, image, or video models. +Like a hiker on a backpacking trip, you want to pack light. Images built +for inference are often many gigabytes. For fast deployment and efficient +operation, only include strictly necessary dependencies. +Another best practice is pinning versions. Having a pinned dependency +tree keeps the system runtime behavior consistent across different envi- +ronments and enables repeated builds of the image with the same result. +Specify exactly which version of each dependency should be included in +the image. +Figure 7.2: Requirements should be pinned to exact versions to prevent future +changes from breaking inference containers. +Tools like uv, poetry, or pip will fl ag any version incompatibilities and throw +an error when building an image. With pinned versions, once an image is +successfully built once, it will always resolve dependencies to the same +versions and protect you against breaking changes. +Breaking changes are particularly common when working with newly +released models. When new versions of models like DeepSeek are +announced, the entire inference ecosystem races to offer day zero sup- +port. +When building images for brand-new models, inference engineers often +rely on overnight builds or other developer pre-releases for dependencies +rather than stable releases. These early versions are more prone to bugs +and often need to be rebuilt on stable releases in the days and weeks +following the model drop. + + +7.1.2  NIMs  183 +7.1.2  +NIMs +NVIDIA Inference Microservices (NIMs) are pre-built Docker containers +for popular open models. +Containers make inference service implementations portable. NVIDIA +created two types of NIMs: +• Multi-LLM NIM: A flexible container for running a family of models on +a supported GPU architecture. +• LLM-specific NIM: An engine optimized for a specific model on a spe- +cific GPU configured for maximum performance. +NIMs are available for various common combinations of model, GPU archi- +tecture, GPU count, and configuration. +A NIM is like any other container. You can use a NIM as a starting point +to build on, as a reference architecture to learn from, or as an out-of-the- +box inference service. +However, if you’re looking for maximum control instead of a done-for-you +configuration, you’ll generally be better off building your own container +from a less opinionated base image rather than adapting a NIM. +7.2  +Autoscaling +The goal of autoscaling is to ensure that you always have enough +resources to serve all incoming requests while maintaining your latency +SLAs without wasting money on idle GPUs. + + +184 Chapter 7: Production +Figure 7.3: Without autoscaling, inference systems waste resources during traffic lulls +and miss SLAs during traffic spikes. +Figure 7.4: A strong autoscaling system for inference matches resources to demand. +Autoscaling systems use Kubernetes, an open-source container orches- +tration system, along with a cluster-level system for provisioning and deal- +locating compute. Kubernetes can run one or more replicas of a model +container, each on its own instance. An instance includes the GPUs and +other hardware resources that the container requires. + + +7.2 Autoscaling 185 +Kubernetes works by composing a group of hardware resources together +into a cluster. This cluster has two types of components: +• Control plane: Makes routing and scaling decisions. +• Worker plane: Runs the actual containerized applications. +Figure 7.5: Kubernetes clusters have a single control plane that orchestrates multiple +workers. +A Kubernetes cluster can run multiple replicas of multiple models. But +how do you decide how many replicas of each model to run? Unless your +traffic is unusually consistent, there probably isn’t one number of replicas +that perfectly matches your needs. +Autoscaling is the practice of dynamically adjusting the number of replicas +allocated to a given model within a cluster. There are two ways to make +autoscaling decisions: +• Utilization: Scale up and down based on GPU utilization signals like +memory usage or compute usage. +• Traffic: Scale up and down based on the number of requests being +processed in the system. +Utilization and traffic don’t always match. For example, in LLM prefi ll, a +few requests with hundreds of thousands of uncached input tokens could +cause much higher utilization than many small requests with high cache +hit rates. + + +186  Chapter 7: Production +Traffic-based scaling decisions can be made proactively, while utilization +is a lagging indicator. Use both in combination to keep system resources +matched with demand. +When designing a traffic-based autoscaling system, you want to configure +five factors: +• Min replicas: What is the minimum number of replicas that stay run- +ning, regardless of traffic? +• Max replicas: What is the maximum number of replicas that you can +allocate when traffic is high? +• Autoscaling window: How long is the sliding timeframe that you use +to measure traffic and make autoscaling decisions? +• Scale down delay: How long after a scale down is suggested do you +wait in case of another traffic spike? +• Concurrency target: How many requests can each replica handle at +once? +The exact configuration determines how well the autoscaling system +achieves its goals of maintaining latency SLAs without wasting resources. +For example, increasing the scale down delay prevents premature scale- +downs for spikey traffic, but could result in unnecessary spend after traffic +truly cools off. +7.2.1  +Concurrency and Batch Sizing +To properly operate a traffic-based autoscaling system, you need a strong +understanding of how much concurrent traffic each instance can handle. +Most model inference services can handle more than one request at a +time via batching. There are several approaches to batching: +• Static batching: The server waits until the batch is full before starting +inference. +• Dynamic batching: The server waits until the batch is full or a config- +ured amount of time has passed before starting inference. +• Continuous batching: The server continuously runs inference, swap- +ping in requests as slots become available. + + +7.2.1 Concurrency and Batch Sizing 187 +Inference engines like vLLM, SGLang, and TensorRT-LLM implement +robust continuous batching (or in-fl ight batching as TensorRT-LLM calls +it), where requests are batched at the token level. This minimizes latency +relative to static batching. +Figure 7.6: Static batching sets a fi xed batch size and waits for the batch to fi ll before +beginning inference, leading to long wait times for early requests. +Figure 7.7: Dynamic batching adds a cutoff time after which a batch is run whether or +not it is full. + + +188 Chapter 7: Production +Figure 7.8: Continuous batching operates at the token level, switching in new +requests as old requests fi nish. +Batch sizing trades off latency for throughput. Increasing the batch size will +produce more throughput overall, but each user’s latency will get worse. +Test performance across multiple batch sizes to fi nd the right fi t for your +model, instance, latency target, and budget. +This is controlled at the autoscaling confi guration level via the concurrency +target and at the replica level via the batch size, which should match. +Once every active replica reaches its maximum concurrency, the autoscal- +ing system knows to spin up more replicas. If enough replicas are kicking +off half-full batches, it’s time to scale back down. + 7.2.2 +Cold Starts +A cold start is the time it takes to spin up a new replica of a model. +The overall performance of an autoscaling system depends on its cold +start speeds. If you can’t spin up replicas fast, it’s hard to confi dently scale +down, leading to over-provisioning. + + +7.2.2 Cold Starts 189 +There are several factors that affect cold start times: +• GPU procurement: How quickly can you add the necessary GPUs to +your cluster and allocate them to the model? +• Image loading: How quickly can you load the container image onto the +newly procured instance? +• Model loading: How quickly can you load the model weights into the +container? +• Engine startup: How quickly can you start your inference engine, +including any compilation time? +Each of these factors needs to be optimized separately. +Figure 7.9: Each step in the cold start process adds to the overall timeline. +Unless you have a pool of warm nodes that you’re fl exing between mod- +els, GPU procurement speed is mostly a function of your cloud provider. +Section 7.3.1 covers procuring GPUs, and node start time is one of the +negotiable factors in a contract. +However, engineers can do a lot on loading images and weights and +starting containers and engines. +Loading images and model weights is a function of how quickly you can +write gigabytes, often hundreds of gigabytes, of data onto your instance. +There are two ways to load images and weights faster: make them smaller +or get more bandwidth. +Including only necessary steps and dependencies makes images smaller +and faster to build into containers, while quantizing model weights has the +additional benefi t of making them faster to load during cold starts. + + +190  Chapter 7: Production +For small models, the strategy used to be baking the weights into the +image to simplify caching and loading. However, now that most models +have dozens or hundreds of billions of parameters, the weights dwarf the +image and are better loaded separately. +Where you load weights from has a massive impact on the bandwidth. +If you’re loading from a third party like Hugging Face, you’re limited by +their egress speed. And storing your weights in an S3 bucket introduces +network latency and data transfer costs. +For loading multi-hundred-billion-parameter models, you need gigabytes +per second of bandwidth. The best way to get this is by loading over net- +work within a node from a source cached physically near the GPU instance +within the same datacenter. +Inference engines like vLLM and SGLang are fast to start up. But engines +like TensorRT-LLM and optimized models with PyTorch have a compila- +tion step that targets the specific hardware resources to build the model +inference engine. These compilations often take several minutes. +In these cases, caching built engines massively improves cold start times. +Both TensorRT-LLM and PyTorch have image caching mechanisms that +make this possible, though you’ll always need to load a cached engine +into an instance with exactly the GPU type, CUDA version, and software +dependencies as the environment that the engine was built in for it to run +properly. +7.2.3  +Routing, Load Balancing, and Queueing +Once there are multiple replicas online, the system needs to make a +decision about which requests to send to which replicas. There are two +types of components that make these decisions: +• Routers: A router works at the request level to determine the ideal place +to send a given request. A router answers “where should this request +go?” +• Load balancers: A load balancer works at the system level to even +out requests between multiple options. A load balancer answers “where +could this request go?” + + +7.2.3  Routing, Load Balancing, and Queueing  191 +In complex systems, there isn’t just one router and one load balancer. +Routing occurs throughout the stack with load balancers injected at key +points to keep system-wide performance stable. +Overall, you want to split load equally across replicas. However, routing +and load balancing are not as simple as saying, “Well, I have 3 replicas +and 12 requests, so let’s put 4 requests on each replica.” +Each request may have a different number of input tokens. If most +requests have 100 input tokens, a request with 10,000 will unbalance a +simple system. Some requests are also better handled by certain replicas. +Examples include: +• KV cache-aware routing: Direct a request to a replica that already has +a matching prefix in its KV cache. +• LoRA-aware routing: Direct a request to a replica that already has the +desired LoRA fine-tuned weights in memory. +Intelligent request routing uses information from the inference engine +and any orchestrators like NVIDIA Dynamo to route requests based on +sequence length, prefix, and LoRA needs. +Load balancing and routing are not enough. When an autoscaling sys- +tem receives more traffic than it can handle, it needs a way to hold onto +requests as it scales up more resources or waits for existing resources +to become available. +A queue is the infrastructure primitive for handling this situation. A standard +queue is a first-in, first-out system for excess requests, though you can do +a more complex implementation like a priority queue to, for example, give +paid users priority over free users in high-traffic scenarios. +As new replicas come online, ensure that the queue sees them and +requests don’t continue waiting for the existing replicas. Each new replica +should immediately be assigned up to its concurrency limit in queued +traffic once active. + + +192  Chapter 7: Production +7.2.4  +Scale to Zero +Advanced autoscaling systems implement a mechanism for scale to zero, +where the system can scale down to zero active replicas if there is no +traffic, then scale up automatically when traffic is received. +Scale to zero relies on two prerequisites: +• Fast cold starts: As users may be waiting live, cold starts must be as +fast as possible. +• Robust queueing: The system needs to be able to hold the incoming +requests until a replica is live. +Even with these capabilities, scale to zero is not a fit for all workloads. +Scale to zero is great for development, when testing is bursty and latency +for the first request is unimportant. And in production, scale to zero is +useful for applications that only get traffic periodically, like an agent that’s +only accessed during business hours in one country or an offline system +designed for daily batch processing jobs. +However, if you’re relying on scale to zero to keep costs low in a laten- +cy-sensitive application that gets light, unscheduled traffic, it’s probably a +sign that your AI application is not yet ready for dedicated infrastructure +and should use pay-per-token APIs until greater scale is reached. +7.2.5  +Independent Component Scaling +AI applications are increasingly built on multi-model, multi-stage com- +pound AI workloads where inference engineers need to coordinate multiple +steps to fulfill a single request. +These steps may have different hardware needs. A voice activity detector +model needs a far less powerful GPU than the transcription model it’s +chunking data for, while the LLM processing that transcript may need a +full node with multiple GPUs. And the scaling parameters for each step +in the pipeline also differ. + + +7.3 Multi-Cloud Capacity Management 193 +Figure 7.10: Independent component scaling gives each model access to appropriate +resources and individual scaling. +For these pipelines, you need to decompose autoscaling decisions and +scale each step individually to right-size resources per step and avoid both +bottlenecks and overprovisioning. +However, every model in the pipeline should run in the same cluster. If it +takes 10 milliseconds to send a message within a cluster and 50 millisec- +onds to send messages between clusters, that 40-millisecond difference +across a 5-step pipeline would be 20 percent of a one-second latency SLA. + 7.3 +Multi-Cloud Capacity Management +Autoscaling within a single cluster works up to a certain point. But high-vol- +ume deployments serving a global user base need thousands of GPUs +distributed around the world. +It’s straightforward to build multi-cloud inference as a collection of siloed +compute across different cloud providers. But in these setups, there’s +no way to use inter-cloud compute fl uidly, and moving workloads across +clouds is a tedious, error-prone process. +True multi-cloud inference requires building a multi-region, multi-provider +bin packing tool, which treats distinct pools of compute as fungible with + + +194 Chapter 7: Production +each other. Like Kubernetes within a single cluster, multi-cloud capacity +management must take a global view, enabling self-healing and global +scheduling. +Figure 7.11: A multi-cloud approach extends the idea of control and workload planes +to a multi-cluster, multi-region system. +Running true multi-cloud inference unlocks: +• Capacity: Pool capacity from multiple providers for greater and more +fl exible GPU access. +• Redundancy: Split inference across providers for resiliency against +outages. +• Latency: Run inference close to your end users to reduce network +latency overhead. +• Compliance: Run inference in compliance with data sovereignty and +other regulatory requirements. +Scaling from one cluster in one cloud to many clusters in many clouds +requires a new coordination layer. A multi-cloud architecture contains: +• Control plane: Handles model deployment and global scaling deci- +sions, receives real-time event streams. + + +7.3.1  GPU Procurement  195 +• Workload planes: Handles direct inference traffic and in-cluster scaling +decisions, reports utilization and demand. +This separation of responsibilities ensures that individual workload planes +can serve traffic independently. If something happens to the control plane +or any given workload plane, other workloads should be unaffected. +7.3.1  +GPU Procurement +There are a good number of companies in the business of providing +access to GPUs. The three major types are: +• Hyperscalers: Large cloud providers like AWS or GCP. +• Neoclouds: GPU-focused clouds like Coreweave or Nebius. +• Resellers: Secondary markets like SF Compute Company. +Players in this space vary in their capacity, availability, and reliability. You +generally pay a premium for hyperscalers and across all providers there +is a tradeoff between cost and factors like uptime SLAs, support, regional +availability, instance configuration, and cluster sizes. +The first challenge is securing capacity. It is often difficult to get your hands +on the GPUs you need, especially the latest hardware. Large clusters are +also hard to get, with relatively few players offering blocks of hundreds +of nodes. +Many cloud providers allocate the majority of in-demand GPUs to their +largest customers on long-term reservations. You may need to work across +multiple cloud providers to get the GPUs you need in the right regions. +Cloud GPUs can be procured via three different mechanisms: +• Reserved: Blocks of hundreds or thousands of GPUs are reserved for +months or years at discounted rates. +• On-demand: Individual instances are available as needed up to a given +quota for a relatively high per-hour cost. +• Spot: Discounted on-demand instances that can be pre-empted at an +agreed-upon notice period, often minutes. + + +196  Chapter 7: Production +Large-scale inference generally uses a blend of GPU sources, with a +baseline of low-cost reserved instances and a mix of on-demand and spot +for handling peaks in traffic. These GPUs are distributed across multiple +clusters worldwide for proximity to end users. +7.3.2  +Geo-Aware Load Balancing +Successful AI applications have users all over the world. Just like an +individual cluster has a load balancer to ensure that every GPU in the +cluster receives the right amount of traffic, a multi-cluster system needs +a global load balancer. +You don’t want a user request sitting around in some queue when there +is spare capacity elsewhere, but you also don’t want to make a habit of +sending a request from Singapore to a server in San Francisco. +As a rule of thumb, it takes five milliseconds for a request to pass through a +time zone. So, sending data from New York to San Francisco takes fifteen +milliseconds one way. Given how small latency budgets are, it’s important +to run workloads as close to end users as possible. +7.3.3  +Building for Reliability +GPUs are infamous for their high failure rate in production. Every engineer +who has done a large-scale training run knows that they need to account +for the eventuality that hardware will fail. +For example, in their Llama 3 paper, Grattafiori and colleagues revealed +that while running 16,000 GPUs for a period of 54 days, the Llama team +experienced 419 unexpected interruptions, primarily due to hardware fail- +ure. This works out to approximately one failure per 50,000 GPU-hours. +50,000 hours might sound like a long time, but running a single node of +eight GPUs for inference for an entire year is over 70,000 GPU-hours. +Inference engineers should expect hardware failure. + + +7.3.3 Building for Reliability 197 +Figure 7.12: Root cause of failures when training Llama 3, adapted from “The Llama 3 +Herd of Models” (Grattafi ori et al., 2024). +GPU health is a node-level concern. When a single GPU fails, other GPUs +on the node often fail next or need to be taken offline for maintenance. +Proactively noting failures, cordoning nodes, and cycling pods keeps indi- +vidual clusters healthy. +GPU failures aren’t the only thing that can bring down inference. Cloud pro- +viders have scheduled maintenance and their own unscheduled downtime. +Every layer of infrastructure must be reinforced to provide high reliability. +Multi-cloud inference brings two new approaches to high reliability: +• Active-active: A high-availability posture where multiple regions or +clusters actively serve live traffic at the same time. If any plane fails, +traffic seamlessly continues on the others. +• Active-passive: A failover posture where a “hot standby” cluster or +region is kept ready but idle. If the active plane fails, traffic is cut over +to the passive plane. + + +198  Chapter 7: Production +When individual clusters, regions, or cloud providers go down, seamlessly +failing over to another workload plane keeps reliability high and latency low. +7.3.4  +Security and Compliance +Cloud infrastructure has been a hot topic for security and compliance +departments for more than twenty years. For AI models to power mis- +sion-critical applications, inference must be both secure and compliant. +Security and compliance conversations generally center around three +areas: +• User data: Security and compliance departments want to ensure all +data, including user inputs and model outputs, is protected. +• Model weights: For companies with fine-tuned or proprietary models, +the weights are an invaluable trade secret. +• Infrastructure: GPUs themselves and access to intelligence are both +targets for abuse. +One of the easiest decisions you can make to improve security is to simply +not store user inputs or model outputs. This may not be possible – you +might have logging requirements or user agreements to retain usage data +for future model training – but if you don’t need to retain user data, you +can reduce your attack surface. +Securing AI inference workloads and associated data is similar to secur- +ing any other containerized workload. Data encryption, container secu- +rity, network and access controls, and workload isolation, all validated by +extensive third-party penetration testing, remain the gold standard. +Increasingly, inference engineers need to support applications running +in regulated industries and compliance-heavy regions. One place where +multi-cloud infrastructure helps is that in order for your application to com- +ply with a certification like SOC 2 Type II or a regulation like HIPAA, your +providers generally must also be compliant. In this case, being able to +move workloads to compliant providers is useful. +Another benefit of multi-cluster infrastructure is running one model across +multiple regions. Certain industries and countries have data residency + + +7.4  Testing and Deployment  199 +requirements, where user data from their country cannot be processed +on servers in a different country. +For example, having one cluster in a provider near Toronto and another +in a provider near New York lets you keep Canadian data in Canada and +American data in the United States while providing minimal latency over- +head to users across the geographic region. +7.4  +Testing and Deployment +In addition to any replica-level testing and benchmarking performed while +configuring the inference engine, it’s important to test systems end-to-end +before deployment. +There are several strategies for testing inference: +• Manual testing: Writing scripts (or clicking buttons) to send synthetic +traffic to an inference service. +• Load testing: Automatically sending a large volume of traffic to test a +system’s ability to scale and maintain performance. +• Shadow traffic: Copying live traffic to test deployments to measure +performance under real-world conditions. +Testing inference services is expensive. It takes engineering time to con- +figure the tests and measure the results, and it takes GPUs to run infer- +ence for the test traffic. To a degree, that’s just the cost of doing business, +but think carefully about how to minimize testing expenses. For example, +shadow traffic testing could start with copying a random sample of pro- +duction traffic, followed by a shorter-duration load test. +When testing, keep in mind that AI product usage generally fluctuates on +daily and weekly cycles. +Once you’re confident in the performance and stability of your updated +inference system, it’s time to deploy to production. + + +200 Chapter 7: Production + 7.4.1 +Zero-Downtime Deployment +Inference engineers use high-availability deployment strategies to avoid +downtime. +A traditional high-availability design is a blue-green deployment. In this +setup, there are two identical environments: the original blue deployment +and a new green deployment running the updated service. Once the green +environment is ready, the full traffic load cuts over from the blue to the +green environment, with the blue environment staying ready for rollback +in case of issues. +However, blue-green is not well suited for large scale inference workloads +due to the same GPU capacity and cost issues that make large-scale test- +ing difficult. If the blue deployment is using 100 GPUs, the green deploy- +ment requires another 100 GPUs before traffic can cut over. +Instead, inference engineers can get similar benefi ts with lower GPU over- +head using canary deployments. Inspired by the canaries that were used +to detect gas in coal mines, a canary deployment catches errors before +they affect large numbers of users. +Figure 7.13: Iteratively shifting traffic over to the new deployment prevents multiple +issues during inference service updates. +A canary deployment is a 4-step process: +1. Build a new deployment of the inference service and get it ready to +handle incoming requests. + + +7.4.2  Cost Estimation  201 +2. Direct a small percentage of the incoming live traffic to the new service. +3. Monitor the new service and ensure it is handling traffic correctly. Revert +if there are any issues. +4. Gradually increase traffic, while monitoring for issues, until the new +deployment handles 100 percent of traffic. +These canary deployments can be rolled out quickly, with just a few min- +utes of traffic ramp, or ramped slowly to ensure stability at each stage. +And with autoscaling, canary deployments don’t increase cost much at +scale because reducing traffic to the production system causes it to scale +down some replicas. +With autoscaling, the new deployment will default to the minimum number +of replicas when there is no traffic. Throughout the canary deployment +process, ensure that the new deployment has enough active replicas to +properly handle requests. Otherwise, users will see a latency spike as +their requests are queued until autoscaling completes. +7.4.2  +Cost Estimation +Switching from consuming tokens from a public API to doing your own +inference on dedicated GPUs requires changing how you think about cost. +Cost on public APIs is simple: a price per million tokens times the number +of tokens you use. There are a couple of variables – cache hits versus +cache misses for input tokens, discounts for high-volume users – but cost +remains a linear function of usage. +One motivation for investing the time and effort in inference engineering is +to take control of your unit economics and escape per-token pricing. But +it’s a difficult mental transition. +The blessing and curse of dedicated inference is that cost is now a function +of many variables. This is good because it gives you control, but it makes +estimation difficult. Factors that affect cost include: +• Batch sizing: Is the deployment optimized for latency with low batch +sizes or throughput with high batch sizes? + + +202 Chapter 7: Production +• Traffic patterns: Is traffic consistently saturating active GPUs, or is +capacity going spare? +• Sequence lengths: How many input and output tokens do requests +have both on average and in outlier cases? +Given this complexity and the difference in cost between input and output +tokens, it’s generally more productive to convert your token price into a +total cost and compare that to dedicated instead of trying to reverse engi- +neer a per-token price from what you pay for GPUs. +Figure 7.14: An equation for estimating the total cost of using per-token APIs in a +product. +Figure 7.15: An equation for estimating the total cost of using dedicated deployments +in a product. +Cost estimates should use a long time horizon, ideally at least a week, to +smooth out variations in usage. + + +7.4.3  Observability  203 +The other factor to consider in dedicated deployments is the cost of +engineering time spent building and maintaining inference systems. This +investment, while justified in increased reliability, security, and control, +should be added to the GPU costs to form a complete picture around total +cost of ownership (TCO) for inference. +7.4.3  +Observability +Inference is mission-critical, so it must be monitored like any other mis- +sion-critical component of an application, with alerting, logs, and observ- +ability built at the right level of abstraction. +The first question is what to monitor. Inference observability includes +measuring: +• Total volume: The number of requests that a model deployment is +receiving. +• Request and response sizes: The input and output sequence lengths +for the requests being processed. +• Response codes: The count of 2XX, 4XX, and 5XX response codes +issued by the model server. +• Latency: Metrics like time to first token, tokens per second, and end- +to-end latency on a P50, P90, and P99 basis. +• Replica count: The number of instances actively serving traffic, and +the number of instances starting up, if any. +• Utilization: The amount of utilization across CPU, host memory, GPU, +and GPU memory. +• Queue depth: For systems with asynchronous traffic, the number of +requests enqueued and waiting to be processed. +These metrics are interdependent. A spike in latency could come from +request volume, but it could also come from long input sequences. Seeing +these metrics together lets inference engineers understand not only what +is happening but also why. + + +204 Chapter 7: Production +When things go wrong, inference engineers need information to fi x issues. +Logs, both server logs and audit logs showing changes to an inference +service, deliver that information in real time. +Observability cannot be siloed. When you build observability for inference, +build it with deep integration into existing observability and alerting tooling +– Grafana, Datadog, PagerDuty, Sentry – to put inference information in +context with the rest of the application. + 7.5 +Client Code +Inference engineering draws on many technologies, from CUDA to Kuber- +netes. But there’s one critical area that’s often overlooked when optimizing +for latency and building for scale: client code. +There are two sides of a call to an inference service: +• Client: The browser, agent, or application making a request to the +inference engine. +• Server: The inference service that handles the client request and +returns the model results. +The industry standard for client code is the OpenAI SDK, which supports +a wide range of compatible providers in addition to OpenAI’s own models. +Popular AI engineering frameworks and libraries like LangChain, Vercel AI +SDK, LiteLLM, LlamaIndex, and dozens more can also serve as clients. +Whether you’re using an existing library or your own code, there is the +potential for latency overhead or throughput bottlenecks. And for real-time +applications, you may need a protocol other than HTTP, like WebSockets, +to deliver a continuous connection. +Figure 7.16: On-server inference time is just a fraction of the end-to-end latency for a +given request. + + +7.5.1  Client Latency Overhead  205 +7.5.1  +Client Latency Overhead +Depending on the client’s internet connection and the protocol used, +establishing a session between a client and server takes a few dozen +milliseconds. +In a high-performance system with a 300-millisecond P95 end-to-end +latency SLA, a TLS handshake costs at least ten percent of that latency +budget before inference even starts. Future requests from the same client +should save time by re-using existing sessions. +Session re-use is not a new idea by any means, and tools like the OpenAI +SDK provide it silently under the hood. However, when building your own +client for non-standard modalities, follow best practices like session re-use. +7.5.2  +Asynchronous inference +Some systems are built for throughput, not latency. Use cases like bulk +document processing and corpus embedding are not latency sensitive, +so it makes sense to switch to asynchronous jobs. +Asynchronous requests are a “fire and forget” approach to executing infer- +ence. +Ordinary synchronous inference requests have a timeout, generally of a +few minutes, after which the request will fail. Asynchronous jobs fix this +by immediately acknowledging the request and later returning the result +of the asynchronous job to a webhook supplied in the original request. +Asynchronous jobs still have time limits, but these requests are usually +measured in hours, not minutes. Along with strong server-side queuing, +asynchronous requests make high-throughput, latency-insensitive sys- +tems more robust and efficient. +7.5.3  +Streaming and Protocol Support +Streaming makes applications feel instant. For language models, stream- +ing text output over HTTP is sufficient. But for other modalities, especially + + +206 Chapter 7: Production +live voice and video, both input and output streams need to be able to +carry more data. +Figure 7.17: One-time HTTP requests and responses are a good fi t for use cases like +text chat, but not for continuous streaming. +The two most common bi-directional streaming client-server connection +protocols are: +• Websockets: For streaming use cases where strong schema enforce- +ment is not required. +• gRPC: For well-defi ned service-to-service communication. +WebSockets are useful for transmitting unstructured and real-time data, +like audio, where the server receiving the request can parse it and pro- +cess it downstream. With WebSockets, a server can support up to a +fi xed, developer-confi gurable number of clients; when that concurrency +is reached, new connections cannot be established and must wait until +either a slot is free or another replica scales up. +Figure 7.18: WebSockets establish a continuous connection for unstructured data like +audio streams. + + +7.5.3 Streaming and Protocol Support 207 +Similar to WebSockets, gRPC enables bi-directional streaming support, +but for structured data. Requests transmitted via gRPC must follow a +predefi ned schema, which takes away the load of having to parse the +input. This additional validation layer makes gRPC slightly slower than +WebSockets. +Figure 7.19: gRPC establishes a continuous connection for well-defi ned service-to- +service communication. + + +208  Chapter 7: Production +7.6  +Production Inference with Baseten +This book contains everything I’ve learned about inference in four years +of working at Baseten. +Baseten is an AI infrastructure company founded in 2019. At Baseten, we +are focused on highly performant and highly available inference for both +open models and custom models. We also offer a platform for pre-training, +post-training, and reinforcement learning. +We run inference for the world’s fastest-growing startups and most inno- +vative enterprises, including Cursor, World Labs, Notion, OpenEvidence, +Clay, Abridge, Gamma, Ambience, Writer, and hundreds more. +At Baseten, we focus on four essential pillars to deliver the fastest mis- +sion-critical inference: +• Performance: Consistent low latencies at scale powered by the +Baseten Inference Stack. +• Infrastructure: Reliable multi-cloud deployments, fast and granular +autoscaling, and robust security. +• Tooling: An intuitive and productive developer experience with logging, +observability, and programmatic access. +• Applied expertise: Hands-on-keyboard implementation and assistance +from forward deployed engineers. +We would be honored to provide fast, reliable inference for your AI-pow- +ered products. +Also, we are continuously hiring for all roles across engineering, sales, +marketing, and operations. To learn more, visit https://baseten.com/ +careers. diff --git a/library/Inference Engineering/Chapter-10-Appendix-A-Inference-Glossary/Chapter-10-source-text.md b/library/Inference Engineering/Chapter-10-Appendix-A-Inference-Glossary/Chapter-10-source-text.md new file mode 100644 index 0000000..dfb19c1 --- /dev/null +++ b/library/Inference Engineering/Chapter-10-Appendix-A-Inference-Glossary/Chapter-10-source-text.md @@ -0,0 +1,598 @@ +# Inference Engineering — Chapter-10: Appendix A: Inference Glossary +- **Source**: /library/Inference Engineering/source-file.pdf +- **PDF pages**: 211–232 +- **Pages without text**: 211, 212, 232 + +--- + + +APPENDIX A +Inference Glossary + + + + + +Inference Glossary  211 +Activation function: A (mostly) differentiable nonlinear function like ReLU +inserted between linear layers to prevent multi-layer neural networks from +collapsing into a single matmul. +Active–active: A high‑availability posture where multiple regions/clusters +actively serve live traffic at once. If any plane fails, traffic seamlessly +continues on the others. +Active–passive: A failover posture where a “hot standby” cluster or region +is kept ready but idle. If the active plane fails, traffic is cut over to the +passive plane. +Ada Lovelace (architecture): NVIDIA’s graphics‑oriented GPU archi- +tecture, released alongside Hopper in 2022. Useful for small models and +cost‑sensitive workloads, not suited for large‑scale LLM inference. +Agent: An AI application that takes action rather than just providing +information. Agent workflows usually rely on multiple inference calls, +often across multiple models and modalities, and require access to +tools. +AI‑native application: A product where the core UX and value depend on +generative models. Inference choices are downstream of app constraints: +modality, latency budget, unit economics, and usage patterns. +Ampere (architecture): An older NVIDIA GPU architecture still used in +legacy or small-scale deployments. Hopper and Blackwell architectures +generally outperform Ampere on both raw speed and cost at scale. +Application Programming Interface (API): A structured interface for +sending requests and receiving responses. Inference engines expose an +API for making queries to models. +Arithmetic intensity: Operations performed per byte moved for a given +algorithm. When compared to a GPU’s ops:byte ratio, arithmetic intensity +indicates whether a kernel is compute bound or memory bound. +Attention: The core transformer mechanism relating a token to prior +tokens via Q/K/V projections and softmax. Attention is a primary target +for optimization due to its compute and memory demands. + + +212  Inference Glossary +Automatic Speech Recognition (ASR): Audio‑in, text‑out transcription +models (e.g., Whisper). Decoder work dominates runtime and benefits +from LLM‑style optimizations and in‑flight batching. +Autoregressive token generation: Iterative generation of tokens where +each token depends on each previous token. Autoregressive token gen- +eration is split into two phases: a prefill phase where input tokens are +processed, and a decode phase where output tokens are generated. +Autoscaling: Scaling the number of replicas serving a given model up +and down automatically based on traffic or utilization. Autoscaling matches +capacity to demand, maintaining latency SLAs and minimizing wasted +spend. +Autoscaling window: The rolling time horizon used to decide scale‑up/ +scale‑down actions. Longer windows keep replica counts steady; shorter +windows react faster to spikes. +B200: NVIDIA Blackwell‑based datacenter GPU with 192 GB of VRAM, 8 +TB/s of memory bandwidth, and 5 petaFLOPS of FP8 compute. +B300: NVIDIA Blackwell‑based datacenter GPU with 288 GB of VRAM, 8 +TB/s of memory bandwidth, and 5 petaFLOPS of FP8 compute. +Bandwidth: The amount of data per second that can pass through mem- +ory like VRAM or an interconnect like NVLink. +Baselines: Initial, carefully recorded measurements of performance and +quality before applying optimizations. Baselines enable clear attribution +of gains or regressions. +Basic Linear Algebra Subprograms (BLAS): A standard interface for +fundamental operations in linear algebra. +Batch: Process multiple inputs simultaneously, common for LLM infer- +ence. +Batch sizing: A core latency‑throughput lever for inference engines. +Larger batches improve total throughput but worsen per‑user latency. + + +Inference Glossary  213 +Benchmark (intelligence): A measurement of a model’s ability to answer +questions correctly or take appropriate actions (e.g., MMLU). +Benchmark (performance): A measurement of an inference service’s +latency and throughput for a given model with a defined workload. +BF16: A 16‑bit floating‑point format with larger exponent than FP16, useful +in training and sometimes inference. Higher dynamic range helps preserve +outliers. +Bin packing (multi‑cloud): The practice of treating heterogeneous pools +of GPUs across clouds, regions, and clusters as a single schedulable +resource, enabled by multi-cloud capacity management infrastructure. +Blackwell (architecture): NVIDIA’s late‑2024 GPU generation featuring +FP4 support, microscaling formats (MXFP8, MXFP4, NVFP4), and high +memory bandwidth. +Blue‑green deployment: Two parallel production environments (“blue” +and “green”); shift traffic between them for zero‑downtime deploys and +quick rollback. +Cache‑aware routing: Steering requests to replicas that already hold +matching prefixes or required LoRAs. Higher cache hit rates yield lower +TTFT. +Canary deployment: A new production deployment that starts with a +small share of live traffic to a new deployment to validate stability and +performance. Over time, the new deployment absorbs all production +traffic. +Causal language model (CLM): A decoder‑only transformer that predicts +the next token given the prior context. All generative LLMs in this book +are CLMs. +Central Processing Unit (CPU): A general-purpose processor optimized +for sequential workloads. CPUs are used for orchestration, scheduling, +networking, and preprocessing, but rarely handle generative AI inference +directly. + + +214  Inference Glossary +Chat template: The model‑specific formatting and serialization of mes- +sages (roles, separators, beginning/end of sequence tokens). +Chunked prefill: Splits long inputs into chunks and overlaps prefill with +decode or other work, preventing single long sequences from monopo- +lizing resources. +Classifier-free guidance: Balances unconditional and prompt‑condi- +tioned denoising passes on each step of image generation. Lower guid- +ance enhances creativity; higher guidance enforces prompt adherence. +CLIP (text encoder): A text/image encoder used in earlier image pipelines +(e.g., SDXL). Modern systems often swap in full LLMs for stronger prompt +understanding. +Closed model: A proprietary model where weights are unavailable, like +GPT-5, Claude Sonnet, or Google Gemini. +Cold start: The time from scaling a replica from zero to its first successful +response (steps include GPU provisioning, container startup, model load, +inference engine compilation). +ComfyUI: A workflow tool for assembling image pipelines (base model, +refiner, LoRAs, ControlNets). Encourages modular, swappable components. +Compute‑bound: An algorithm limited by available FLOPS rather than +memory bandwidth. LLM prefill and image/video generation are usually +compute bound. +Context Parallelism (CP): Replicates weights across GPUs and partitions +the attention context. Essential for video models where attention works +across a massive latent space. +Context window: The maximum number of tokens that a model can +process across input, reasoning, and output for a single request. +Continuous batching (in‑flight): Token‑level interleaving of requests +so GPU slots are always utilized. Minimizes per‑user latency penalties +of batching. + + +Inference Glossary  215 +Control plane (multi‑cloud): Global orchestrator for deploying models +and allocating resources. +Core (CUDA): A general-purpose arithmetic unit that executes a wide +range of scalar and element-wise operations. +Core (Tensor): A specialized hardware unit optimized for mixed-precision +matrix multiply-accumulate (MMA) operations. Tensor Cores are the most +important type of compute for inference. +Cross‑attention: Conditioning one sequence (Q) on another’s K/V (e.g., +text conditioning images). Common in multimodal and denoising pipelines. +cuBLAS: CUDA’s BLAS implementation offering high‑quality GEMM and +related primitives. +CUDA: NVIDIA’s programming model and platform for GPU kernels, +graphs, memory, and execution. +CUDA driver: A low-level interface between the application and the GPU +hardware to manage memory and execution. +CUDA graph: A directed acyclic graph (DAG) of kernels and other GPU +operations for optimizing repeated workflows. +CUDA kernel: A user-defined function that executes parallelized code +on the GPU. +CUDA runtime: A developer-facing API for launching kernels and man- +aging memory. +cuDNN: Primitives for building deep neural networks in CUDA. +CuTe: A domain-specific C++ template library that abstracts tiled tensor +operations to help developers compose precision-aware, hardware-opti- +mized GEMM and fused kernels. +CUTLASS: A CUDA C++ template library that provides building blocks for +writing high-performance, architecture-tuned GEMM and related kernels. + + +216  Inference Glossary +Data sovereignty: Legal constraints around where model inputs and out- +puts are processed and stored geographically. +Decode: The memory-bound phase of LLM inference where the autore- +gressive generation loop emits one token per forward pass. +DeepGEMM: A library of clean and efficient GEMM kernels created by +the DeepSeek AI team with strong performance in FP8. +Denoising model: The heart of diffusion pipelines that iteratively refines +latent noise into an image or video. +Diarization: Segmenting audio by speaker (“who spoke when”); often +paired with VAD in ASR pipelines. +Diffusers (library): Reference implementations for image and video gen- +eration pipelines. +Disaggregation: Separating prefill and decode onto independently scaling +engines running on separate hardware resources. +Distillation: Training a smaller student to emulate a larger teacher model +based on probability distributions, not just outputs, to retain model behavior +on fewer parameters. +Docker: Containerization technology for building standardized packages +of inference services with their dependencies. +Dockerfile: A human-readable file with well-specified, machine-interpre- +table instructions for creating an image. +Dynamic batching: Dynamic batching starts a batch when the batch is full +or a short timer elapses, whichever comes first. Balances latency stability +with utilization; superseded by continuous batching for LLMs. +Dynamic range (quantization): The range of absolute values that can +be represented in a number format. Floating-point numbers have a higher +dynamic range than integers with the same number of bytes thanks to their +exponent-mantissa structure. + + +Inference Glossary  217 +EAGLE (speculation): A small, purpose‑built draft model trained to con- +sume hidden states and propose multiple tokens for high acceptance rates +in speculative decoding. +Elo (quality meta‑metric): Head‑to‑head win‑rate style scoring to compare +model quality. Useful directional signal beyond intelligence benchmarks. +Embedding model: Encodes text or image input into fixed‑dimensional +vectors for semantic similarity, used in RAG and agent memory. Modern +variants often use LLM backbones and Matryoshka representations. +Encoder: Network that converts raw inputs into internal representations +(e.g., audio features in Whisper). Paired with a decoder in encoder‑de- +coder models. +Evals: Task‑specific tests that mirror real-world use cases for a model, +used for product-specific model intelligence testing. +Expert Parallelism (EP): Shards experts of an MoE across GPUs; each +GPU contains multiple full experts. Increases total throughput with low +inter-GPU communication overhead. +Few‑step image generation: Models that produce usable images in eight +or fewer steps. Eighty to ninety percent faster but with noticeable quality +tradeoffs; strong fit for real‑time applications. +Feynman (architecture): A future NVIDIA generation after Rubin. Details +are limited; expect continued emphasis on low‑precision and memory +bandwidth. +Fine‑tuning: Adapts a pretrained base to a domain, often enabling much +smaller models to meet quality needs. +FlashAttention: A series of optimized attention kernels that minimize +memory traffic. FlashAttention 3 is written for Hopper, FlashAttention 4 +targets Blackwell. +Floating-point data formats: Precisions like FP16, FP8, and FP4 used +in inference with high dynamic range and an exponent-mantissa structure. + + +218  Inference Glossary +FLOPS: Floating‑point operations per second, typically measured on +Tensor Cores. +Foundation model: A model trained on broad data that serves as a base +for multiple downstream tasks. Foundation models (e.g., GPT, Claude, +Llama) are typically fine‑tuned or used directly via prompting. +Function calling: Also known as tool calling or tool use, a model is given a +set of available functions along with a prompt and returns a structured out- +put including both selected functions and arguments for those functions. +GB200: An NVIDIA superchip that pairs a Grace CPU with a B200 GPU via +high-bandwidth NVLink chip-to-chip connection. GB200s are used in rack- +scale NVLink systems like the NVL72 and are useful for KV cache offload- +ing, LoRA swapping, and other techniques that benefit from NVLink-C2C. +General matrix-matrix multiplication (GEMM): An algorithm in BLAS +and the key operation for inference. +Generative AI: A class of models that, in contrast to predictive ML models, +create new content across modalities (text, images, audio, video, code) +by learning the underlying patterns of training data. +Generative Pretrained Transformer (GPT): A family of large language +models for text generation created by OpenAI. +GH200: An NVIDIA superchip that pairs a Grace CPU with an H200 GPU via +high-bandwidth NVLink chip-to-chip connection. GH200s are used in rack- +scale NVLink systems like the NVL72 and are useful for KV cache offload- +ing, LoRA swapping, and other techniques that benefit from NVLink-C2C. +Goodhart’s Law: “When a measure becomes a target, it ceases to be a +good measure.” +GPU node: A standard chassis of 8 interconnected GPUs with NVLink +and NVSwitch. +Grace CPU: ARM-based NVIDIA CPU with high-bandwidth chip-to-chip +interconnects between the CPU and GPU. Used alongside Hopper and +Blackwell GPUs. + + +Inference Glossary  219 +Graphics Processing Unit (GPU): A highly parallel processor originally +designed for graphics rendering and now widely used for training and +inference of generative AI models. +gRPC: Structured, schema‑first bidirectional streaming protocol. +Head (attention): One independent attention computation within a layer. +High-Bandwidth Memory (HBM): The memory used for VRAM on data- +center GPUs. Recent generations include HBM3, HBM3e, and HBM4. +Hopper (architecture): NVIDIA’s 2022 GPU generation featuring FP8 +support and async programming features. +Hyperscaler: Generalized cloud service providers like AWS and GCP. +Image generation pipeline: Foundation models for image generations +are pipelines of multiple models: a text encoder, an iterative denoiser, +and a VAE. +Inference: Serving AI models in production. +Inference engine: A high‑performance runtime (vLLM, SGLang, Tensor- +RT‑LLM) with support for optimization techniques like batching, caching, +quantization, and speculation. +InfiniBand: Inter‑node interconnect for scaling inference and training +across multiple nodes. While InfiniBand bandwidth is higher than alterna- +tives like Ethernet, it is substantially lower than NVLink. +Input sequence: The tokens provided to a model as part of a request, +processed during the prefill phase of inference. +Input Sequence Length (ISL): The number of tokens in the input +sequence for a given request. +Instance (cloud): The provisioned virtual machine that includes GPU(s), +CPU and RAM resources, storage, networking, and interconnect. + + +220  Inference Glossary +Integer data formats: Number formats like INT8 and INT4 with limited +dynamic range. +Inter‑token latency (ITL): Time between generated tokens during decode. +Converts to perceived TPS (e.g., 2 milliseconds ITL equates to 500 TPS). +In‑flight batching: See continuous batching. Token‑level interleaving for +high utilization with stable latency. +Iterative denoising (diffusion): Start from noise and progressively refine +into an image or video in latent space. +Jitter traffic (bench): Adding randomness to arrival times and sequence +shapes to more closely mirror real traffic than uniform or bursty synthetic +loads. +Kernel fusion: Taking two or more kernels and re-implementing them +into a single kernel that handles both operations, avoiding unnecessary +round-trips through memory. +KV cache: Stored K/V tensors for each token to avoid recomputing atten- +tion, turning the attention equation from a quadratic-time to a linear-time +operation. +L0/L1/L2 caches (GPU): On‑chip cache memory hierarchy for instruc- +tions, shared memory, and global cache. +Large Language Model (LLM): A type of generative AI model that takes +a text prompt and returns a new sequence of text. Many famous gener- +ative AI model families, including GPT, Claude, Llama, and DeepSeek, +are LLMs. +Latency percentiles: Measuring latency on a percentile basis (P50/P90/ +P95/P99) for awareness of both the average and the worst-case user +experience. +Latent consistency: A few‑step strategy that predicts target latents +directly, possibly repeated for refinement. Very fast; lower fidelity than +full diffusion. + + +Inference Glossary  221 +Latent space (images/videos): Lower‑dimensional representation where +denoising occurs (e.g., 128×128). +LLM: Large language model (e.g., GPT-5, Llama, DeepSeek). +Load testing: Sending sustained high traffic to probe throughput limits, +queue behavior, and autoscaling. +Local (edge) inference: Running inference on end-user devices like +phones and computers. +Logit biasing: Nudging or constraining token probabilities to steer struc- +tured outputs (e.g., JSON/tool calls). Applied post‑logits before sampling. +Logits: A vector of non-normalized probabilities, one per token in the +model’s vocabulary, generated in each forward pass during decode. +Lookahead decoding: Constructs n‑grams during inference to enable +draft token prediction without a separate model. +LoRA: Low-rank adaptation, a lightweight fine-tuning method that pro- +duces small changes to models. Inference services often need to swap +between thousands of LoRAs for a single foundation model. +Machine learning (ML): Predictive modeling for tasks like classification and +trend forecasting, as opposed to generative AI which creates novel outputs. +Matmul: Matrix multiplication. +Matryoshka representations (embeddings): Nested vector schemes +allowing variable dimensionality where the early part of the vector encodes +more semantic meaning. Allows tradeoffs between vector size and embed- +ding quality. +Medusa (speculation): Adds extra decoder heads via fine‑tuning to gen- +erate multiple draft tokens per pass. +Microscaling formats: Floating-point data formats like MXFP8, MXFP4, +and NVFP4 that use blockwise quantization with small-block scale factors +(e.g., every 32 elements) to improve accuracy. + + +222  Inference Glossary +Mixture of Experts (MoE): A model architecture where linear layers of +weights are separated into sparse experts. A router activates a subset of +experts for each forward pass. +Model parallelism (overview): Splitting work across GPUs via Tensor, +Expert, or Pipeline Parallelism. Parallelism strategy depends on model +size, topology, and latency versus throughput goals. +Multi‑cloud capacity management: A global scheduler placing work- +loads across providers and regions. +Multi‑Instance GPU (MIG): A capability in larger Ampere, Hopper, Black- +well, and Rubin GPUs where the GPU can be carved into up to eight slices +of memory and seven slices of compute. +Multi‑node inference: Scaling across two or mode nodes using InfiniBand +when one node of eight GPUs doesn’t have enough VRAM for weights, +activations, and KV cache. Requires appropriate parallelism strategies, +either PP or EP between as TP uses too much all-to-all communication +for InfiniBand. +Neocloud: Specialized cloud service providers focused on GPUs like +Coreweave and Nebius. +Neural audio codec: A learned encoder that compresses audio into +tokens and paired decoder that turns tokens back into audio. +NIM: A pre‑packaged, containerized microservice for a specific model +created by NVIDIA. +Node: The physical 8‑GPU base unit with NVLink/NVSwitch. Multi‑node +adds InfiniBand between nodes. +NVFP4: NVIDIA’s 4‑bit floating‑point microscaling number format with dual +scale factors and blockwise quantization with a block size of 16. +NVIDIA Dynamo: An open‑source distributed serving platform for KV +reuse, disaggregation, and multi‑GPU/multi‑node orchestration. + + +Inference Glossary  223 +NVL72: A rack‑scale Blackwell system interconnecting 72 GPUs and 36 +CPUs. Purpose‑built for serving very large models with extreme through- +put. +NVLink: A one-to-one communication layer between GPUs, up to 1800 +GB/s on Blackwell and 900 GB/s on Hopper. +NVSwitch: An all-to-all communication layer on top of NVLink for coordi- +nation among all GPUs in a node. +N‑gram speculation: Uses observed n‑grams from prefill to propose long +draft sequences during decode. Extremely effective for code completion. +Offline inference: Asynchronous batch processing of large jobs, opti- +mized for throughput and cost over per-request latency. +Omni‑modal: Models that accept multiple modalities of inputs (text, +images, video, audio) and produce multiple modalities of output. +Online inference: Real-time serving of requests, optimized for tight +latency budgets. +ONNX: An intermediate representation and runtime for models. +Open model: A model whose weights are freely available, like Llama, +DeepSeek, or Whisper. +Ops:byte ratio (GPU): Peak operations per byte of memory bandwidth for +a GPU at a given precision. Compare with arithmetic intensity to diagnose +bottlenecks. +Output sequence: The tokens generated by a model during the decode +phase of inference. +Output Sequence Length (OSL): The number of tokens in the output +sequence generated by a model for a given request. +Out‑of‑memory error (OOM): A common failure where the GPU runs out +of VRAM to load weights or execute inference. + + +224  Inference Glossary +PagedAttention: An optimization for attention where KV blocks are stored +in fixed‑size pages to improve performance, especially with long context. +PCIe (GPU form factor): A form factor for datacenter GPUs that uses +standard PCI express slots for connection. PCIe GPUs often have lower +base specs and fewer interconnect options than SXM variants of the same +GPU. +Perceived TPS: Tokens per second observed by a single user during +streaming output. This latency metric is a more specific term for what +people usually mean when they say TPS. +Pipeline Parallelism (PP): Splits layers into stages across GPUs. While +acceptable for multi‑node with dense models; PP introduces bubbles in the +pipeline where some GPUs are idle while waiting for other steps to finish. +Prefill: The compute-bound phase of LLM inference where the input +sequence is processed and the KV cache is built. +Prefix caching: Reuses KV for shared prefixes across requests to skip +prefill. Majorly improves TTFT for code completion, multi-turn chat, and +agents. +Pretraining: Large‑scale (usually self‑supervised) training on broad cor- +pora to create a base model. +Prompt: The instruction to the model; for diffusion also includes a negative +prompt and step/guidance parameters. +PyTorch compile (torch.compile): Graph capture and kernel selection/ +fusion targeting a specific GPU. Cache compiled engines to cut cold‑start +times. +PyTorch Profiler: A developer tool measuring CPU and GPU time and +memory per operation. +Quantization (post‑training): Lowering precision of weights, activations, +and potentially KV cache to reduce compute and memory bandwidth +demands. + + +Inference Glossary  225 +Quantization‑aware training: A training technique in which quantization +scales are computed and weights are optimized jointly so that the final +model is already calibrated for low-precision deployment. +Queue (request): Holds excess traffic while autoscaling brings replicas +online. +Real‑time factor (RTF): A measurement of how quickly ASR models +can transcribe audio. Transcribing an hour of audio in six seconds is an +RTF of 600. +Retrieval‑augmented generation (RAG): A common application pattern +that fetches additional context for the LLM beyond the prompt. +Ring attention: A Context Parallelism mechanism in which GPUs pass +partial attention results in a ring. Reduces all‑to‑all pressure for very large +contexts. +Roofline model: Plots arithmetic intensity with bandwidth and compute +ceilings, creating a visual guide on whether to optimize memory or com- +pute. +Rotary positional embeddings (RoPE): A positional encoding scheme +that encodes positions as learned rotations, improving long-context extrap- +olation at the cost of higher memory demands for attention during infer- +ence. +Routing (inference): Placing requests on replicas based on load, KV +cache, available LoRAs, and sequence shapes to improve speed and +utilization. +Rubin (architecture): Next NVIDIA generation (2026) introducing HBM4 +and CPX for compute‑bound workloads. +Sampling (decode): The process of selecting an output token based on +the generated logits. Common strategies include greedy (argmax), tem- +perature‑based sampling, top‑k, and top‑p (nucleus) sampling. +Scale factor (quantization): Multipliers used to map low‑precision values +to their original number formats. + + +226  Inference Glossary +Scale to zero: Turn off all replicas when idle; spin up on demand. Requires +fast cold starts and robust queueing; best for predictable or dev workloads. +SDXL: An instructive, earlier diffusion image pipeline (base + refiner + CLIP). +Modern systems retain the structure with larger, more capable components. +Service Level Agreement (SLA): A contractual promise of latency, +throughput, uptime, or other performance factor from a system. +Service Level Objective (SLO): An internal target designed to meet or +beat the SLA for a given system. +SGLang: A fast inference engine with flexible frontend/backends and +strong MoE support. +Shadow traffic: Mirroring real production requests to a candidate deploy- +ment. +SNAC (audio decoder): A performant audio decoder path often paired +with TTS token streams. +Softmax: Converts scores to probabilities in attention and normalizes +logits to a probability distribution in decoding. +Sparsity (FLOPS): In tensors with 2:4 structured sparsity, where 50 per- +cent of the values are 0, Tensor Cores can skip multiplication by 0. Most +inference is dense, not sparse. +Special Function Unit (SFU): A dedicated hardware unit that accelerates +specific math operations like sine and cosine, keeping specialized oper- +ations off of CUDA Cores. +Speculative decoding: A family of strategies for generating and validating +draft tokens to generate multiple tokens per forward pass during decode. +Streaming Multiprocessor (SM): GPU compute unit containing cores +and cache. + + +Inference Glossary  227 +Structured output: LLM output that adheres to a specific schema. Cre- +ated by constraining generation to a supplied schema via logit biasing +rather than via prompting. +SXM (GPU form factor): A socketed GPU module that supports higher-band- +width connections and delivers more power than PCIe. SXM form factor GPUs +often have higher base specs and are the standard for inference. +Temperature: Controls randomness in token selection: lower values (e.g., +0.1) make output more deterministic; higher values (e.g., 1.5) increase +diversity. +Tensor Parallelism (TP): Splits tensor operations across GPUs within a +node. Best per‑user latency; requires frequent all‑reduce synchronization. +TensorRT: NVIDIA’s optimized runtime for high‑performance inference +with fused kernels, quantization, and other optimizations. +TensorRT-LLM: An inference engine built by NVIDIA that provides a +Python API and both TensorRT‑engine and PyTorch‑backend execution +paths with fused kernels, quantization, and speculative decoding. +Thread: The minimal execution unit on a GPU. Kernels launch many +threads to achieve massive parallelism. +Throughput: Total work per unit time (e.g., total tokens per second). +Time to first byte (TTFB): Time until first byte of output is returned, a +latency metric. +Time to first token (TTFT): Time until first token of output is returned, a +latency metric. +Token: The atomic unit of text processing in LLMs. A token is an integer +that represents a string of characters. In English, there is approximately +a 4:3 token:word ratio for most tokenizers. +Tokenizer: Deterministically converts strings into sequences of tokens, +and vise versa. Models have different tokenizers, and more efficient +tokenizers improve end-to-end latency. + + +228  Inference Glossary +Tokens per second (TPS): See perceived TPS. A latency metric for the +number of tokens streamed to the end user per second. +Training: The process of learning model weights from data using back- +propagation and optimization. Training is compute‑intensive, typically runs +on large GPU clusters, and produces the weights used in inference. +Transformer: The foundational architecture behind generative AI models. +Transformers (library): Reference implementations for LLMs and other +transformers-based models. +Triton Inference Server: A production serving framework by NVIDIA with +support for multiple backends. +VAE (variational autoencoder): Used in inference to decode from latent +space to pixel space for image and video generation models (also used +for encoding from pixel to latent space during training). +Vector database: A database for storing and querying the semantic vec- +tors created by embedding models. +Vector similarity: A check between two vectors to see how close together +they are based on an equation like cosine similarity. Vectors with high +similarity encode similar semantic meaning. +Vera CPU: ARM-based NVIDIA CPU with high-bandwidth chip-to-chip +interconnects between the CPU and GPU. Succeeds Grace GPUs along- +side the Rubin GPU architecture generation. +Vision‑language model (VLM): Accepts images and video plus text +prompts and outputs text. +vLLM: A widely adopted inference engine with broad model and hardware +support and strong defaults. +Vocabulary: The total set of tokens, usually more than 100,000, that an +LLM uses to represent data. + + +Inference Glossary  229 +Voice activity detection (VAD): A lightweight model that segments +streams/files into speech‑containing chunks for ASR. +VRAM (device memory): On‑GPU memory used for weights, KV, and +activations. Total VRAM gates model size and KV headroom; bandwidth +gates decode TPS. +WebSocket: Lightweight, bidirectional streaming transport. Ideal for +unstructured audio chunks and real‑time UX. +Weights‑only quantization: Reduces precision for model weights in lin- +ear layers while preserving other model components like KV cache and +attention at higher precision. A conservative approach to quantization with +the best quality preservation but the lowest performance improvements. +Workload plane (multi‑cloud): An individual cluster with compute +resources that runs inference and processes requests. + + + diff --git a/library/Inference Engineering/Chapter-11-Appendix-B-Recommended-Reading/Chapter-11-source-text.md b/library/Inference Engineering/Chapter-11-Appendix-B-Recommended-Reading/Chapter-11-source-text.md new file mode 100644 index 0000000..0a5d1a4 --- /dev/null +++ b/library/Inference Engineering/Chapter-11-Appendix-B-Recommended-Reading/Chapter-11-source-text.md @@ -0,0 +1,420 @@ +# Inference Engineering — Chapter-11: Appendix B: Recommended Reading +- **Source**: /library/Inference Engineering/source-file.pdf +- **PDF pages**: 233–258 +- **Pages without text**: 234, 258 + +--- + + +APPENDIX B +Recommended +Reading + + + + + +Recommended Reading  233 +This book is an introduction to the field of inference engineering. There is +endless depth to explore in every one of the technologies and techniques +behind performant inference at scale. +If you’re in the market for another book to continue learning, I have three +recommendations: +• AI Engineering: Building Applications with Foundation Models by Chip +Huyen (O’Reilly Media, 2025): This incredibly popular book introduces +the full breadth of AI engineering topics. +• Build a Large Language Model (From Scratch) by Sebastian Raschka +(Manning, 2024): This hands-on book provides a detailed look at LLM +architecture. +• AI Systems Performance Engineering: Optimizing Model Training and +Inference Workloads with GPUs, CUDA, and PyTorch by Chris Fregly +(O’Reilly Media, 2025): This brand-new book focuses on building for +performance. +The AI industry moves fast, and new models, research, and imple- +mentations are constantly being released. My colleagues and I pub- +lish our latest work on the Baseten blog, which you can access at +https://www.baseten.com/blog. +This appendix provides a list of papers, documentation, books, and blogs +to further support your next steps as an inference engineer. Resources are +organized by topic and alphabetized by title within each section. + + +234  Recommended Reading +Architecture +“Attention is All You Need,” by Ashish Vaswani et al. +(Neural Information Processing Systems, 2017), https:// +arxiv.org/abs/1706.03762 +“BERT: Pre-training of Deep Bidirectional Transformers for +Language Understanding,” by Jacob Devlin et al. (North +American Chapter of the Association for Computational +Linguistics, 2019), https://arxiv.org/abs/1810.04805 +“BLIP-2: Bootstrapping Language-Image Pre-training with +Frozen Image Encoders and Large Language Models,” +by Junnan Li et al. (International Conference on Machine +Learning, 2023), https://arxiv.org/abs/2301.12597 +Deep Learning, by Ian Goodfellow, Yoshua Bengio, and +Aaron Courville (The MIT Press, 2016), https://www. +deeplearningbook.org/ +Deep Learning with Python (2nd Edition), by François +Chollet (Manning, 2021), https://www.manning.com/books/ +deep-learning-with-python-second-edition +“Denoising Diffusion Probabilistic Models,” by Jonathan +Ho et al. (ArXiv abs/2006.11239, 2020), https://arxiv.org/ +abs/2006.11239 + + +Recommended Reading  235 +“DiT: Scalable Diffusion Models with Transformers,” by +William Peebles and Saining Xie (International Conference +on Computer Vision (ICCV), 2022), https://arxiv.org/ +abs/2212.09748 +“FlashAttention: Fast and Memory-Efficient Exact +Attention with IO Awareness,” by Tri Dao et al. (ArXiv +abs/2205.14135, 2022), https://arxiv.org/abs/2205.14135 +“FlashAttention-2: Faster Attention with Better Parallelism +and Work Partitioning,” by Tri Dao (ArXiv abs/2307.08691, +2023), https://arxiv.org/abs/2307.08691 +“FlashAttention-3: Fast and Accurate Attention with +Asynchrony and Low-precision,” by Jay Shah et al. (ArXiv +abs/2407.08608, 2024), https://arxiv.org/abs/2407.08608 +Flash-Attention-4, by Tri Dao (Dao AI Research Lab, +2025), https://github.com/Dao-AILab/flash-attention +“Imagen Video: High Definition Video Generation +with Diffusion Models,” by Jonathan Ho et al. (ArXiv +abs/2210.02303, 2022), https://arxiv.org/abs/2210.02303 + + +236  Recommended Reading +“Language Models Are Few-Shot Learners,” by Tom +Brown et al. (ArXiv abs/2005.14165), https://arxiv.org/ +abs/2005.14165 +“Learning Transferable Visual Models from Natural +Language Supervision,” by Alec Radford et al. +(International Conference on Machine Learning, 2021), +https://arxiv.org/abs/2103.00020 +“Longformer: The Long-Document Transformer,” by Iz +Beltagy et al. (ArXiv abs/2004.05150, 2020), https://arxiv. +org/abs/2004.05150 +“Mamba: Linear-Time Sequence Modeling with Selective +State Spaces,” by Albert Gu and Tri Dao (ArXiv +abs/2312.00752, 2023), https://arxiv.org/abs/2312.00752 +“Matryoshka Representation Learning,” by Aditya Kusupati +et al. (Neural Information Processing Systems, 2022), +https://arxiv.org/abs/2205.13147 +“Outrageously Large Neural Networks: The Sparsely-Gated +Mixture-of-Experts Layer,” by Noam Shazeer et al. (ArXiv +abs/1701.06538, 2017), https://arxiv.org/abs/1701.06538 + + +Recommended Reading  237 +“Reformer: The Efficient Transformer,” by Nikita Kitaev +et al. (ArXiv abs/2001.04451, 2020), https://arxiv.org/ +abs/2001.04451 +“Robust Speech Recognition via Large-Scale Weak +Supervision,” by Alec Radford et al. (International +Conference on Machine Learning, 2022), https://arxiv.org/ +abs/2212.04356 +“RoFormer: Enhanced Transformer with Rotary Position +Embedding,” by Jianlin Su et al. (ArXiv abs/2104.09864, +2021), https://arxiv.org/abs/2104.09864 +“SDXL: Improving Latent Diffusion Models for High- +Resolution Image Synthesis,” by Dustin Podell et al. (ArXiv +abs/2307.01952, 2023), https://arxiv.org/abs/2307.01952 +“Segment Anything,” by Alexander Kirillov et al. (2023 +IEEE/CVF International Conference on Computer Vision +(ICCV), 2023), https://arxiv.org/abs/2304.02643 +“Sentence-BERT: Sentence Embeddings Using Siamese +BERT-Networks,” by Nils Reimers and Iryna Gurevych +(ArXiv abs/1908.10084, 2019), https://arxiv.org/ +abs/1908.10084 + + +238  Recommended Reading +“The Llama 3 Herd of Models,” by Aaron Grattafiori et al. +(ArXiv 2407.21783, 2024), https://arxiv.org/abs/2407.21783 +“Video Diffusion Models,” by Jonathan Ho et al. (ArXiv +abs/2204.03458, 2022), https://arxiv.org/abs/2204.03458 +“Visual Instruction Tuning,” by Haotian Liu et al. (ArXiv +abs/2304.08485, 2023), https://arxiv.org/abs/2304.08485 + + +Recommended Reading  239 +Developer Tools +BitsAndBytes by bitsandbytes-foundation, https://github. +com/TimDettmers/bitsandbytes +ComfyUI by comfyanonymous, https://github.com/ +comfyanonymous/ComfyUI +“CUDA by Example: An Introduction to General Purpose +GPU Programming,” by Jason Sanders and Edward +Kandrot (NVIDIA developer, 2025), https://developer. +nvidia.com/cuda-example +“CUDA C++ Programming Guide Release 13.0,” by NVIDIA +(2025), https://docs.nvidia.com/cuda/cuda-c-programming- +guide/ +“CUDA cuBLAS Release 13.0,” by NVIDIA (2025), https:// +docs.nvidia.com/cuda/cublas/ +CUTLASS by NVIDIA, https://github.com/NVIDIA/cutlass + + +240  Recommended Reading +DeepGEMM by DeepSeek-ai, https://github.com/ +deepseek-ai/DeepGEMM +Hugging Face Diffusers by Hugging Face, https:// +huggingface.co/docs/diffusers/index +LMCache by LMCache Project, https://github.com/ +LMCache/LMCache +“NVIDIA Dynamo Documentation,” by NVIDIA (2025), +https://docs.nvidia.com/dynamo/latest/index.html +NVIDIA Nsight Systems by NVIDIA, https://developer. +nvidia.com/nsight-systems +NVIDIA Triton Inference Server by NVIDIA, https://github. +com/triton-inference-server/server + + +Recommended Reading  241 +ONNX Runtime by Microsoft, https://onnxruntime.ai/ +“PyTorch Performance Tuning Guide,” by Szymon Migacz +(PyTorch Foundation, 2020), https://pytorch.org/tutorials/ +recipes/recipes/tuning_guide.html +“PyTorch Profiler,” by Shivam Raikundalia (PyTorch +Foundation, 2021), https://pytorch.org/tutorials/recipes/ +recipes/profiler_recipe.html +SGLang Project by LMSYS Org, https://github.com/sgl- +project/sglang +“TensorRT Documentation,” by NVIDIA (2025), https:// +docs.nvidia.com/deeplearning/tensorrt/ +TensorRT-LLM by NVIDIA, https://github.com/NVIDIA/ +TensorRT-LLM + + +242  Recommended Reading +Transformers by Hugging Face, https://huggingface.co/ +docs/transformers/index +vLLM Project by The Linux Foundation, https://github.com/ +vllm-project/vllm + + +Recommended Reading  243 +Frontier Open Models +DeepSeek, by DeepSeek AI, https://huggingface.co/ +deepseek-ai +FLUX, by Black Forest Labs, https://huggingface.co/black- +forest-labs +Gemma, by Google, https://huggingface.co/google +GLM, by Z.ai, +https://huggingface.co/zai-org +GPT OSS, by OpenAI, https://huggingface.co/openai +Kimi, by Moonshot AI, https://huggingface.co/moonshotai + + +244  Recommended Reading +Llama, by Meta Llama, https://huggingface.co/meta-llama +MiniMax, by MiniMax AI, https://huggingface.co/MiniMaxAI +Mistral, by Mistral AI, https://huggingface.co/mistralai +Nemotron, by NVIDIA, https://huggingface.co/nvidia +Orpheus, by Canopy Labs, https://huggingface.co/ +canopylabs +Qwen, by Alibaba Qwen, https://huggingface.co/Qwen + + +Recommended Reading  245 +Wan, by Wan-AI, +https://huggingface.co/Wan-AI +Whisper, by OpenAI, https://huggingface.co/openai + + +246  Recommended Reading +GPU Infrastructure +Designing Data-Intensive Applications, by Martin +Kleppmann (O’Reilly Media, 2017) https://dataintensive. +net/ +Grace Hopper / Grace Blackwell Systems by NVIDIA, +https://www.nvidia.com/en-us/data-center/grace-cpu/ +GPU Glossary, by Frye et al. (Modal, 2025), https://modal. +com/gpu-glossary +InfiniBand, by NVIDIA, https://www.nvidia.com/en-us/ +networking/products/infiniband/ +Kubernetes Documentation, by The Kubernetes Authors +(The Linux Foundation, 2025), https://kubernetes.io/docs/ +home/ + + +Recommended Reading  247 +NVIDIA Blackwell Architecture Technical Brief: Built for the +Age of AI Reasoning, (NVIDIA, 2025), https://resources. +nvidia.com/en-us-blackwell-architecture?ncid=no-ncid +NVIDIA H100 Tensor Core GPU Architecture: Exceptional +Performance, Scalability and Security for the Data Center, +(NVIDIA, 2023), https://resources.nvidia.com/en-us- +hopper-architecture/nvidia-h100-tensor-c +“NVIDIA Tesla: A Unified Graphics and Computing +Architecture," by E. Lindholm et al. (IEEE Micro, March– +April 2008), +https://ieeexplore.ieee.org/document/4523358 +NVLink / NVSwitch, by NVIDIA, https://www.nvidia.com/ +en-us/data-center/nvlink/ +Programming Massively Parallel Processors: A Hands-on +Approach, by Wen-mei Hwu, David Kirk, Izzat El Hajj +(Morgan Kaufmann, 2022), https://www.goodreads.com/ +work/editions/10244675-programming-massively-parallel- +processors-a-hands-on-approach +SemiAnalysis, by Dylan Patel (SemiAnalysis, 2025), +https://semianalysis.com/ + + +248  Recommended Reading +Site Reliability Engineering: How Google Runs Production +Services, edited by Betsy Beyer et al. (O’Reilly Media, +2017), https://sre.google/books/ + + +Recommended Reading  249 +Inference Optimization Research +“Adversarial Diffusion Distillation,” by Axel Sauer et al. +(European Conference on Computer Vision, 2023), https:// +arxiv.org/abs/2311.17042 +“Train Short, Test Long: Attention with Linear Biases +Enables Input Length Extrapolation,” by Ofir Press, Noah +Smith, and Mike Lewis (ArXiv abs/2108.12409, 2021), +https://arxiv.org/abs/2108.12409 +“AWQ: Activation-aware Weight Quantization for LLM +Compression and Acceleration,” by Song Han (MIT, 2024), +https://github.com/mit-han-lab/llm-awq +Cache-DIT by Vipshop, https://github.com/vipshop/cache- +dit +“CacheBlend: Fast Large Language Model Serving for +RAG with Cached Knowledge Fusion,” by Jiayi Yao et +al. (Proceedings of the Twentieth European Conference +on Computer Systems, 2024), https://arxiv.org/ +abs/2405.16444 + + +250  Recommended Reading +“Adding Conditional Control to Text-to-Image +Diffusion Models,” by Lymin Zhang et al. (International +Conference on Computer Vision, 2023), https://arxiv.org/ +abs/2302.05543 +“Beyond the Buzz: A Pragmatic Take on Inference +Disaggregation,” by Tiyasa Mitra et al. (ArXiv +abs/2506.05508, 2025), https://arxiv.org/abs/2506.05508 +“Break the Sequential Dependency of LLM Inference +Using Lookahead Decoding,” by Yichao Fu et al. (ArXiv +abs/2402.02057, 2024), https://arxiv.org/abs/2402.02057 +“EAGLE: Speculative Sampling Requires Rethinking +Feature Uncertainty,” by Yuhui Li et al. (ArXiv +abs/2401.15077, 2024), https://arxiv.org/abs/2401.15077 +“EAGLE-2: Faster Inference of Language Models with +Dynamic Draft Trees,” by Yuhui Li et al. (Conference on +Empirical Methods in Natural Language Processing, +2024), https://arxiv.org/abs/2406.16858 +“EAGLE-3: Scaling up Inference Acceleration of Large +Language Models via Training-Time Test,” by Yuhui Li +et al. (ArXiv abs/2503.01840, 2025), https://arxiv.org/ +abs/2503.01840 + + +Recommended Reading  251 +“FlashInfer: Efficient and Customizable Attention +Engine for LLM Inference Serving,” by Ye et al. (ArXiv +abs/2501.01005, 2025), +https://arxiv.org/abs/2501.01005 +“GPTQ: Accurate Post-Training Quantization for +Generative Pre-trained Transformers,” by Elias Frantar +(ArXiv abs/2210.17323, 2022), https://arxiv.org/ +abs/2210.17323 +“High-Resolution Image Synthesis with Latent Diffusion +Models,” by Robin Rombach et al. (Conference on +Computer Vision and Pattern Recognition (CVPR), 2021), +https://arxiv.org/abs/2112.10752 +“Latent Consistency Models: Synthesizing High-Resolution +Images with Few-Step Inference,” by Simian Luo (ArXiv +abs/2310.04378, 2023), https://arxiv.org/abs/2310.04378 +“LLM.int8(): 8-bit Matrix Multiplication for Transformers +at Scale,” by Tim Dettmers et al. (ArXiv abs/2208.07339, +2022), https://arxiv.org/abs/2208.07339 +“Medusa: Simple LLM Inference Acceleration Framework +with Multiple Decoding Heads,” by Tianle Cai et al. (ArXiv +abs/2401.10774, 2024), https://arxiv.org/abs/2401.10774 + + +252  Recommended Reading +“Megatron-LM: Training Multi-Billion Parameter Language +Models Using Model Parallelism,” by Mohammad Shoeybi +et al. (ArXiv abs/1909.08053, 2019), https://arxiv.org/ +abs/1909.08053 +“Efficient Memory Management for Large Language +Model Serving with PagedAttention,” by Woosuk +Kwon et al. (Proceedings of the 29th Symposium on +Operating Systems Principles, 2023), https://arxiv.org/ +abs/2309.06180 +“Fast Inference from Transformers via Speculative +Decoding,” by Yaniv Leviathan et al. (International +Conference on Machine Learning, 2022), https://arxiv.org/ +abs/2211.17192 +“Ring Attention with Blockwise Transformers for Near- +Infinite Context,” by Hao Lin et al. (ArXiv abs/2310.01889, +2023), https://arxiv.org/abs/2310.01889 +“SageAttention: Accurate 8-Bit Attention for Plug-and- +play Inference Acceleration,” by Jintao Zhang et al. (ArXiv +abs/2410.02367, 2024), https://arxiv.org/abs/2410.02367 +Sequence/Context Parallelism, by Megatron-LM for +NVIDIA https://github.com/NVIDIA/Megatron-LM + + +Recommended Reading  253 +SmoothQuant by Song Han (MIT), https://github.com/mit- +han-lab/smoothquant +“SparseGPT: Massive Language Models Can Be +Accurately Pruned in One-Shot,” by Elias Frantar and Dan +Alistarh, (ArXiv abs/2301.00774, 2023), https://arxiv.org/ +abs/2301.00774 +“SpecVLM: Fast Speculative Decoding in Vision-Language +Models,” Haiduo Huang et al. (ArXiv abs/2509.11815, +2025), +https://arxiv.org/abs/2509.11815 +“TeaCache: Efficient KV Cache Compression via Tensor +Decomposition,” by Feng Lu et al. (Alibaba TongYi Vision +Intelligence Lab, ArXiv abs/2411.19108, 2025), https:// +github.com/ali-vilab/TeaCache + + +254  Recommended Reading +Intelligence Evaluation +ARC AGI Prize by Greg Kamradt (2025), https://arcprize. +org/ +Evals for AI Engineers: Systematically Measuring and +Improving AI Applications, by Shreya Shankar and Hamel +Husain (O’Reilly Media, forthcoming 2026) https://www. +oreilly.com/library/view/evals-for-ai/9798341660717/ +Grade School Math: Training Verifiers to Solve Math Word +Problems, by Karl Cobbe and Vineet Kosaraju (ArXiv +abs/2110.14168, 2021), https://github.com/openai/grade- +school-math +“How to Fine-Tune Qwen3 to GPT-4o Level Performance,” +by Greg Schoeninger (Fine-Tune Fridays, Oxen AI, 2025), +https://ghost.oxen.ai/how-to-fine-tune-qwen3-to-gpt-4o- +level-performance/ +“Humanity’s Last Exam,” by Long Phan et al. (Center for AI +Safety and Scale AI, ArXiv abs/2501.14249, 2025), https:// +www.lastexam.ai/ +“HumanEval: Evaluating Large Language Models Trained +on Code,” by Michelle Pokrass, Qiming Yuan, and Yichen +Xu (OpenAI, 2021), https://github.com/openai/human-eval + + +Recommended Reading  255 +“MMLU: Measuring Massive Multitask Language +Understanding,” by Dan Hendrycks et al. (Proceedings of +the International Conference on Learning Representations +(ICLR), 2021), https://github.com/hendrycks/test +“MTEB: Massive Text Embedding Benchmark,” by Niklas +Muenninghoff et al. (Conference of the European Chapter +of the Association for Computational Linguistics, 2022), +https://arxiv.org/abs/2210.07316 +“SWE-Bench: Can Language Models Resolve Real-World +Github Issues?” by Carlos Jimenez et al. (Proceedings of +the International Conference on Learning Representations +(ICLR), 2024), https://www.swebench.com/ + + + diff --git a/library/Inference Engineering/Front-Matter/Front-Matter-source-text.md b/library/Inference Engineering/Front-Matter/Front-Matter-source-text.md new file mode 100644 index 0000000..f0010b4 --- /dev/null +++ b/library/Inference Engineering/Front-Matter/Front-Matter-source-text.md @@ -0,0 +1,226 @@ +# Inference Engineering — Front-Matter: Front matter (cover, title, contents) +- **Source**: /library/Inference Engineering/source-file.pdf +- **PDF pages**: 1–10 +- **Pages without text**: 1, 2, 10 + +--- + + + + + + + + +INFERENCE +ENGINEERING +BY PHILIP KIELY + + +Inference Engineering by Philip Kiely +Published by Baseten Books +Copyright © 2026 Baseten Labs, Inc +All rights reserved. No portion of this book may be reproduced in any form without written permission from +the publisher, except as permitted by U.S. copyright law. For permissions contact Baseten. +Cover art by Luke de Haas +Edited by Robin Bourjaily +ISBN: 979-8-9943597-2-3 + + +Table of Contents +Preface....................................................................................................9 +Chapter 0: Inference............................................................................15 +Chapter 1: Prerequisites ....................................................................23 +1.1 +Scale and Specialization..............................................................26 +1.2 +About Your App............................................................................27 +1.2.1 AI-Native Applications......................................................28 +1.2.2 Online versus Offline........................................................29 +1.2.3 Consumer versus B2B......................................................30 +1.3 +Model Selection............................................................................31 +1.3.1 Model Evaluation..............................................................31 +1.3.2 Fine-Tuning for Domain-Specific Quality..........................32 +1.3.3 Distillation.........................................................................33 +1.4 +Measuring Latency and Throughput............................................35 +1.4.1 Latency Percentiles..........................................................36 +1.4.2 End-to-End Metrics...........................................................37 +Chapter 2: Models................................................................................39 +2.1 +Neural Networks...........................................................................42 +2.1.1 Linear Layers and Matmul................................................44 +2.1.2 Activation Functions.........................................................44 +2.2 +LLM Inference Mechanics............................................................46 +2.2.1 LLM Architecture..............................................................49 +2.2.2 Transformer Blocks..........................................................50 +2.2.3 Attention...........................................................................52 +2.2.4 Mixture of Experts Models................................................53 +2.3 +Image Generation Inference Mechanics......................................55 +2.3.1 Image Generation Model Architecture..............................57 +2.3.2 Few-Step Image Generation Models................................59 +2.3.3 Video Generation..............................................................59 + + +4  Table of Contents +2.4 +Calculating Inference Bottlenecks................................................61 +2.4.1 Ops:Byte Ratio and Arithmetic Intensity...........................62 +2.4.2 LLM Inference Bottlenecks...............................................63 +2.4.3 Image Generation Inference Bottlenecks.........................67 +2.5 +Optimizing Attention.....................................................................67 +Chapter 3: Hardware............................................................................71 +3.1 +GPU Architecture.........................................................................74 +3.1.1 Compute...........................................................................74 +3.1.2 Memory and Caches........................................................76 +3.2 +GPU Architecture Generations.....................................................77 +3.2.1 Hopper GPUs...................................................................79 +3.2.2 Ada Lovelace GPUs.........................................................80 +3.2.3 Blackwell GPUs................................................................81 +3.2.4 Rubin GPUs......................................................................81 +3.2.5 Grace and Vera CPUs......................................................82 +3.3 +Instances......................................................................................83 +3.3.1 Multi-GPU Instances.........................................................84 +3.3.2 Multi-Instance GPUs.........................................................86 +3.4 +Other Datacenter Accelerator Options.........................................87 +3.5 +Local Inference.............................................................................89 +3.5.1 Desktop Inference............................................................90 +3.5.2 Mobile Inference...............................................................91 +Chapter 4: Software.............................................................................93 +4.1 +CUDA...........................................................................................96 +4.1.1 CUDA Kernels for Inference.............................................98 +4.1.2 CUDA Kernel Selection....................................................98 +4.1.3 Reducing Memory Accesses with Kernel Fusion...........100 +4.2 +Deep Learning Frameworks and Libraries.................................101 +4.2.1 PyTorch..........................................................................102 +4.2.2 Model File Formats.........................................................103 +4.2.3 ONNX Runtime and TensorRT.......................................104 +4.2.4 Transformers and Diffusers............................................105 +4.3 +Inference Engines......................................................................105 +4.3.1 vLLM...............................................................................106 + + +Table of Contents  5 +4.3.2 SGLang..........................................................................108 +4.3.3 TensorRT-LLM...............................................................109 +4.4 +NVIDIA Dynamo.........................................................................111 +4.5 +Performance Benchmarking and Load Testing..........................112 +4.5.1 Performance Benchmarking Tooling..............................113 +4.5.2 Performance Benchmarking Tips...................................114 +4.5.3 Profiling Performance.....................................................114 +Chapter 5: Techniques......................................................................117 +5.1 +Quantization...............................................................................120 +5.1.1 Number Formats.............................................................121 +5.1.2 Quantization Approaches...............................................125 +5.1.3 Measuring Quality Impact...............................................128 +5.2 +Speculative Decoding................................................................129 +5.2.1 Draft-Target Speculative Decoding................................131 +5.2.2 Medusa...........................................................................132 +5.2.3 EAGLE............................................................................133 +5.2.4 N-gram Speculation and Lookahead Decoding..............134 +5.3 +Caching......................................................................................136 +5.3.1 Prefix Caching and KV Cache Re-Use...........................136 +5.3.2 Where to Store the KV Cache........................................139 +5.3.3 Cache-Aware Routing....................................................140 +5.3.4 Long Context Handling...................................................141 +5.4 +Model Parallelism.......................................................................142 +5.4.1 Tensor Parallelism for Lower Latency............................144 +5.4.2 Expert Parallelism for Higher Throughput......................145 +5.4.3 Multi-Node Inference......................................................146 +5.5 +Disaggregation...........................................................................148 +5.5.1 How Disaggregation Works............................................148 +5.5.2 When to Use Disaggregation..........................................149 +5.5.3 Dynamic Disaggregation with NVIDIA Dynamo..............150 +Chapter 6: Modalities .......................................................................153 +6.1 +Vision Language Models............................................................156 +6.1.1 Video Processing for Vision Language Models..............158 +6.1.2 Omni-Modal Models.......................................................159 + + +6  Table of Contents +6.2 +Embedding Models....................................................................159 +6.2.1 Embedding Model Architecture......................................160 +6.2.2 Embedding Model Inference...........................................161 +6.3 +ASR Models...............................................................................162 +6.3.1 Single-Chunk Latency Optimization...............................163 +6.3.2 Long File Latency Optimization......................................164 +6.3.3 Diarization.......................................................................166 +6.4 +TTS Models................................................................................166 +6.4.1 Streaming Real-Time Text to Speech............................168 +6.4.2 Speech-to-Speech Models.............................................168 +6.5 +Image Generation Models..........................................................169 +6.5.1 Image Generation Kernel Optimization..........................170 +6.5.2 One Weird Trick for Faster Image Generation...............172 +6.6 +Video Generation Models...........................................................173 +6.6.1 Attention Optimization and Quantization........................174 +6.6.2 Context Parallelism.........................................................176 +Chapter 7: Production.......................................................................177 +7.1 +Containerization.........................................................................179 +7.1.1 Dependency Management.............................................181 +7.1.2 NIMs...............................................................................183 +7.2 +Autoscaling.................................................................................183 +7.2.1 Concurrency and Batch Sizing.......................................186 +7.2.2 Cold Starts......................................................................188 +7.2.3 Routing, Load Balancing, and Queueing........................190 +7.2.4 Scale to Zero..................................................................192 +7.2.5 Independent Component Scaling...................................192 +7.3 +Multi-Cloud Capacity Management............................................193 +7.3.1 GPU Procurement..........................................................195 +7.3.2 Geo-Aware Load Balancing............................................196 +7.3.3 Building for Reliability.....................................................196 +7.3.4 Security and Compliance................................................198 +7.4 +Testing and Deployment............................................................199 +7.4.1 Zero-Downtime Deployment...........................................200 +7.4.2 Cost Estimation..............................................................201 +7.4.3 Observability...................................................................203 +7.5 +Client Code................................................................................204 + + +Table of Contents  7 +7.5.1 Client Latency Overhead................................................205 +7.5.2 Asynchronous inference.................................................205 +7.5.3 Streaming and Protocol Support....................................205 +7.6 +Production Inference with Baseten............................................208 +Appendix A: Inference Glossary..........................................................209 +Appendix B: Recommended Reading..................................................231 +Architecture................................................................................234 +Developer Tools.........................................................................239 +Frontier Open Models................................................................243 +GPU Infrastructure.....................................................................246 +Inference Optimization Research...............................................249 +Intelligence Evaluation...............................................................254 +Acknowledgements..............................................................................257 + + + diff --git a/library/Inference Engineering/book-structure.md b/library/Inference Engineering/book-structure.md new file mode 100644 index 0000000..e9249ca --- /dev/null +++ b/library/Inference Engineering/book-structure.md @@ -0,0 +1,41 @@ +# Inference Engineering — Book Structure + +- **Author**: Philip Kiely +- **Format**: PDF +- **PDF pages**: 259 (cover image on p.1; blank pages 2, 10 and others are layout spacers) +- **Reading order**: 11 sections. Chapter-01 is the unnumbered Preface; Chapter-02 is the book's "Chapter 0", so reading number = book chapter + 2 (Chapter-09 = book Chapter 7). Appendices A and B are Chapter-10 and Chapter-11. Bookmark page numbers equal PDF page numbers. + +| Reading chapter | PDF pages | Section title | +| --- | --- | --- | +| Chapter-01-Preface | 11–16 | Preface | +| Chapter-02-Inference | 17–24 | Chapter 0: Inference | +| Chapter-03-Prerequisites | 25–40 | Chapter 1: Prerequisites | +| Chapter-04-Models | 41–72 | Chapter 2: Models | +| Chapter-05-Hardware | 73–94 | Chapter 3: Hardware | +| Chapter-06-Software | 95–118 | Chapter 4: Software | +| Chapter-07-Techniques | 119–154 | Chapter 5: Techniques | +| Chapter-08-Modalities | 155–178 | Chapter 6: Modalities | +| Chapter-09-Production | 179–210 | Chapter 7: Production | +| Chapter-10-Appendix-A-Inference-Glossary | 211–232 | Appendix A: Inference Glossary | +| Chapter-11-Appendix-B-Recommended-Reading | 233–258 | Appendix B: Recommended Reading | + +## Source Text Index +Extracted from `source-file.pdf` by `tools/split_book.py`. Read these instead of the PDF. + +| Folder | Section | PDF pages | Pages without extractable text | +| --- | --- | --- | --- | +| Front-Matter | Front matter (cover, title, contents) | 1–10 | 1 (cover image), 2, 10 | +| Chapter-01-Preface | Preface | 11–16 | none | +| Chapter-02-Inference | Chapter 0: Inference | 17–24 | 18 | +| Chapter-03-Prerequisites | Chapter 1: Prerequisites | 25–40 | 26, 40 | +| Chapter-04-Models | Chapter 2: Models | 41–72 | 42, 72 | +| Chapter-05-Hardware | Chapter 3: Hardware | 73–94 | 74, 94 | +| Chapter-06-Software | Chapter 4: Software | 95–118 | 96, 118 | +| Chapter-07-Techniques | Chapter 5: Techniques | 119–154 | 120, 154 | +| Chapter-08-Modalities | Chapter 6: Modalities | 155–178 | 156 | +| Chapter-09-Production | Chapter 7: Production | 179–210 | 180 | +| Chapter-10-Appendix-A-Inference-Glossary | Appendix A: Inference Glossary | 211–232 | 212, 232 | +| Chapter-11-Appendix-B-Recommended-Reading | Appendix B: Recommended Reading | 233–258 | 234, 258 | +| Back-Matter | Acknowledgements | 259 | none | + +All 259 pages are covered exactly once. The listed blank pages are layout spacers (chapter-opener backs), not lost figures. Figures inside chapters may still be images the text lost; check the PDF page only if asked. diff --git a/library/Inference Engineering/source-file.pdf b/library/Inference Engineering/source-file.pdf new file mode 100644 index 0000000..b1f8149 Binary files /dev/null and b/library/Inference Engineering/source-file.pdf differ diff --git a/tools/split_book.py b/tools/split_book.py index cedb6dc..d337fd1 100644 --- a/tools/split_book.py +++ b/tools/split_book.py @@ -125,12 +125,29 @@ def build_focus(): return secs +def build_inference(): + d = fitz.open(ROOT / "Inference Engineering" / "source-file.pdf") + top = [(t, p) for lvl, t, p in d.get_toc() if lvl <= 2] + # Skip the Table of Contents bookmark: it is front matter (pages 1-10 with the cover and TOC). + top = [(t, p) for t, p in top if t != "Table of Contents"] + last_body = next(p for t, p in top if t == "Acknowledgements") + secs = [("Front-Matter", "Front matter (cover, title, contents)", 1, top[0][1] - 1, "")] + body = [(t, p) for t, p in top if p < last_body] + for i, (t, p) in enumerate(body): + end = (body[i + 1][1] - 1) if i + 1 < len(body) else last_body - 1 + title = re.sub(r"^Chapter \d+:\s*", "", t) + secs.append((ch(i + 1), title, p, end, "")) + secs.append(("Back-Matter", "Acknowledgements", last_body, len(d), "")) + return secs + + BUILDERS = { "Context Engineering": build_cetx, "The 12 Week Year": build_12wy, "Your Best Year Ever": build_ybye, "Spec Driven Development": build_sdd, "FOCUS Architecture": build_focus, + "Inference Engineering": build_inference, }