13KB
Inference Engineering — Front-Matter: Front matter (cover, title, contents)
- Source: /library/Inference Engineering/source-file.pdf
- PDF pages: 1–10
- Pages without text: 1, 2, 10
INFERENCE ENGINEERING BY PHILIP KIELY
Inference Engineering by Philip Kiely Published by Baseten Books Copyright © 2026 Baseten Labs, Inc All rights reserved. No portion of this book may be reproduced in any form without written permission from the publisher, except as permitted by U.S. copyright law. For permissions contact Baseten. Cover art by Luke de Haas Edited by Robin Bourjaily ISBN: 979-8-9943597-2-3
Table of Contents Preface....................................................................................................9 Chapter 0: Inference............................................................................15 Chapter 1: Prerequisites ....................................................................23 1.1 Scale and Specialization..............................................................26 1.2 About Your App............................................................................27 1.2.1 AI-Native Applications......................................................28 1.2.2 Online versus Offline........................................................29 1.2.3 Consumer versus B2B......................................................30 1.3 Model Selection............................................................................31 1.3.1 Model Evaluation..............................................................31 1.3.2 Fine-Tuning for Domain-Specific Quality..........................32 1.3.3 Distillation.........................................................................33 1.4 Measuring Latency and Throughput............................................35 1.4.1 Latency Percentiles..........................................................36 1.4.2 End-to-End Metrics...........................................................37 Chapter 2: Models................................................................................39 2.1 Neural Networks...........................................................................42 2.1.1 Linear Layers and Matmul................................................44 2.1.2 Activation Functions.........................................................44 2.2 LLM Inference Mechanics............................................................46 2.2.1 LLM Architecture..............................................................49 2.2.2 Transformer Blocks..........................................................50 2.2.3 Attention...........................................................................52 2.2.4 Mixture of Experts Models................................................53 2.3 Image Generation Inference Mechanics......................................55 2.3.1 Image Generation Model Architecture..............................57 2.3.2 Few-Step Image Generation Models................................59 2.3.3 Video Generation..............................................................59
4 Table of Contents 2.4 Calculating Inference Bottlenecks................................................61 2.4.1 Ops:Byte Ratio and Arithmetic Intensity...........................62 2.4.2 LLM Inference Bottlenecks...............................................63 2.4.3 Image Generation Inference Bottlenecks.........................67 2.5 Optimizing Attention.....................................................................67 Chapter 3: Hardware............................................................................71 3.1 GPU Architecture.........................................................................74 3.1.1 Compute...........................................................................74 3.1.2 Memory and Caches........................................................76 3.2 GPU Architecture Generations.....................................................77 3.2.1 Hopper GPUs...................................................................79 3.2.2 Ada Lovelace GPUs.........................................................80 3.2.3 Blackwell GPUs................................................................81 3.2.4 Rubin GPUs......................................................................81 3.2.5 Grace and Vera CPUs......................................................82 3.3 Instances......................................................................................83 3.3.1 Multi-GPU Instances.........................................................84 3.3.2 Multi-Instance GPUs.........................................................86 3.4 Other Datacenter Accelerator Options.........................................87 3.5 Local Inference.............................................................................89 3.5.1 Desktop Inference............................................................90 3.5.2 Mobile Inference...............................................................91 Chapter 4: Software.............................................................................93 4.1 CUDA...........................................................................................96 4.1.1 CUDA Kernels for Inference.............................................98 4.1.2 CUDA Kernel Selection....................................................98 4.1.3 Reducing Memory Accesses with Kernel Fusion...........100 4.2 Deep Learning Frameworks and Libraries.................................101 4.2.1 PyTorch..........................................................................102 4.2.2 Model File Formats.........................................................103 4.2.3 ONNX Runtime and TensorRT.......................................104 4.2.4 Transformers and Diffusers............................................105 4.3 Inference Engines......................................................................105 4.3.1 vLLM...............................................................................106
Table of Contents 5 4.3.2 SGLang..........................................................................108 4.3.3 TensorRT-LLM...............................................................109 4.4 NVIDIA Dynamo.........................................................................111 4.5 Performance Benchmarking and Load Testing..........................112 4.5.1 Performance Benchmarking Tooling..............................113 4.5.2 Performance Benchmarking Tips...................................114 4.5.3 Profiling Performance.....................................................114 Chapter 5: Techniques......................................................................117 5.1 Quantization...............................................................................120 5.1.1 Number Formats.............................................................121 5.1.2 Quantization Approaches...............................................125 5.1.3 Measuring Quality Impact...............................................128 5.2 Speculative Decoding................................................................129 5.2.1 Draft-Target Speculative Decoding................................131 5.2.2 Medusa...........................................................................132 5.2.3 EAGLE............................................................................133 5.2.4 N-gram Speculation and Lookahead Decoding..............134 5.3 Caching......................................................................................136 5.3.1 Prefix Caching and KV Cache Re-Use...........................136 5.3.2 Where to Store the KV Cache........................................139 5.3.3 Cache-Aware Routing....................................................140 5.3.4 Long Context Handling...................................................141 5.4 Model Parallelism.......................................................................142 5.4.1 Tensor Parallelism for Lower Latency............................144 5.4.2 Expert Parallelism for Higher Throughput......................145 5.4.3 Multi-Node Inference......................................................146 5.5 Disaggregation...........................................................................148 5.5.1 How Disaggregation Works............................................148 5.5.2 When to Use Disaggregation..........................................149 5.5.3 Dynamic Disaggregation with NVIDIA Dynamo..............150 Chapter 6: Modalities .......................................................................153 6.1 Vision Language Models............................................................156 6.1.1 Video Processing for Vision Language Models..............158 6.1.2 Omni-Modal Models.......................................................159
6 Table of Contents 6.2 Embedding Models....................................................................159 6.2.1 Embedding Model Architecture......................................160 6.2.2 Embedding Model Inference...........................................161 6.3 ASR Models...............................................................................162 6.3.1 Single-Chunk Latency Optimization...............................163 6.3.2 Long File Latency Optimization......................................164 6.3.3 Diarization.......................................................................166 6.4 TTS Models................................................................................166 6.4.1 Streaming Real-Time Text to Speech............................168 6.4.2 Speech-to-Speech Models.............................................168 6.5 Image Generation Models..........................................................169 6.5.1 Image Generation Kernel Optimization..........................170 6.5.2 One Weird Trick for Faster Image Generation...............172 6.6 Video Generation Models...........................................................173 6.6.1 Attention Optimization and Quantization........................174 6.6.2 Context Parallelism.........................................................176 Chapter 7: Production.......................................................................177 7.1 Containerization.........................................................................179 7.1.1 Dependency Management.............................................181 7.1.2 NIMs...............................................................................183 7.2 Autoscaling.................................................................................183 7.2.1 Concurrency and Batch Sizing.......................................186 7.2.2 Cold Starts......................................................................188 7.2.3 Routing, Load Balancing, and Queueing........................190 7.2.4 Scale to Zero..................................................................192 7.2.5 Independent Component Scaling...................................192 7.3 Multi-Cloud Capacity Management............................................193 7.3.1 GPU Procurement..........................................................195 7.3.2 Geo-Aware Load Balancing............................................196 7.3.3 Building for Reliability.....................................................196 7.3.4 Security and Compliance................................................198 7.4 Testing and Deployment............................................................199 7.4.1 Zero-Downtime Deployment...........................................200 7.4.2 Cost Estimation..............................................................201 7.4.3 Observability...................................................................203 7.5 Client Code................................................................................204
Table of Contents 7 7.5.1 Client Latency Overhead................................................205 7.5.2 Asynchronous inference.................................................205 7.5.3 Streaming and Protocol Support....................................205 7.6 Production Inference with Baseten............................................208 Appendix A: Inference Glossary..........................................................209 Appendix B: Recommended Reading..................................................231 Architecture................................................................................234 Developer Tools.........................................................................239 Frontier Open Models................................................................243 GPU Infrastructure.....................................................................246 Inference Optimization Research...............................................249 Intelligence Evaluation...............................................................254 Acknowledgements..............................................................................257