Book-Reader-Assist/library/Inference Engineering/Chapter-06-Software/Chapter-06-source-text.md

Inference Engineering — Chapter-06: Software

• Source: /library/Inference Engineering/source-file.pdf
• PDF pages: 95–118
• Pages without text: 95, 96, 118

---

CHAPTER 4 Software

Software  95

Software NVIDIA’s market dominance in the inference space is in no small part due to the robust and mature software ecosystem around its hardware. Hardware iteration cycles are slow. Best-in-class hardware companies like Apple and NVIDIA release new architectures and generations at most yearly, with two-year release cycles being more common. But software iteration is fast. Often, to run a newly released open model on day zero, you need to install a nightly build or other prerelease version of each of your software dependencies just to get support for the new model. Software’s fast iteration cycle and lower barrier to entry dramatically expands the landscape of inference engineering. While hardware centers on NVIDIA and a few competitors, there are countless companies building software at various levels of the inference stack. For inference engineers, these are some key players: • NVIDIA: Invests heavily in its own sometimes-proprietary software eco- system, from CUDA up to Dynamo. • Hugging Face: Maintains a model registry for all open models plus transformers and diffusers. • The Linux Foundation: Maintains hardware-agnostic projects like PyTorch and vLLM. • LMSYS Org: Develops essential tools for inference and evaluation, most notably SGLang. There are thousands more companies, universities, and research institu- tions making essential open-source contributions to inference. The software space is too big and too fast to exhaustively document in any book, much less in a single chapter. Instead, this chapter presents foundational technologies with long-term relevance. Throughout the chapter, technologies are presented with increasing levels of abstraction:

96  Chapter 4: Software • CUDA: Direct communication to the GPU for explicit control over com- putations and memory (section 4.1). • Deep learning frameworks: Abstractions over CUDA for training, exporting, and running neural networks in Python (section 4.2). • Inference engines: Highly configurable PyTorch-backed inference for common architectures (section 4.3). • NVIDIA Dynamo: Sits on top of inference engines to power large-scale deployments (section 4.4). Most inference engineering today happens at the higher levels of abstrac- tion, configuring and deploying inference engines and orchestrating infer- ence across multiple GPUs. No matter what level of the stack you work at, it’s essential to have a strong mental model for the adjacent levels of abstraction to guide your work. 4.1  CUDA CUDA is how you write code to run on NVIDIA GPUs. More formally, CUDA is NVIDIA’s proprietary computing platform and pro- gramming model for executing parallel tasks on GPUs. Both “platform” and “programming model” are broad definitions; it’s easier to look at CUDA in its component parts: • CUDA kernel: A user-defined function that executes parallelized code on the GPU. • CUDA graph: A directed acyclic graph (DAG) of kernels and other GPU operations for optimizing repeated workflows. • CUDA driver: A low-level interface between the application and the GPU hardware to manage memory and execution. • CUDA runtime: A developer-facing API for launching kernels and man- aging memory. CUDA – which stands for Compute Unified Device Architecture, though the acronym is rarely expanded today – is the foundation for the entire generative AI ecosystem on NVIDIA GPUs.

4.1 CUDA 97 CUDA is not a programming language. Instead, CUDA programs are spec- ifi ed in a programming language, most often C++, then compiled into separate CPU and GPU code by a compiler like nvcc. A kernel is just a function that does some parallel computation. Whenever you see the phrase “CUDA kernel” you can replace it with “a piece of code written for NVIDIA GPUs.” For a “Hello, World!” kernel, consider these six lines of C++ code which take in an array of length n and double each element in the array. Figure 4.1: Example CUDA kernel doubles each value in an array. Ordinarily, on a CPU, this function would run in linear time, with each ele- ment in the array being doubled sequentially. But on a GPU, thousands of elements can be processed simultaneously, making this function much more efficient. Writing CUDA kernels shifts inference engineering from thinking about algorithms to thinking about implementations. For example, the traditional attention algorithm that is central to generative AI can be expressed in a few dozen lines of code. However, FlashAttention, which is the same mathematical operation, takes tens of thousands of lines of code to implement the algorithm in a more memory-efficient manner for a spe- cifi c GPU.

98  Chapter 4: Software 4.1.1  CUDA Kernels for Inference Writing CUDA kernels does not mean building from scratch. The prior art to build upon predates CUDA by decades. BLAS (Basic Linear Algebra Subprograms), first implemented for Fortran in the 1970s, is a specification for common linear algebra operations from dot products to matrix multiplication. cuBLAS, the CUDA implementation for BLAS, brings this specification to CUDA in the form of pre-built kernels for essential linear algebra opera- tions. Similarly, cuDNN (CUDA Deep Neural Network) provides primitives for neural networks. Within BLAS, the most frequently used operation in inference is GEMM (General Matrix-Matrix Multiplication). Every linear layer in a model uses matrix multiplication, and cuBLAS provides a strong starting point. But you aren’t limited to the cuBLAS implementation. As operations like GEMM are so essential for inference, you may need more fine-grained control. You might write different GEMM kernels for matrices of different shapes or to better run on specific GPU architectures. CUTLASS is a template library that provides building blocks for writing high-performance kernels. For example, FlashAttention 3 uses CUTLASS. CuTe, another template library, introduces abstractions for tiled tensor operations for recent architectures. With tools like CUTLASS and CuTe, kernels can be written at a higher level of abstraction while retaining strong performance. Another source for kernels is FlashInfer, a library with high-performance implementations of kernels for LLM inference, including many optimized attention kernels and fused sampling functions. 4.1.2  CUDA Kernel Selection Most inference engineers will never need to write their own kernels. How- ever, kernel selection – choosing the best kernel from a range of options – is an important part of inference optimization.

4.1.2  CUDA Kernel Selection  99 Kernel implementations are highly specialized. CUDA exposes low-level APIs for memory management and parallel computing, and kernel engi- neers make implementation decisions based on the exact specifications of the hardware they’re writing for. It’s important to understand how closely tied kernel implementations are to specific hardware details. Kernels often have hard-coded values based on the memory bandwidth or the number or layout of Tensor Cores on a given GPU. A kernel written for an H100 will likely fail to take advantage of the archi- tecture and extra memory of a B200, while a kernel written for that B200 could be backwards incompatible with the previous-generation Hopper architecture. With each generation of GPUs, porting handwritten kernels to run optimally on the new architecture takes substantial engineering work. Most kernel selection is automatic. Deep learning frameworks and infer- ence engines have pre-configured kernels for various architectures, while PyTorch and TensorRT-LLM include automatic kernel selection in their compilation steps. However, you might manually choose a few kernels for essential algo- rithms to speed up inference. For example, most production-ready GEMM kernels come from cuBLAS. But when the DeepSeek lab released their updated version of Deep- Seek-V3, they also released DeepGEMM, which provides more efficient GEMM kernels for running matrix multiplication in FP8 on the Hopper GPU architecture. Manual kernel selection lets you insert a DeepGEMM kernel as a plugin to speed up a specific step in inference, like multiplying two matrices of precise dimensions. Just be careful to ensure compatibility. For example, if you were to upgrade to a B200 GPU, you’d have to either swap the kernel back, wait for Blackwell support in DeepGEMM (now supported as of the time of publication), or port the kernel yourself.

100  Chapter 4: Software 4.1.3  Reducing Memory Accesses with Kernel Fusion Running two different kernels back-to-back on the same data results in wasted reads from and writes to memory. As a simple example, imagine two kernels: multiply_by_2 and multiply_by_3. If these kernels were run back-to-back, the sequence would be:

1. Read the input vector [1, 2, 3] from memory.
2. Run multiply_by_2 on the vector.
3. Save the output vector [2, 4, 6] to memory.
4. Read the new input vector [2, 4, 6] from memory.
5. Run multiply_by_3 on the vector.
6. Save the final output vector [6, 12, 18] to memory. The inefficiency is clear: steps three and four comprise an unnecessary round trip to memory. During decode, the bandwidth-bound phase of LLM inference, an inference engine can’t afford unnecessary reads from or writes to memory. Kernel fusion is the process of taking two or more kernels and re-imple- menting them into a single kernel that handles both operations. In this example, the fused kernel would be multiply_by_6 and the new order of operations would be:
7. Read the input vector [1, 2, 3] from memory.
8. Run multiply_by_6 on the vector.
9. Save the final output vector [6, 12, 18] to memory. In practice, kernel fusion is far more complicated – functions are more complex, and data overlap isn’t as clean. But there are common patterns in kernel fusion for inference, like combining matrix multiplication, bias adding, and activation.

4.2 Deep Learning Frameworks and Libraries 101 Figure 4.2: Kernel fusion reduces reads and writes between memory and compute within a GPU. Kernel fusion can be an automatic or a manual process. Compilers can identify straightforward fusion opportunities and automatically create fused kernels. But more sophisticated algorithms, like FlashAttention, require handwritten fused kernels, which are used via plugins during inference. 4.2 Deep Learning Frameworks and Libraries Deep learning frameworks and libraries are the bridge between working directly in CUDA and using off-the-shelf inference engines like vLLM. These libraries are used in both training and inference. Over the past few years, PyTorch has emerged as the clear leader at this level of the stack. There are two other frameworks that, for concision, I will only mention briefl y: • TensorFlow: An end-to-end machine learning platform officially sup- ported by Google, TensorFlow was prominent in the 2010s ML era but has fallen out of favor today. • JAX: A research project unofficially associated with Google, JAX pres- ents a simpler interface without as many legacy features and opera- tions. However, as the documentation warns, expect sharp edges.

102 Chapter 4: Software The remainder of this section focuses on PyTorch and the technologies around and above it in the stack. 4.2.1 PyTorch PyTorch is a Python package for describing tensor operations. Originally created at Meta and now a part of the Linux Foundation, PyTorch is the industry standard technology underlying both training and inference for generative AI models. Personally, I’ve been a Python programmer for my entire career, and I fi nd writing low-level C++ difficult. With PyTorch, I can write highly per- formant inference code in Python for both CPUs and GPUs, but I always have the option to dip down into CUDA when necessary by plugging in specifi c kernels. PyTorch can train any kind of neural network. The PyTorch documentation shows a basic neural network example: Figure 4.3: A basic neural network in PyTorch, adapted from the PyTorch documentation. While this is a very simple example, you may recognize the linear layers and ReLU activation functions discussed in chapter 2 as key pieces of neural networks.

4.2.2  Model File Formats  103 PyTorch automatically computes gradients for any differentiable function via its autograd module. This is what makes PyTorch so powerful for training – you define a computation graph, and you get a gradient to train against. But PyTorch is special because it isn’t just great for training, it’s also powerful for inference. PyTorch balances built-in functions and automatic performance optimizations with manual control where you need it. The step that transforms a model from training to inference is compila- tion. PyTorch compilation (torch.compile) targets a specific GPU and performs automatic kernel selection and kernel fusion to ensure optimal performance. torch.compile can’t fuse plugin kernels like DeepGEMM, FlashAtten- tion, or custom kernels. This limits its utility for LLM inference, where most kernels are custom. However, PyTorch compilation is useful for optimizing less common model architectures and compiling long sequences of light- weight kernels. When you are optimizing a model that has a custom or rare architecture, you may have to rewrite functions to be more abstract – especially with respect to Python-specific language features – for com- pilation to succeed. PyTorch alone is a powerful and flexible tool for building high-performance inference services. But there is a rich ecosystem on top of PyTorch that makes it faster to implement, compile, and execute optimized code for common model architectures. 4.2.2  Model File Formats The dominant file format for serializing model weights is safetensors, cre- ated by Hugging Face. Safetensors is a replacement for generic formats like bin designed specifi- cally for holding model weights. The safety in safetensors comes from the fact that unlike a general format that can execute arbitrary Python code during deserialization, safetensors only hold tensor data, not executable code.

104  Chapter 4: Software Generative AI models have hundreds of gigabytes of weights. These weights are split across dozens of safetensors files. The safetensors for- mat uses memory mapping to ensure that the files are loadable without allocating the full memory, which makes loading model weights faster and safer. Another leading format, ONNX (Open Neural Network Exchange), stores weights along with an execution graph for the model. Where the safeten- sors format separates the weights from the architecture, ONNX bundles them together. ONNX files are highly portable. With deep integration into PyTorch and support for multiple hardware options, ONNX is a great alternative to safetensors for when you want to store model graphs, not just weights. 4.2.3  ONNX Runtime and TensorRT ONNX Runtime and TensorRT are high-performance inference runtimes. PyTorch models can be exported to the ONNX format, which ONNX Run- time can execute directly or TensorRT can compile into a highly optimized engine. ONNX Runtime TensorRT Open source, associated with the Linux Foundation Mix of proprietary and open components, built by NVIDIA First-class exporter in the PyTorch ecosystem Integrated with PyTorch via Torch- TensorRT Supports many types of GPUs NVIDIA GPUs only ONNX Runtime is an open community standard, while TensorRT is spe- cific to NVIDIA GPUs. The export process looks somewhat like Torch compilation. However, these standards do not support every data structure, type, and operation within PyTorch. The export process can identify these issues in PyTorch code, but this gets tricky with more complex models. An example is DeepSeek V3, which introduces Multi-Latent Attention (MLA). MLA, as implemented in PyTorch, is difficult to export, but the

4.2.4  Transformers and Diffusers  105 transformers architecture overall is simple enough that hand-fusing ker- nels is feasible. Today, it’s increasingly popular to skip directly from PyTorch to an inference engine like vLLM or TensorRT-LLM for models that these engines support, bypassing the intermediate representation step and exporting weights only as safetensors. ONNX Runtime and TensorRT are still widely used – Ten- sorRT especially for its strong out-of-the-box runtime for image and video models – but the industry is bifurcating between the control of handwritten PyTorch code or the convenience of prebuilt inference engines. 4.2.4  Transformers and Diffusers The transformers and diffusers libraries by Hugging Face are built on PyTorch but are not designed to run large-scale production inference. Instead, these libraries offer reference implementations of models for inference engineers to learn from and adapt. While these libraries are toolboxes for tinkering, they do include essential information about models and useful utilities for building model inference servers. The config.json file that ships with models implemented for transformers and diffusers contains essential information, and the libraries’ utilities for Hugging Face operations like downloading model weights are widely used. You’ll find transformers or diffusers sample code in the model card for most popular open models on Hugging Face. This sample code is great for understanding the exact input and output spec of a model or for running local inference and notebooks. But for production, you’ll want to either write and compile PyTorch code directly or use a production-ready inference engine. 4.3  Inference Engines There are three competitive inference engines on the market: vLLM, SGLang, and TensorRT-LLM. These frameworks offer good out-of-the-box performance for LLMs and other modalities with similar architectures (chapter 6).

106  Chapter 4: Software In late 2025, vLLM and SGLang also began supporting some image and video generation models via vLLM Omni and SGLang Diffusion, respec- tively. TensorRT-LLM does not support image or video generation models. Inference engineers can also use TensorRT or PyTorch directly to run these models (section 6.5). Inference engines are powerful because they are configurable. Working with pre-optimized components at a higher level of abstraction, inference engineers can spend their time testing combinations of techniques rather than repeating routine implementations. At a very high level, vLLM and SGLang are more general tools that are easier to adopt and have day zero support for more models, while Ten- sorRT-LLM has a steeper learning curve but usually achieves the best performance. Engine vLLM SGLang TensorRT-LLM Performance Good Good Best Ease of use Easy Easy Hard Model support Most Most Some Hardware GPU, TPU NVIDIA, AMD NVIDIA only License Apache 2.0 Apache 2.0 Apache 2.0 Each framework runs out of the box with core features like continuous batching and supports the main performance optimization techniques – post-training quantization, speculative decoding, prefix caching, parallel- ism, disaggregation. At Baseten, we use all three frameworks, though we use TensorRT-LLM the most. Inference engineers should be familiar with all three and select on a deployment-by-deployment basis. 4.3.1  vLLM vLLM has the largest market share among inference engines. GitHub stars are a rough measure for popularity, but at the time of publication vLLM has twice as many stars as SGLang and TensorRT-LLM combined.

4.3.1 vLLM 107 First released in the summer of 2023, vLLM is the oldest of these inference engines by a few months. Originally created at UC Berkeley, vLLM is now hosted by The PyTorch Project within The Linux Foundation. vLLM’s best selling point is its broad support. It supports the most hard- ware options – NVIDIA, AMD, and Intel GPUs along with Google TPUs – as well as the most models and architectures. Just about every open LLM out there integrates with vLLM from day zero. vLLM also supports multimodal inference via vLLM Omni, which extends the engine to support image, audio, and video inputs and outputs. One of the core principles of inference engineering is that the more constraints you can introduce, the better performance you can achieve. vLLM’s broad platform can achieve impressive performance results when properly confi gured, but in my experience it falls short of the highest-end performance possible with narrow frameworks like TensorRT-LLM. vLLM’s developer experience is built around the vllm serve command, with server confi guration passed in as fl ags. Figure 4.4: vLLM inference example on eight GPUs. vLLM is pip-installable and provides official Docker images with pre-bun- dled dependencies and support for various hardware architectures. You should use vLLM when: • You want to quickly stand up a model server that will offer solid perfor- mance out of the box for almost any open model. • You want to run an “Omni” model with multiple input and output modal- ities.

108 Chapter 4: Software • You are using a smaller GPU or older architecture where TensorRT-LLM offers few performance benefi ts. 4.3.2 SGLang SGLang is the other major community-driven fast inference framework. First released in December 2023, SGLang has risen to prominence along- side Chinese open models like DeepSeek and Qwen and is the engine of choice for inference at xAI. SGLang’s unique angle on the problem of model serving is expressed in its developer experience, which pairs a fast backend runtime with a fl exi- ble frontend language. In practice, that means you can choose individual components of your engine for deep customization without needing to rewrite everything else from scratch. SGLang supports both NVIDIA and AMD GPUs, and has strong day-zero support for a wide range of models. SGLang works closely with labs like DeepSeek, Qwen, Kimi, and Z AI to release optimized implementations of new architectural features like DeepSeek’s Multi-Latent Attention. SGLang has invested heavily in supporting large-scale deployments of MoE LLMs, specifi cally multi-node deployments on systems like GB200 NVL72 for high throughput. These systems offer extremely cost-efficient inference for large models with signifi cant traffic. SGLang’s developer experience is built around the sglang.launch_server command, with server confi guration passed in as fl ags. Figure 4.5: SGLang inference example on eight GPUs.

4.3.3  TensorRT-LLM  109 SGLang also supports image and video generation model inference via SGLang Diffusion. SGLang Diffusion introduces a pipeline abstraction which orchestrates a number of stages. This flexible approach maps closely to the architec- ture of image and video generation models. For performance, SGLang Diffusion adds support for various diffusion-specific parallelism methods and re-uses the scheduler and optimized kernels from the main SGLang package. You should use SGLang when: • You want excellent out-of-the-box throughput with decent latency on large MoE models like DeepSeek and Kimi. • You want the inference engine experience for image and video gener- ation models. • You want control and customization and are excited to participate in the SGLang community. 4.3.3  TensorRT-LLM TensorRT-LLM is NVIDIA’s open-source inference engine. Of the three main options, TensorRT-LLM offers the highest performance and the most flexibility to expert users. A note on naming: There are two major versions of TensorRT-LLM. Only the older version is actually related to TensorRT: • TensorRT-LLM V0 (0.X.Y): Major versions starting with zero are a plugin for NVIDIA TensorRT. • TensorRT-LLM V1 (1.X.Y): Major versions starting with one are a standalone package based on PyTorch with no dependency on Ten- sorRT. Originally, TensorRT-LLM built a TensorRT engine for serving lan- guage models. With the modern PyTorch-based version, TensorRT-LLM bypasses the intermediate representation of TensorRT and uses PyTorch directly.

110 Chapter 4: Software TensorRT-LLM V1 was released in the summer of 2025. Deployments of the previous major version are still common – always be sure to check which version you are using. TensorRT-LLM achieves the best performance in large part because it has access to kernels written by NVIDIA engineers, including some closed- source kernels. These handwritten and manually fused kernels offer excel- lent support for the latest hardware architectures like Hopper and Blackwell and for NVIDIA-specifi c number formats like NVFP4. TensorRT-LLM offers a robust implementation of in-fl ight batching (token- level continuous batching), which helps with throughput. It also supports just about every model performance optimization setting you could ask for, including quantization, speculation algorithms, prefi x caching, chunked prefi ll, fl exible parallelism, and disaggregation. With V1, TensorRT-LLM introduces a developer experience that looks a lot like vLLM and SGLang. However, in addition to fl ag arguments on the trtllm-serve command, it expects a config.yaml fi le for deeper customization. Figure 4.6: TensorRT-LLM inference example on eight GPUs. The best way to install TensorRT-LLM is by running it via one of NVIDIA’s official Docker containers

4.4  NVIDIA Dynamo  111 Use TensorRT-LLM when: • You are running a well-supported model architecture on a Hopper or later GPU. • You are willing to do extra engineering work to get the best possible performance. • Optionally, you are planning to use NVIDIA Dynamo for serving and want the most deeply integrated engine. 4.4  NVIDIA Dynamo NVIDIA Dynamo is a distributed system for model serving first announced at NVIDIA GTC in March 2025. Dynamo works with every inference engine – vLLM, SGLang, and Ten- sorRT-LLM – as backends, with Dynamo itself providing an orchestration layer for large-scale deployments. Dynamo provides support for essential model performance techniques: • KV cache re-use: Retaining KV information between requests and routing requests based on prefix match. • Disaggregation: Separating prefill and decode onto individually opti- mized engines with independent scaling. • Multi-node parallelism: Optionally using two or more nodes of GPUs in a single replica for a model, usually with Expert Parallelism. Each of these techniques will be detailed in chapter 5. As with the infer- ence engines, there is a lot of work for inference engineers to do to con- figure Dynamo for their use case and achieve maximum performance. Dynamo’s thoughtful abstractions for distributed KV routing, disaggrega- tion, and multi-node model parallelism provide high-performance aggre- gation of information during runtime, allowing for real-time adjustments to configuration as traffic fluctuates. For example, you can automatically scale up and down prefill and decode workers with an SLA-based planner operating on user-defined TTFT and TPS constraints.

112  Chapter 4: Software As a general principle, the more scale you have, the more tools and tech- niques there are available to you for inference optimization. Dynamo is built for scale: big models, big traffic. It excels at serving foun- dation models like the trillion-parameter Kimi family to a large number of concurrent users. For smaller models, Dynamo can still offer moderate performance improvements on large-scale deployments. If you’re building an inference API for a built-from scratch foundation model or serving an open model in a high-usage product, Dynamo is a great choice. But many deployments don’t need the additional complexity of Dynamo. Unless you’re operating with enough volume for disaggregation and KV-aware routing to matter, Dynamo will be unnecessary work and excess overhead. In these cases, you can use inference engines directly. Dynamo is the newest project covered in this chapter, and features are still being built out. Dynamo is open source under the Apache 2.0 license. The community around Dynamo is active, and the project welcomes con- tributions with a public CI and support from NVIDIA engineers. 4.5  Performance Benchmarking and Load Testing Benchmarking is an essential part of model performance optimization. Without precise, accurate performance benchmarks, there’s no way of knowing if your optimizations are actually working. A high-quality benchmark simulates real life as closely as possible. The best benchmark is to shadow real-world production traffic onto the system you are testing. Shadowing is the process of copying incoming requests onto the test system so that you can benchmark its performance without affecting the original request. If you can’t shadow real usage, you’ll need to simulate it. LLM performance is affected by a number of factors. When simulating traffic, you need to match your expected production workload on multiple dimensions:

4.5.1  Performance Benchmarking Tooling  113 • Sequence lengths: Time to first token and memory usage rely on the input sequence length (ISL) and output sequence length (OSL), mean- ing the number of tokens in the prompt and response. • Volume and pattern of traffic: Batching and server load depend on the number of concurrent requests. Jitter traffic to mimic real usage. • Request contents: The actual prompt within each request affects per- formance factors like cache hit rate and draft token acceptance. • Input parameters: Settings like temperature and reasoning effort that affect inference should be set to their anticipated production values. Remember, optimization is about tradeoffs and constraints. If you’re max- imizing benchmark performance against bad inputs, performance in pro- duction won’t match expectations. 4.5.1  Performance Benchmarking Tooling As a performance benchmark should closely reflect production traffic, everyone’s benchmarking setup should look a bit different. But there are a few common tools: • SGLang Genai-bench: A CLI and dashboard by the SGLang team for benchmarking models deployed with any inference framework. • NVIDIA GenAI-Perf: A client-side tool by NVIDIA for measuring latency and throughput on varied traffic. • Locust: An open-source load-testing tool, not specific to generative AI systems, that simulates as many as millions of simultaneous users. Another great tool for benchmarking is open-source evals datasets – from general evals like MMLU and gsm8k to domain-specific evals like SWE- bench. While the purpose of benchmarking work is to measure performance, not model output quality, these eval datasets serve two purposes: acting as a set of varied and realistic inputs, and spot checking that performance optimizations haven’t impacted model output quality. When possible, choose an eval dataset that matches the expected use of your production system, like HumanEval when reducing latency for a code completion system.

114 Chapter 4: Software 4.5.2 Performance Benchmarking Tips Along with being realistic, great benchmarks are also consistent. Make sure your benchmarks send enough traffic to get a good read on perfor- mance without being swayed by outliers. When in doubt, run a benchmark multiple times and average the results. Before you do any performance optimization work, start with a solid base- line benchmark. As you test optimizations, keep a consistent confi guration in your benchmarking setup, and test each optimization individually as well as collectively to fully understand what is driving performance improve- ment. In some cases, optimizations can work against each other, like trying to run speculative decoding with large batch sizes. The principle of changing one thing at a time applies to your benchmarking confi guration as well. It’s common to need to test various traffic patterns or sequence shapes, but as with any experiment only change one variable at a time to ensure that you are getting clear results. 4.5.3 Profi ling Performance Profi ling is one click deeper than benchmarking. Where a benchmark gives a single fi gure (e.g., the P90 TTFT is 350 ms), a profi ling tool shows where each of those milliseconds was spent in the inference process. Benchmarking tells you how your system is performing; profi ling tells you why it’s performing that way. Figure 4.7: A kernel profi ler shows you how long each operation within a kernel takes to execute, revealing bottlenecks. Most inference engineers won’t need to do profi ling as part of their daily work. When using an already high-performance tool like the inference

4.5.3  Profiling Performance  115 engine TensorRT-LLM, your workflow is a cycle of configuration and benchmarking – profiling would be extraneous. However, if you’re contributing to an inference framework like vLLM or SGLang, writing your own inference service in PyTorch, or operating at the cutting edge of a new modality like video generation, performance profiling should be part of your toolkit. The most popular profiling tools for inference are: • PyTorch Profiler: An easy-to-use profiling library for capturing step-by- step performance metrics (CPU time, GPU time, memory usage) during inference. • NVIDIA Nsight Systems (NSys): A featureful but complex tool for GPU and CPU sampling and tracing that provides system-wide analy- sis across multiple GPUs and their interconnects. • NVIDIA Nsight Compute (NCU): A profiling utility and CLI for in-depth analysis of individual CUDA kernels on both compute and memory usage. In addition, frameworks like TensorFlow and TensorRT ship with their own built-in profilers. Profilers are valuable because they give you granular information about compute and memory usage, which guides your optimization work toward improving the most expensive steps in your inference pipeline. For example, using PyTorch Profiler you might find that activation functions are taking an unusually long time due to excess memory reads, and figure out how to write a fused kernel that runs activations alongside attention to prevent the excess reads. Then, you would insert that new kernel into your PyTorch code and re-run system-level benchmarks to see if you’ve achieved your latency targets. Together, profiling and benchmarking give you the information you need to improve system performance and, eventually, the confidence to deploy your optimizations in production.