42KB
Inference Engineering — Chapter-09: Production
- Source: /library/Inference Engineering/source-file.pdf
- PDF pages: 179–210
- Pages without text: 179, 180
CHAPTER 7 Production
Production 179
Production The purpose of inference engineering is to make generative AI models faster, less expensive, and more reliable to operate, which in turn lets you build better products. This promise is only fulfilled if your inference engineering work makes it to production and scales alongside the hypergrowth and viral spikes that successful AI products generate. When you scale production traffic, your assumptions are rigorously tested. Everything from sequence shapes to traffic patterns to what topic a user decides to chat about impacts your observed performance in production. And maintaining secure, robust infrastructure is an entirely different skillset from optimizing model inference on the GPU. No matter how fast and efficiently a single instance can serve a model, with enough traffic, the service will be overwhelmed. That is not a PyTorch problem or a CUDA problem, it’s an infrastructure problem, and requires a different mindset and different technologies. Scaling in production introduces new complexities about where and how to get GPUs, balance traffic across them, and prevent downtime. Plus, cost accounting gets messy in the transition from paying per million tokens to paying directly for infrastructure. Latency in production comes from more than just prefill and decode. You need to evaluate your system end-to-end and eliminate inefficiencies in the server, the network, and even the client in situations where you can own or influence client code. This chapter introduces the essential considerations for scaling low-la- tency, high-throughput inference in production. And at the end, I’ll invite you to try Baseten for deploying mission-critical inference workloads. 7.1 Containerization Containerization is the practice of packaging an application together with its dependencies to standardize deployment in production. Containers
180 Chapter 7: Production turn a program into a packaged artifact that can run anywhere – no more “it works on my machine.” Containers are lightweight because they share the underlying host operating system kernel (in this context, a kernel refers to a Linux kernel, not a CUDA kernel). This makes containers well-suited for packaging inference services. For most developers, containerization is synonymous with Docker. Work- ing with containers introduces more specific terminology: • Container: An actively running environment that isolates an application and its dependencies. • Image: An executable package that contains everything you need to run a piece of software. • Dockerfile: A human-readable file with well-specified, machine-inter- pretable instructions for creating an image. • Registry: A central repository for managing, storing, sharing, and dis- tributing images. NVIDIA, several cloud providers, and Docker themselves all operate con- tainer registries. A popular registry for AI is Docker Hub – Docker Hub is to images as Hugging Face is to model weights or PyPi is to Python packages. Docker containers are composed of layers. You can take a pre-configured base image and add other layers with additional filesystem changes on top of it. There are three types of layers: • Base image: Either an operating system distribution like Ubuntu or a more complex image taken from a registry. The base image itself is composed of multiple layers. • Additional layers: Filesystem changes including dependencies, applica- tion code, and configuration files as specified by Dockerfile instructions. • Container layer: A thin, ephemeral layer created at runtime. Any changes to the running container, like creating, updating, or deleting files, are written to this layer and are lost when the container terminates.
7.1.1 Dependency Management 181 Figure 7.1: Docker containers are composed of layers, from a base image up to an ephemeral, writable top layer. Inference engines like vLLM and SGLang offer official base images for active releases. It’s generally a good idea to start from one of these proven images, rather than building your own from scratch. 7.1.1 Dependency Management Dependency chains for inference are long and fragile. Getting to a work- ing build is hard, which makes containerization essential for preserving a known good build in an ecosystem where breaking changes are all too common. Images are built for a specifi c GPU architecture and model. A container includes many runtime components: • CUDA toolkit version: The specifi c versions of CUDA, cuDNN, and drivers compatible with the rest of your stack. • Python packages: Dependencies like torch, transformers, and diffusers.
182 Chapter 7: Production • Inference engine: The version of vLLM, SGLang, TensorRT-LLM, or any other inference engine used. • System packages: Linux packages like ffmpeg, especially common when working with audio, image, or video models. Like a hiker on a backpacking trip, you want to pack light. Images built for inference are often many gigabytes. For fast deployment and efficient operation, only include strictly necessary dependencies. Another best practice is pinning versions. Having a pinned dependency tree keeps the system runtime behavior consistent across different envi- ronments and enables repeated builds of the image with the same result. Specify exactly which version of each dependency should be included in the image. Figure 7.2: Requirements should be pinned to exact versions to prevent future changes from breaking inference containers. Tools like uv, poetry, or pip will fl ag any version incompatibilities and throw an error when building an image. With pinned versions, once an image is successfully built once, it will always resolve dependencies to the same versions and protect you against breaking changes. Breaking changes are particularly common when working with newly released models. When new versions of models like DeepSeek are announced, the entire inference ecosystem races to offer day zero sup- port. When building images for brand-new models, inference engineers often rely on overnight builds or other developer pre-releases for dependencies rather than stable releases. These early versions are more prone to bugs and often need to be rebuilt on stable releases in the days and weeks following the model drop.
7.1.2 NIMs 183 7.1.2 NIMs NVIDIA Inference Microservices (NIMs) are pre-built Docker containers for popular open models. Containers make inference service implementations portable. NVIDIA created two types of NIMs: • Multi-LLM NIM: A flexible container for running a family of models on a supported GPU architecture. • LLM-specific NIM: An engine optimized for a specific model on a spe- cific GPU configured for maximum performance. NIMs are available for various common combinations of model, GPU archi- tecture, GPU count, and configuration. A NIM is like any other container. You can use a NIM as a starting point to build on, as a reference architecture to learn from, or as an out-of-the- box inference service. However, if you’re looking for maximum control instead of a done-for-you configuration, you’ll generally be better off building your own container from a less opinionated base image rather than adapting a NIM. 7.2 Autoscaling The goal of autoscaling is to ensure that you always have enough resources to serve all incoming requests while maintaining your latency SLAs without wasting money on idle GPUs.
184 Chapter 7: Production Figure 7.3: Without autoscaling, inference systems waste resources during traffic lulls and miss SLAs during traffic spikes. Figure 7.4: A strong autoscaling system for inference matches resources to demand. Autoscaling systems use Kubernetes, an open-source container orches- tration system, along with a cluster-level system for provisioning and deal- locating compute. Kubernetes can run one or more replicas of a model container, each on its own instance. An instance includes the GPUs and other hardware resources that the container requires.
7.2 Autoscaling 185 Kubernetes works by composing a group of hardware resources together into a cluster. This cluster has two types of components: • Control plane: Makes routing and scaling decisions. • Worker plane: Runs the actual containerized applications. Figure 7.5: Kubernetes clusters have a single control plane that orchestrates multiple workers. A Kubernetes cluster can run multiple replicas of multiple models. But how do you decide how many replicas of each model to run? Unless your traffic is unusually consistent, there probably isn’t one number of replicas that perfectly matches your needs. Autoscaling is the practice of dynamically adjusting the number of replicas allocated to a given model within a cluster. There are two ways to make autoscaling decisions: • Utilization: Scale up and down based on GPU utilization signals like memory usage or compute usage. • Traffic: Scale up and down based on the number of requests being processed in the system. Utilization and traffic don’t always match. For example, in LLM prefi ll, a few requests with hundreds of thousands of uncached input tokens could cause much higher utilization than many small requests with high cache hit rates.
186 Chapter 7: Production Traffic-based scaling decisions can be made proactively, while utilization is a lagging indicator. Use both in combination to keep system resources matched with demand. When designing a traffic-based autoscaling system, you want to configure five factors: • Min replicas: What is the minimum number of replicas that stay run- ning, regardless of traffic? • Max replicas: What is the maximum number of replicas that you can allocate when traffic is high? • Autoscaling window: How long is the sliding timeframe that you use to measure traffic and make autoscaling decisions? • Scale down delay: How long after a scale down is suggested do you wait in case of another traffic spike? • Concurrency target: How many requests can each replica handle at once? The exact configuration determines how well the autoscaling system achieves its goals of maintaining latency SLAs without wasting resources. For example, increasing the scale down delay prevents premature scale- downs for spikey traffic, but could result in unnecessary spend after traffic truly cools off. 7.2.1 Concurrency and Batch Sizing To properly operate a traffic-based autoscaling system, you need a strong understanding of how much concurrent traffic each instance can handle. Most model inference services can handle more than one request at a time via batching. There are several approaches to batching: • Static batching: The server waits until the batch is full before starting inference. • Dynamic batching: The server waits until the batch is full or a config- ured amount of time has passed before starting inference. • Continuous batching: The server continuously runs inference, swap- ping in requests as slots become available.
7.2.1 Concurrency and Batch Sizing 187 Inference engines like vLLM, SGLang, and TensorRT-LLM implement robust continuous batching (or in-fl ight batching as TensorRT-LLM calls it), where requests are batched at the token level. This minimizes latency relative to static batching. Figure 7.6: Static batching sets a fi xed batch size and waits for the batch to fi ll before beginning inference, leading to long wait times for early requests. Figure 7.7: Dynamic batching adds a cutoff time after which a batch is run whether or not it is full.
188 Chapter 7: Production Figure 7.8: Continuous batching operates at the token level, switching in new requests as old requests fi nish. Batch sizing trades off latency for throughput. Increasing the batch size will produce more throughput overall, but each user’s latency will get worse. Test performance across multiple batch sizes to fi nd the right fi t for your model, instance, latency target, and budget. This is controlled at the autoscaling confi guration level via the concurrency target and at the replica level via the batch size, which should match. Once every active replica reaches its maximum concurrency, the autoscal- ing system knows to spin up more replicas. If enough replicas are kicking off half-full batches, it’s time to scale back down. 7.2.2 Cold Starts A cold start is the time it takes to spin up a new replica of a model. The overall performance of an autoscaling system depends on its cold start speeds. If you can’t spin up replicas fast, it’s hard to confi dently scale down, leading to over-provisioning.
7.2.2 Cold Starts 189 There are several factors that affect cold start times: • GPU procurement: How quickly can you add the necessary GPUs to your cluster and allocate them to the model? • Image loading: How quickly can you load the container image onto the newly procured instance? • Model loading: How quickly can you load the model weights into the container? • Engine startup: How quickly can you start your inference engine, including any compilation time? Each of these factors needs to be optimized separately. Figure 7.9: Each step in the cold start process adds to the overall timeline. Unless you have a pool of warm nodes that you’re fl exing between mod- els, GPU procurement speed is mostly a function of your cloud provider. Section 7.3.1 covers procuring GPUs, and node start time is one of the negotiable factors in a contract. However, engineers can do a lot on loading images and weights and starting containers and engines. Loading images and model weights is a function of how quickly you can write gigabytes, often hundreds of gigabytes, of data onto your instance. There are two ways to load images and weights faster: make them smaller or get more bandwidth. Including only necessary steps and dependencies makes images smaller and faster to build into containers, while quantizing model weights has the additional benefi t of making them faster to load during cold starts.
190 Chapter 7: Production For small models, the strategy used to be baking the weights into the image to simplify caching and loading. However, now that most models have dozens or hundreds of billions of parameters, the weights dwarf the image and are better loaded separately. Where you load weights from has a massive impact on the bandwidth. If you’re loading from a third party like Hugging Face, you’re limited by their egress speed. And storing your weights in an S3 bucket introduces network latency and data transfer costs. For loading multi-hundred-billion-parameter models, you need gigabytes per second of bandwidth. The best way to get this is by loading over net- work within a node from a source cached physically near the GPU instance within the same datacenter. Inference engines like vLLM and SGLang are fast to start up. But engines like TensorRT-LLM and optimized models with PyTorch have a compila- tion step that targets the specific hardware resources to build the model inference engine. These compilations often take several minutes. In these cases, caching built engines massively improves cold start times. Both TensorRT-LLM and PyTorch have image caching mechanisms that make this possible, though you’ll always need to load a cached engine into an instance with exactly the GPU type, CUDA version, and software dependencies as the environment that the engine was built in for it to run properly. 7.2.3 Routing, Load Balancing, and Queueing Once there are multiple replicas online, the system needs to make a decision about which requests to send to which replicas. There are two types of components that make these decisions: • Routers: A router works at the request level to determine the ideal place to send a given request. A router answers “where should this request go?” • Load balancers: A load balancer works at the system level to even out requests between multiple options. A load balancer answers “where could this request go?”
7.2.3 Routing, Load Balancing, and Queueing 191 In complex systems, there isn’t just one router and one load balancer. Routing occurs throughout the stack with load balancers injected at key points to keep system-wide performance stable. Overall, you want to split load equally across replicas. However, routing and load balancing are not as simple as saying, “Well, I have 3 replicas and 12 requests, so let’s put 4 requests on each replica.” Each request may have a different number of input tokens. If most requests have 100 input tokens, a request with 10,000 will unbalance a simple system. Some requests are also better handled by certain replicas. Examples include: • KV cache-aware routing: Direct a request to a replica that already has a matching prefix in its KV cache. • LoRA-aware routing: Direct a request to a replica that already has the desired LoRA fine-tuned weights in memory. Intelligent request routing uses information from the inference engine and any orchestrators like NVIDIA Dynamo to route requests based on sequence length, prefix, and LoRA needs. Load balancing and routing are not enough. When an autoscaling sys- tem receives more traffic than it can handle, it needs a way to hold onto requests as it scales up more resources or waits for existing resources to become available. A queue is the infrastructure primitive for handling this situation. A standard queue is a first-in, first-out system for excess requests, though you can do a more complex implementation like a priority queue to, for example, give paid users priority over free users in high-traffic scenarios. As new replicas come online, ensure that the queue sees them and requests don’t continue waiting for the existing replicas. Each new replica should immediately be assigned up to its concurrency limit in queued traffic once active.
192 Chapter 7: Production 7.2.4 Scale to Zero Advanced autoscaling systems implement a mechanism for scale to zero, where the system can scale down to zero active replicas if there is no traffic, then scale up automatically when traffic is received. Scale to zero relies on two prerequisites: • Fast cold starts: As users may be waiting live, cold starts must be as fast as possible. • Robust queueing: The system needs to be able to hold the incoming requests until a replica is live. Even with these capabilities, scale to zero is not a fit for all workloads. Scale to zero is great for development, when testing is bursty and latency for the first request is unimportant. And in production, scale to zero is useful for applications that only get traffic periodically, like an agent that’s only accessed during business hours in one country or an offline system designed for daily batch processing jobs. However, if you’re relying on scale to zero to keep costs low in a laten- cy-sensitive application that gets light, unscheduled traffic, it’s probably a sign that your AI application is not yet ready for dedicated infrastructure and should use pay-per-token APIs until greater scale is reached. 7.2.5 Independent Component Scaling AI applications are increasingly built on multi-model, multi-stage com- pound AI workloads where inference engineers need to coordinate multiple steps to fulfill a single request. These steps may have different hardware needs. A voice activity detector model needs a far less powerful GPU than the transcription model it’s chunking data for, while the LLM processing that transcript may need a full node with multiple GPUs. And the scaling parameters for each step in the pipeline also differ.
7.3 Multi-Cloud Capacity Management 193 Figure 7.10: Independent component scaling gives each model access to appropriate resources and individual scaling. For these pipelines, you need to decompose autoscaling decisions and scale each step individually to right-size resources per step and avoid both bottlenecks and overprovisioning. However, every model in the pipeline should run in the same cluster. If it takes 10 milliseconds to send a message within a cluster and 50 millisec- onds to send messages between clusters, that 40-millisecond difference across a 5-step pipeline would be 20 percent of a one-second latency SLA. 7.3 Multi-Cloud Capacity Management Autoscaling within a single cluster works up to a certain point. But high-vol- ume deployments serving a global user base need thousands of GPUs distributed around the world. It’s straightforward to build multi-cloud inference as a collection of siloed compute across different cloud providers. But in these setups, there’s no way to use inter-cloud compute fl uidly, and moving workloads across clouds is a tedious, error-prone process. True multi-cloud inference requires building a multi-region, multi-provider bin packing tool, which treats distinct pools of compute as fungible with
194 Chapter 7: Production each other. Like Kubernetes within a single cluster, multi-cloud capacity management must take a global view, enabling self-healing and global scheduling. Figure 7.11: A multi-cloud approach extends the idea of control and workload planes to a multi-cluster, multi-region system. Running true multi-cloud inference unlocks: • Capacity: Pool capacity from multiple providers for greater and more fl exible GPU access. • Redundancy: Split inference across providers for resiliency against outages. • Latency: Run inference close to your end users to reduce network latency overhead. • Compliance: Run inference in compliance with data sovereignty and other regulatory requirements. Scaling from one cluster in one cloud to many clusters in many clouds requires a new coordination layer. A multi-cloud architecture contains: • Control plane: Handles model deployment and global scaling deci- sions, receives real-time event streams.
7.3.1 GPU Procurement 195 • Workload planes: Handles direct inference traffic and in-cluster scaling decisions, reports utilization and demand. This separation of responsibilities ensures that individual workload planes can serve traffic independently. If something happens to the control plane or any given workload plane, other workloads should be unaffected. 7.3.1 GPU Procurement There are a good number of companies in the business of providing access to GPUs. The three major types are: • Hyperscalers: Large cloud providers like AWS or GCP. • Neoclouds: GPU-focused clouds like Coreweave or Nebius. • Resellers: Secondary markets like SF Compute Company. Players in this space vary in their capacity, availability, and reliability. You generally pay a premium for hyperscalers and across all providers there is a tradeoff between cost and factors like uptime SLAs, support, regional availability, instance configuration, and cluster sizes. The first challenge is securing capacity. It is often difficult to get your hands on the GPUs you need, especially the latest hardware. Large clusters are also hard to get, with relatively few players offering blocks of hundreds of nodes. Many cloud providers allocate the majority of in-demand GPUs to their largest customers on long-term reservations. You may need to work across multiple cloud providers to get the GPUs you need in the right regions. Cloud GPUs can be procured via three different mechanisms: • Reserved: Blocks of hundreds or thousands of GPUs are reserved for months or years at discounted rates. • On-demand: Individual instances are available as needed up to a given quota for a relatively high per-hour cost. • Spot: Discounted on-demand instances that can be pre-empted at an agreed-upon notice period, often minutes.
196 Chapter 7: Production Large-scale inference generally uses a blend of GPU sources, with a baseline of low-cost reserved instances and a mix of on-demand and spot for handling peaks in traffic. These GPUs are distributed across multiple clusters worldwide for proximity to end users. 7.3.2 Geo-Aware Load Balancing Successful AI applications have users all over the world. Just like an individual cluster has a load balancer to ensure that every GPU in the cluster receives the right amount of traffic, a multi-cluster system needs a global load balancer. You don’t want a user request sitting around in some queue when there is spare capacity elsewhere, but you also don’t want to make a habit of sending a request from Singapore to a server in San Francisco. As a rule of thumb, it takes five milliseconds for a request to pass through a time zone. So, sending data from New York to San Francisco takes fifteen milliseconds one way. Given how small latency budgets are, it’s important to run workloads as close to end users as possible. 7.3.3 Building for Reliability GPUs are infamous for their high failure rate in production. Every engineer who has done a large-scale training run knows that they need to account for the eventuality that hardware will fail. For example, in their Llama 3 paper, Grattafiori and colleagues revealed that while running 16,000 GPUs for a period of 54 days, the Llama team experienced 419 unexpected interruptions, primarily due to hardware fail- ure. This works out to approximately one failure per 50,000 GPU-hours. 50,000 hours might sound like a long time, but running a single node of eight GPUs for inference for an entire year is over 70,000 GPU-hours. Inference engineers should expect hardware failure.
7.3.3 Building for Reliability 197 Figure 7.12: Root cause of failures when training Llama 3, adapted from “The Llama 3 Herd of Models” (Grattafi ori et al., 2024). GPU health is a node-level concern. When a single GPU fails, other GPUs on the node often fail next or need to be taken offline for maintenance. Proactively noting failures, cordoning nodes, and cycling pods keeps indi- vidual clusters healthy. GPU failures aren’t the only thing that can bring down inference. Cloud pro- viders have scheduled maintenance and their own unscheduled downtime. Every layer of infrastructure must be reinforced to provide high reliability. Multi-cloud inference brings two new approaches to high reliability: • Active-active: A high-availability posture where multiple regions or clusters actively serve live traffic at the same time. If any plane fails, traffic seamlessly continues on the others. • Active-passive: A failover posture where a “hot standby” cluster or region is kept ready but idle. If the active plane fails, traffic is cut over to the passive plane.
198 Chapter 7: Production When individual clusters, regions, or cloud providers go down, seamlessly failing over to another workload plane keeps reliability high and latency low. 7.3.4 Security and Compliance Cloud infrastructure has been a hot topic for security and compliance departments for more than twenty years. For AI models to power mis- sion-critical applications, inference must be both secure and compliant. Security and compliance conversations generally center around three areas: • User data: Security and compliance departments want to ensure all data, including user inputs and model outputs, is protected. • Model weights: For companies with fine-tuned or proprietary models, the weights are an invaluable trade secret. • Infrastructure: GPUs themselves and access to intelligence are both targets for abuse. One of the easiest decisions you can make to improve security is to simply not store user inputs or model outputs. This may not be possible – you might have logging requirements or user agreements to retain usage data for future model training – but if you don’t need to retain user data, you can reduce your attack surface. Securing AI inference workloads and associated data is similar to secur- ing any other containerized workload. Data encryption, container secu- rity, network and access controls, and workload isolation, all validated by extensive third-party penetration testing, remain the gold standard. Increasingly, inference engineers need to support applications running in regulated industries and compliance-heavy regions. One place where multi-cloud infrastructure helps is that in order for your application to com- ply with a certification like SOC 2 Type II or a regulation like HIPAA, your providers generally must also be compliant. In this case, being able to move workloads to compliant providers is useful. Another benefit of multi-cluster infrastructure is running one model across multiple regions. Certain industries and countries have data residency
7.4 Testing and Deployment 199 requirements, where user data from their country cannot be processed on servers in a different country. For example, having one cluster in a provider near Toronto and another in a provider near New York lets you keep Canadian data in Canada and American data in the United States while providing minimal latency over- head to users across the geographic region. 7.4 Testing and Deployment In addition to any replica-level testing and benchmarking performed while configuring the inference engine, it’s important to test systems end-to-end before deployment. There are several strategies for testing inference: • Manual testing: Writing scripts (or clicking buttons) to send synthetic traffic to an inference service. • Load testing: Automatically sending a large volume of traffic to test a system’s ability to scale and maintain performance. • Shadow traffic: Copying live traffic to test deployments to measure performance under real-world conditions. Testing inference services is expensive. It takes engineering time to con- figure the tests and measure the results, and it takes GPUs to run infer- ence for the test traffic. To a degree, that’s just the cost of doing business, but think carefully about how to minimize testing expenses. For example, shadow traffic testing could start with copying a random sample of pro- duction traffic, followed by a shorter-duration load test. When testing, keep in mind that AI product usage generally fluctuates on daily and weekly cycles. Once you’re confident in the performance and stability of your updated inference system, it’s time to deploy to production.
200 Chapter 7: Production 7.4.1 Zero-Downtime Deployment Inference engineers use high-availability deployment strategies to avoid downtime. A traditional high-availability design is a blue-green deployment. In this setup, there are two identical environments: the original blue deployment and a new green deployment running the updated service. Once the green environment is ready, the full traffic load cuts over from the blue to the green environment, with the blue environment staying ready for rollback in case of issues. However, blue-green is not well suited for large scale inference workloads due to the same GPU capacity and cost issues that make large-scale test- ing difficult. If the blue deployment is using 100 GPUs, the green deploy- ment requires another 100 GPUs before traffic can cut over. Instead, inference engineers can get similar benefi ts with lower GPU over- head using canary deployments. Inspired by the canaries that were used to detect gas in coal mines, a canary deployment catches errors before they affect large numbers of users. Figure 7.13: Iteratively shifting traffic over to the new deployment prevents multiple issues during inference service updates. A canary deployment is a 4-step process:
- Build a new deployment of the inference service and get it ready to handle incoming requests.
7.4.2 Cost Estimation 201 2. Direct a small percentage of the incoming live traffic to the new service. 3. Monitor the new service and ensure it is handling traffic correctly. Revert if there are any issues. 4. Gradually increase traffic, while monitoring for issues, until the new deployment handles 100 percent of traffic. These canary deployments can be rolled out quickly, with just a few min- utes of traffic ramp, or ramped slowly to ensure stability at each stage. And with autoscaling, canary deployments don’t increase cost much at scale because reducing traffic to the production system causes it to scale down some replicas. With autoscaling, the new deployment will default to the minimum number of replicas when there is no traffic. Throughout the canary deployment process, ensure that the new deployment has enough active replicas to properly handle requests. Otherwise, users will see a latency spike as their requests are queued until autoscaling completes. 7.4.2 Cost Estimation Switching from consuming tokens from a public API to doing your own inference on dedicated GPUs requires changing how you think about cost. Cost on public APIs is simple: a price per million tokens times the number of tokens you use. There are a couple of variables – cache hits versus cache misses for input tokens, discounts for high-volume users – but cost remains a linear function of usage. One motivation for investing the time and effort in inference engineering is to take control of your unit economics and escape per-token pricing. But it’s a difficult mental transition. The blessing and curse of dedicated inference is that cost is now a function of many variables. This is good because it gives you control, but it makes estimation difficult. Factors that affect cost include: • Batch sizing: Is the deployment optimized for latency with low batch sizes or throughput with high batch sizes?
202 Chapter 7: Production • Traffic patterns: Is traffic consistently saturating active GPUs, or is capacity going spare? • Sequence lengths: How many input and output tokens do requests have both on average and in outlier cases? Given this complexity and the difference in cost between input and output tokens, it’s generally more productive to convert your token price into a total cost and compare that to dedicated instead of trying to reverse engi- neer a per-token price from what you pay for GPUs. Figure 7.14: An equation for estimating the total cost of using per-token APIs in a product. Figure 7.15: An equation for estimating the total cost of using dedicated deployments in a product. Cost estimates should use a long time horizon, ideally at least a week, to smooth out variations in usage.
7.4.3 Observability 203 The other factor to consider in dedicated deployments is the cost of engineering time spent building and maintaining inference systems. This investment, while justified in increased reliability, security, and control, should be added to the GPU costs to form a complete picture around total cost of ownership (TCO) for inference. 7.4.3 Observability Inference is mission-critical, so it must be monitored like any other mis- sion-critical component of an application, with alerting, logs, and observ- ability built at the right level of abstraction. The first question is what to monitor. Inference observability includes measuring: • Total volume: The number of requests that a model deployment is receiving. • Request and response sizes: The input and output sequence lengths for the requests being processed. • Response codes: The count of 2XX, 4XX, and 5XX response codes issued by the model server. • Latency: Metrics like time to first token, tokens per second, and end- to-end latency on a P50, P90, and P99 basis. • Replica count: The number of instances actively serving traffic, and the number of instances starting up, if any. • Utilization: The amount of utilization across CPU, host memory, GPU, and GPU memory. • Queue depth: For systems with asynchronous traffic, the number of requests enqueued and waiting to be processed. These metrics are interdependent. A spike in latency could come from request volume, but it could also come from long input sequences. Seeing these metrics together lets inference engineers understand not only what is happening but also why.
204 Chapter 7: Production When things go wrong, inference engineers need information to fi x issues. Logs, both server logs and audit logs showing changes to an inference service, deliver that information in real time. Observability cannot be siloed. When you build observability for inference, build it with deep integration into existing observability and alerting tooling – Grafana, Datadog, PagerDuty, Sentry – to put inference information in context with the rest of the application. 7.5 Client Code Inference engineering draws on many technologies, from CUDA to Kuber- netes. But there’s one critical area that’s often overlooked when optimizing for latency and building for scale: client code. There are two sides of a call to an inference service: • Client: The browser, agent, or application making a request to the inference engine. • Server: The inference service that handles the client request and returns the model results. The industry standard for client code is the OpenAI SDK, which supports a wide range of compatible providers in addition to OpenAI’s own models. Popular AI engineering frameworks and libraries like LangChain, Vercel AI SDK, LiteLLM, LlamaIndex, and dozens more can also serve as clients. Whether you’re using an existing library or your own code, there is the potential for latency overhead or throughput bottlenecks. And for real-time applications, you may need a protocol other than HTTP, like WebSockets, to deliver a continuous connection. Figure 7.16: On-server inference time is just a fraction of the end-to-end latency for a given request.
7.5.1 Client Latency Overhead 205 7.5.1 Client Latency Overhead Depending on the client’s internet connection and the protocol used, establishing a session between a client and server takes a few dozen milliseconds. In a high-performance system with a 300-millisecond P95 end-to-end latency SLA, a TLS handshake costs at least ten percent of that latency budget before inference even starts. Future requests from the same client should save time by re-using existing sessions. Session re-use is not a new idea by any means, and tools like the OpenAI SDK provide it silently under the hood. However, when building your own client for non-standard modalities, follow best practices like session re-use. 7.5.2 Asynchronous inference Some systems are built for throughput, not latency. Use cases like bulk document processing and corpus embedding are not latency sensitive, so it makes sense to switch to asynchronous jobs. Asynchronous requests are a “fire and forget” approach to executing infer- ence. Ordinary synchronous inference requests have a timeout, generally of a few minutes, after which the request will fail. Asynchronous jobs fix this by immediately acknowledging the request and later returning the result of the asynchronous job to a webhook supplied in the original request. Asynchronous jobs still have time limits, but these requests are usually measured in hours, not minutes. Along with strong server-side queuing, asynchronous requests make high-throughput, latency-insensitive sys- tems more robust and efficient. 7.5.3 Streaming and Protocol Support Streaming makes applications feel instant. For language models, stream- ing text output over HTTP is sufficient. But for other modalities, especially
206 Chapter 7: Production live voice and video, both input and output streams need to be able to carry more data. Figure 7.17: One-time HTTP requests and responses are a good fi t for use cases like text chat, but not for continuous streaming. The two most common bi-directional streaming client-server connection protocols are: • Websockets: For streaming use cases where strong schema enforce- ment is not required. • gRPC: For well-defi ned service-to-service communication. WebSockets are useful for transmitting unstructured and real-time data, like audio, where the server receiving the request can parse it and pro- cess it downstream. With WebSockets, a server can support up to a fi xed, developer-confi gurable number of clients; when that concurrency is reached, new connections cannot be established and must wait until either a slot is free or another replica scales up. Figure 7.18: WebSockets establish a continuous connection for unstructured data like audio streams.
7.5.3 Streaming and Protocol Support 207 Similar to WebSockets, gRPC enables bi-directional streaming support, but for structured data. Requests transmitted via gRPC must follow a predefi ned schema, which takes away the load of having to parse the input. This additional validation layer makes gRPC slightly slower than WebSockets. Figure 7.19: gRPC establishes a continuous connection for well-defi ned service-to- service communication.
208 Chapter 7: Production 7.6 Production Inference with Baseten This book contains everything I’ve learned about inference in four years of working at Baseten. Baseten is an AI infrastructure company founded in 2019. At Baseten, we are focused on highly performant and highly available inference for both open models and custom models. We also offer a platform for pre-training, post-training, and reinforcement learning. We run inference for the world’s fastest-growing startups and most inno- vative enterprises, including Cursor, World Labs, Notion, OpenEvidence, Clay, Abridge, Gamma, Ambience, Writer, and hundreds more. At Baseten, we focus on four essential pillars to deliver the fastest mis- sion-critical inference: • Performance: Consistent low latencies at scale powered by the Baseten Inference Stack. • Infrastructure: Reliable multi-cloud deployments, fast and granular autoscaling, and robust security. • Tooling: An intuitive and productive developer experience with logging, observability, and programmatic access. • Applied expertise: Hands-on-keyboard implementation and assistance from forward deployed engineers. We would be honored to provide fast, reliable inference for your AI-pow- ered products. Also, we are continuously hiring for all roles across engineering, sales, marketing, and operations. To learn more, visit https://baseten.com/ careers.