5. A serving engine
The model from Chapter 1 generates for one prompt at a time. A serving engine takes requests from many users at once and keeps them all moving. This chapter builds the core of such an engine, one technique per section.
| Section | You learn |
|---|---|
| 5.1 The life of a request | A batched model step for an engine, and a first engine that serves one request at a time. |
| 5.2 Continuous batching | A scheduler, and three batching policies compared on the same workload. |
| 5.3 A paged KV cache | Blocks and block tables, the idea behind MAX Serve’s paged KV cache, and the memory they save. |
| 5.4 Prefix caching | Sharing the cache blocks of a common prompt prefix between requests. |
| 5.5 Speculative decoding | Drafting tokens with SmolLM2-135M, verifying them with SmolLM2-360M, and when it pays off. |
Related handbook pages: batching, PagedAttention, prefix caching, and speculative decoding.