5. A serving engine

The model from Chapter 1 generates for one prompt at a time. A serving engine takes requests from many users at once and keeps them all moving. This chapter builds the core of such an engine, one technique per section.

SectionYou learn
5.1 The life of a requestA batched model step for an engine, and a first engine that serves one request at a time.
5.2 Continuous batchingA scheduler, and three batching policies compared on the same workload.
5.3 A paged KV cacheBlocks and block tables, the idea behind MAX Serve’s paged KV cache, and the memory they save.
5.4 Prefix cachingSharing the cache blocks of a common prompt prefix between requests.
5.5 Speculative decodingDrafting tokens with SmolLM2-135M, verifying them with SmolLM2-360M, and when it pays off.

Related handbook pages: batching, PagedAttention, prefix caching, and speculative decoding.