Build and serve an LLM with MAX

This book takes you from zero to serving your own language model with MAX, Modular’s framework for AI inference. You start by running a real model in three lines of Python. Then you rebuild that model from scratch, check every part against MAX’s own implementation, and go on to the techniques that production inference engines use: distributing a model across devices, writing your own Mojo kernels, testing and benchmarking, building a serving engine, bringing up newer models, and serving your own architecture.

Why this book

Most explanations of LLM inference stop at diagrams, and most inference engines are too large to read. This book sits between the two. You write every piece yourself, small enough to read in one sitting, and you run it on a real model so that each claim is a number you measured: the logits match the reference, the KV cache stops each new token from recomputing the whole text, batching raises throughput.

You do it with MAX’s own APIs, the same ones its production models use, so what you learn carries over directly:

  • max.pipelines: AutoPipeline, AutoModel, and AutoTokenizer for running models from Python.
  • max.tensor and max.functional: tensors and operations that run eagerly, like NumPy, and compile into fast graphs.
  • max.nn: modules and layers for building models.
  • Mojo custom ops: kernels you write yourself and call from Python.
  • max.graph: the graph API that compile builds on, for full control.
  • max serve, max generate, and max benchmark: the command-line tools for serving, testing, and measuring models.

Every section is a notebook with interactive figures you can hover, zoom, and slide through, and every notebook checks its own results.

The model

The book uses SmolLM2-135M-Instruct. It is a real chat model, small enough to run on a laptop CPU, and it uses the Llama architecture that most open models share: RMSNorm, rotary position embeddings, grouped-query attention, and a SwiGLU MLP. What you build for it applies to Llama, Qwen, Mistral, and their relatives, and Chapter 6 extends it to Qwen3, Gemma 3, and mixture-of-experts models.

Get started

Check your hardware first

Before you install anything, check that MAX supports your machine and GPU in the MAX system requirements. MAX runs on Linux and on macOS with Apple silicon. On NVIDIA, the notebooks need an Ampere-generation GPU or newer; on an older one, such as a T4 or P100, they run on the CPU instead.

You can run the notebooks in Colab or Kaggle with nothing installed, or on your own machine.

In Colab or Kaggle

Every section page starts with Open in Colab and Open in Kaggle buttons. The first cell of every notebook installs what the notebook needs, so run the cells in order from the top:

# The setup cell, shortened: it installs MAX and this book's package
# with uv, but only if they're missing.
if importlib.util.find_spec("max") is None:
    uv_install("max[all]")  # from https://whl.modular.com/nightly/simple/
if importlib.util.find_spec("minillm") is None:
    uv_install("git+https://github.com/modular/max-llm-book@main")

max[all] is the MAX package with everything the book uses: the Python APIs, the max and mojo commands, serving, and the Jupyter %%mojo magic. minillm is the package you build in the book, installed so that later sections can import what earlier ones wrote.

Choose the runtime before you run the first cell:

  • In Colab, the CPU runtime works for every notebook. For a GPU, choose an L4 or A100 runtime. MAX’s matrix multiplication needs an NVIDIA GPU from the Ampere generation or later, so on a T4 runtime the setup cell hides the GPU and the notebook runs on the CPU.
  • In Kaggle, turn on Internet in the notebook settings so that the setup cell can download packages. Kaggle’s GPUs, the T4 and the P100, are older than Ampere, so the notebooks run on the CPU there.

On your own machine

Install with pixi or uv, on a machine that meets the system requirements.

With pixi, which installs Python, MAX, and every other dependency into a project environment:

  1. Install pixi:

    curl -fsSL https://pixi.sh/install.sh | sh
    
  2. Clone the book:

    git clone https://github.com/modular/max-llm-book
    cd max-llm-book
    
  3. Open the notebooks in JupyterLab:

    pixi run notebook
    

With uv and Python 3.10 or later:

  1. Create and activate an environment:

    uv venv --python 3.12
    source .venv/bin/activate
    
  2. Install MAX from the nightly package index:

    uv pip install --prerelease allow --index-strategy unsafe-best-match \
      --extra-index-url https://whl.modular.com/nightly/simple/ "max[all]"
    
  3. Clone the book and install its package, with JupyterLab:

    git clone https://github.com/modular/max-llm-book
    cd max-llm-book
    uv pip install -e . jupyterlab
    
  4. Open the notebooks:

    jupyter lab notebooks
    

Check the installation, and whether MAX found a GPU:

max --version
python -c "from max.driver import accelerator_count; print('GPUs:', accelerator_count())"

The book uses the MAX nightly build, because some of the APIs it teaches are newer than the last stable release. Every notebook runs on a CPU and uses a GPU automatically when MAX finds a supported one.

About the numbers

Timings, speedups, and other measurements on these pages come from one run on an Apple-silicon Mac with a Metal GPU. Treat them as examples. Your numbers depend on your hardware and vary from run to run; the notebooks check results, such as matching logits, that hold on every machine.

Chapters

ChapterWhat you do
1. FoundationsRun and serve SmolLM2, then rebuild it from scratch: tensors, weights, layers, attention, the full model checked against MAX’s, sampling, and a KV cache.
2. Distributed tensorsShard tensors and the model across a device mesh, simulated on one machine, and check that the result doesn’t change.
3. Custom ops in MojoWrite a kernel in Mojo, call it from your model, and see the graph API that compile builds on.
4. Testing and benchmarkingTest layers against references, verify logits, and benchmark and profile models and kernels.
5. A serving engineBuild the core of an inference engine: continuous batching, a paged KV cache, prefix caching, and speculative decoding.
6. Bigger modelsExtend your model to Qwen3, Gemma 3, and mixture-of-experts.
7. Production servingServe your own architecture with max serve, benchmark it, use structured output and tools, and call state-of-the-art open models on Modular’s cloud.

Read the chapters in order the first time: each section builds on code from earlier ones, which the minillm package collects for you.

The handbook

The Modular handbook explains how LLM inference works in production: metrics, batching, KV cache management, quantization, and GPU kernels. This book is its hands-on companion. Each section links the handbook pages for the ideas it implements, so you can read the concept and then build it.

Prerequisites

You should be comfortable with Python and NumPy-style array code. Knowing what a neural network layer is helps; the book explains everything specific to transformers as it goes. Section 6.2 uses Gemma 3, whose weights require you to accept Google’s license on Hugging Face and log in with hf auth login.

1. Foundations

This chapter goes from running a model to rebuilding it. You run and serve SmolLM2-135M-Instruct with MAX, then rewrite it from scratch, one part per section, until your version produces the same logits as MAX’s and generates text with a KV cache.

SectionYou learn
1.1 Run and serve an LLMAutoPipeline, AutoModel, max generate, max serve, and the two numbers that describe speed.
1.2 Tensors and eager executionmax.tensor, max.functional, and max.nn: eager execution and compilation.
1.3 Tokens, embeddings, and weightsWhat a config and a checkpoint hold, and how to build the token embedding from them.
1.4 Building blocksRMSNorm, the SwiGLU MLP, and rotary position embeddings.
1.5 AttentionCausal grouped-query attention, first from basic operations and then with flash attention.
1.6 The full modelAssembling 30 layers and matching MAX’s logits.
1.7 Generation and samplingThe generation loop, temperature, top-k, and top-p.
1.8 The KV cacheCaching keys and values so that decoding stops redoing work.

Every section is a notebook that runs in a few minutes, on a GPU if you have one and on a CPU otherwise. Sections 1.2 to 1.8 add code to the minillm package as they go, so each section can start from the previous ones without rerunning them.

By the end, you have a working LLM inference stack that you wrote yourself, and you know the parts of MAX it is built from. Chapter 2 splits that model across several devices.

1.1 Run and serve an LLM

Open in Colab Open in Kaggle View on GitHub

In this section you run a real language model with MAX four ways, from three lines of Python to an OpenAI-compatible server:

  1. AutoPipeline: text in, text out.
  2. AutoTokenizer and AutoModel: the tokens and scores underneath.
  3. max generate: the same thing from the command line, with timings.
  4. max serve: an HTTP server that any OpenAI client can call.

The model is SmolLM2-135M-Instruct, a 135M-parameter chat model from Hugging Face. It is small enough to run anywhere, and its architecture (the Llama family) is the one most open models use. The rest of the book rebuilds it from scratch.

If you are new to LLM inference, read What is LLM inference? in the Modular handbook first.

Setup

MAX runs on a GPU if it finds one, otherwise on the CPU. Every call below asks for float32 numbers. MAX requires float32 on a CPU, since it runs bfloat16 weights only on GPUs; on a GPU, float32 keeps the results comparable with the rest of the book. The first run downloads the model (about 270 MB) and compiles it, which takes a minute; later runs reuse both caches.

import logging
import time

import numpy as np
import plotly.graph_objects as go
from max.pipelines import AutoPipeline
from openai import OpenAI
from transformers import AutoTokenizer as HFTokenizer

from minillm import viz

# MAX logs performance hints that don't matter at this model size.
logging.getLogger("max").setLevel(logging.ERROR)

MODEL = "HuggingFaceTB/SmolLM2-135M-Instruct"
FLOAT32 = dict(quantization_encoding="float32")

Generate text in three lines

AutoPipeline loads a model and its tokenizer and generates text, much like transformers.pipeline. top_k=1 makes generation greedy: it always picks the most likely next token, so the output is the same on every run.

pipe = AutoPipeline(MODEL, **FLOAT32)
completion = pipe(["The capital of France is"], max_new_tokens=16, top_k=1)
assert "Paris" in completion[0]
completion
['The capital of France is Paris. Paris is the largest city in France and the capital of the French department']

A plain string goes to the model as is, and the output starts with the prompt. An instruct model is trained on conversations, so a list of chat messages usually works better. The pipeline formats them with the model’s chat template and returns only the reply.

messages = [{"role": "user", "content": "What is the capital of France?"}]
pipe([messages], max_new_tokens=32, top_k=1)
['The capital of France is Paris.']

Look underneath: tokens and scores

A language model doesn’t read text. A tokenizer splits the text into tokens (whole words, pieces of words, or punctuation) and maps each to an integer ID. SmolLM2’s vocabulary has 49,152 tokens.

tokenizer = pipe.tokenizer
ids = tokenizer(["The capital of France is"])["input_ids"][0]
print(ids)
viz.tokens([tokenizer.decode([i]) for i in ids], ids)
The·capital·of·France·is
[504, 3575, 282, 4649, 314]

Each chip is one token; hover over it for its ID, and the dots mark spaces. Common words are single tokens, usually with their leading space. Rare words, numbers, and code split into pieces:

text = (
    "Tokenizers split unbelievably rare words, 3.14159, and snake_case_names."
)
pieces = tokenizer([text])["input_ids"][0]
viz.tokens([tokenizer.decode([i]) for i in pieces], pieces)
Tokenizers·split·unbelievably·rare·words,·3.14159,·and·snake_case_names.

A chat message is text and tokens too. The chat template wraps each message in special tokens and adds a default system prompt. MAX loads the model’s Hugging Face tokenizer, so the transformers library can show what the question from earlier becomes before the model sees it:

template = HFTokenizer.from_pretrained(MODEL).apply_chat_template(
    messages, add_generation_prompt=True, tokenize=False
)
chat_ids = tokenizer([template])["input_ids"][0]
print(len(chat_ids), "tokens")
viz.tokens([tokenizer.decode([i]) for i in chat_ids], chat_ids)
<|im_start|>system↵You·are·a·helpful·AI·assistant·named·SmolLM,·trained·by·Hugging·Face<|im_end|>↵<|im_start|>user↵What·is·the·capital·of·France?<|im_end|>↵<|im_start|>assistant↵
37 tokens

Given the IDs, the model produces one score (a logit) for every token in the vocabulary: its prediction of what comes next. AutoModel.generate can return those scores. Softmax turns them into probabilities: it exponentiates each logit \(z_i\) and divides by the total, so every probability is positive and they sum to 1:

\[p_i = \frac{e^{z_i}}{\sum_j e^{z_j}}\]

model = pipe.model
out = model.generate(input_ids=[ids], max_new_tokens=1, output_logits=True)

logits = out.logits[0][0]  # first prompt, first new token: [vocab_size]
probs = np.exp(logits - logits.max())
probs /= probs.sum()
assert tokenizer.decode([int(probs.argmax())]) == " Paris"
Show the code 20 lines
top = np.argsort(probs)[::-1][:10]
fig = go.Figure(
    go.Bar(
        x=probs[top],
        y=[repr(tokenizer.decode([int(i)])) for i in top],
        orientation="h",
        marker_color=viz.BLUE,
        text=[f"{p:.1%}" for p in probs[top]],
        textposition="outside",
        customdata=top,
        hovertemplate="%{y} (ID %{customdata}): %{x:.2%}<extra></extra>",
    )
)
fig.update_layout(
    title="The ten likeliest tokens after 'The capital of France is'",
    height=420,
    xaxis=dict(title="probability", tickformat=".0%", range=[0, 1.1]),
    yaxis=dict(autorange="reversed", showgrid=False),
)
viz.show(fig)

Generation repeats this step: pick a token from the distribution, append it, and run the model again. Picking the top token is greedy decoding; sampling from the distribution gives more varied text. Section 1.7 implements both.

Measure it

Two numbers describe how fast a model feels:

  • Time to first token (TTFT) is how long the user waits before anything appears. It covers prefill, which processes the whole prompt at once.
  • Time per output token (TPOT) is how long each later token takes. Each one is a decode step that processes a single new token.

Generating one token measures prefill; generating 65 adds 64 decode steps. Each timing below is the median of three runs:

def seconds(prompt: list[int], n_tokens: int) -> float:
    start = time.perf_counter()
    model.generate(
        input_ids=[prompt], max_new_tokens=n_tokens, top_k=1, ignore_eos=True
    )
    return time.perf_counter() - start


def ttft_tpot(prompt: list[int], repeats: int = 3) -> tuple[float, float]:
    """Returns the median TTFT and TPOT of ``prompt``, in seconds."""
    ttft = float(np.median([seconds(prompt, 1) for _ in range(repeats)]))
    total = float(np.median([seconds(prompt, 65) for _ in range(repeats)]))
    return ttft, (total - ttft) / 64


ttft, tpot = ttft_tpot(ids)
print(
    f"TTFT {ttft * 1000:.0f} ms, TPOT {tpot * 1000:.1f} ms ({1 / tpot:.0f} tokens/s)"
)
TTFT 11 ms, TPOT 9.2 ms (108 tokens/s)

Prefill processes every prompt token, so TTFT should grow with the prompt. A decode step processes one token, so TPOT should barely change. Check both with prompts of random token IDs from 8 to 2,048 tokens long:

rng = np.random.default_rng(0)
prompt_lengths = [8, 64, 256, 1024, 2048]
ttfts, tpots = [], []
for length in prompt_lengths:
    prompt = rng.integers(0, len(probs), length).tolist()
    seconds(prompt, 2)  # warm up this length
    ttft_s, tpot_s = ttft_tpot(prompt)
    ttfts.append(ttft_s * 1000)
    tpots.append(tpot_s * 1000)
    print(
        f"{length:5} prompt tokens: TTFT {ttfts[-1]:6.1f} ms, "
        f"TPOT {tpots[-1]:5.1f} ms"
    )
    8 prompt tokens: TTFT   11.0 ms, TPOT   9.3 ms
   64 prompt tokens: TTFT   14.8 ms, TPOT   9.2 ms
  256 prompt tokens: TTFT   73.6 ms, TPOT   9.4 ms
 1024 prompt tokens: TTFT  670.0 ms, TPOT   9.4 ms
 2048 prompt tokens: TTFT 2470.4 ms, TPOT   9.1 ms
Show the code 23 lines
fig = go.Figure(
    [
        go.Scatter(
            x=prompt_lengths,
            y=ms,
            name=name,
            mode="lines+markers",
            line_color=color,
            hovertemplate=name + ", %{x} tokens: %{y:.1f} ms<extra></extra>",
        )
        for name, ms, color in [
            ("time to first token", ttfts, viz.ORANGE),
            ("time per output token", tpots, viz.BLUE),
        ]
    ]
)
fig.update_layout(
    title="Prefill grows with the prompt; a decode step barely does",
    height=420,
    xaxis=dict(title="prompt length (tokens, log scale)", type="log", dtick=1),
    yaxis=dict(title="ms (log scale)", type="log", dtick=1),
)
viz.show(fig)

TTFT grows with the prompt, steeply past a few hundred tokens, while TPOT stays nearly flat. A decode step still attends over the whole sequence, but that work is small next to reading every weight once per step.

ignore_eos=True keeps generating past the end-of-sequence token, so every run does the same amount of work. LLM inference metrics in the handbook defines these and the other metrics engines report.

Close the pipeline to free its memory. max generate and max serve below load the model again.

pipe.close()

The command line: max generate

The max command, installed with MAX, runs the same pipeline from a shell. In a notebook, a line that starts with ! runs as a shell command. max generate applies the chat template to --prompt and prints timing statistics after the text.

!max generate --model HuggingFaceTB/SmolLM2-135M-Instruct --quantization-encoding float32 \
    --prompt "What is the capital of France?" --max-new-tokens 32 --top-k 1 2>/dev/null
The capital of France is Paris.
Prompt size: 37
Output size: 8
Startup time: 0.32830238342285156 ms
Time to first token: 29.28781509399414 ms
Prompt eval throughput (context-encoding): 1263.3240097035216 tokens per second
Time per Output Token: 9.826013020106725 ms
Eval throughput (token-generation): 101.77067727813153 tokens per second
Total Latency: 98.06990623474121 ms
Total Throughput: 10.196807954586895 req/s

The lines to read are time to first token and time per output token: TTFT and TPOT again. Expect different values from the ones above, since this is one timed run of a different prompt. The prompt size is 37 tokens, not the 7 words typed, because the chat template adds the special tokens and system prompt shown earlier. 2>/dev/null hides the startup log. Run max generate --help to see every option.

Serve the model

max serve starts an HTTP server with an OpenAI-compatible API, so any OpenAI client library or tool can use your model. In a terminal you would run:

max serve --model HuggingFaceTB/SmolLM2-135M-Instruct \
  --quantization-encoding float32

From a notebook, start it in the background and wait until its /health endpoint answers:

Show start_max_serve 29 lines
from __future__ import annotations

import subprocess
import time
import urllib.request


def start_max_serve(
    model: str, *flags: str, port: int = 8000
) -> subprocess.Popen:
    """Starts ``max serve`` and returns once its health check answers.

    Call ``terminate()`` on the returned process to stop the server.
    """
    process = subprocess.Popen(
        ["max", "serve", "--model", model, "--port", str(port), *flags],
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
    )
    while True:
        try:
            urllib.request.urlopen(f"http://localhost:{port}/health")
            return process
        except OSError:
            if process.poll() is not None:
                raise RuntimeError(
                    "max serve exited; run it in a terminal to see why"
                ) from None
            time.sleep(1)

Saved to minillm/server.py.

start = time.perf_counter()
server = start_max_serve(MODEL, "--quantization-encoding", "float32")
print(f"server ready in {time.perf_counter() - start:.0f} s")
server ready in 17 s

Now call it with the official openai package. The API key is unused but required by the client.

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
reply = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "Name three primary colors."}],
    max_tokens=128,
    temperature=0,
)
print(reply.choices[0].message.content)
print(
    reply.usage.prompt_tokens,
    "prompt tokens,",
    reply.usage.completion_tokens,
    "completion tokens",
)
assert reply.choices[0].message.content.strip()
The primary colors are:

1. Red: A warm, reddish-brown color that is often associated with fire, blood, and fire.
2. Blue: A cool, deep, and deep blue color that is often associated with water, sky, and sky.
3. Yellow: A bright, golden yellow color that is often associated with earth, light, and light.

These colors are often used to represent the primary elements of a painting or artwork, and they can be used to create a sense of depth and dimensionality.
35 prompt tokens, 112 completion tokens

Chat apps show the reply as it is generated. With stream=True the server sends each token as soon as the model produces it:

stream = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "Count from one to ten in words."}],
    max_tokens=48,
    temperature=0,
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)
1. One
2. Two
3. Three
4. Four
5. Five
6. Six
7. Seven
8. Eight
9. Nine
10. Ten
server.terminate()
_ = server.wait()

OpenAI-compatible API in the handbook covers the request format, and Inference parameters explains temperature, top_p, and the other sampling options.

What MAX did

Each call above ran the same pipeline:

The generation loop: text goes through the tokenizer to token IDs, the model turns them into logits, the sampler picks the next token, and the token is appended to the sequence before the next step.The generation loop: text goes through the tokenizer to token IDs, the model turns them into logits, the sampler picks the next token, and the token is appended to the sequence before the next step.

Inside, MAX also compiled the model into optimized kernels, kept a KV cache so each decode step processes one token instead of the whole sequence, and scheduled requests so the server can batch them. In the rest of the book you build each of these parts yourself:

WhereYou buildWith
1.2Tensors, eager execution, compilationmax.tensor, max.functional
1.3Tokens, embeddings, weight loadingmax.nn.Embedding, safetensors
1.4RMSNorm, rotary embeddings, the MLPmax.nn layers
1.5Causal grouped-query attentionmax.functional, flash attention
1.6The full model, checked against AutoModelmax.nn.Module, compile
1.7Generation and samplingyour model
1.8A KV cacheyour model
2The model split across several devicesmax.tensor.sharding
3Your own kernels in Mojo%%mojo, F.custom, max.graph
4 to 7Testing, a serving engine, bigger models, production serving

Exercises

  1. Ask the chat model a few questions of your own. Where does a 135M model go wrong?
  2. Pass temperature=0.8 instead of top_k=1 and run the same prompt several times. What changes?
  3. Load the model with AutoModel(MODEL, max_batch_size=4, **FLOAT32) and time four prompts at once (input_ids=[ids] * 4). How do TTFT and TPOT change compared with one prompt?

Next

Section 1.2 introduces the tensor library you build the model with: max.tensor, max.functional, and eager execution.

1.2 Tensors and eager execution

Open in Colab Open in Kaggle View on GitHub

Every model in this book is written with three MAX modules:

  • max.tensor: the Tensor type.
  • max.functional (imported as F): operations on tensors, such as F.softmax and F.silu.
  • max.nn: Module and layers such as Linear, built from the two above.

This section covers what you need from each: creating tensors, running operations, and moving data to and from NumPy. It also shows how MAX differs from NumPy. Operations run eagerly, and the same code also compiles into a fast graph.

import logging
import time

import numpy as np
import plotly.graph_objects as go
from max import functional as F
from max.driver import CPU, accelerator_count
from max.dtype import DType
from max.graph import TensorType
from max.nn import Linear, Module
from max.tensor import Tensor, default_device, default_dtype

from minillm import viz

logging.getLogger("max").setLevel(logging.ERROR)

Pick a device

MAX runs the same code on CPUs and on NVIDIA, AMD, and Apple silicon GPUs. max.tensor.defaults() returns the dtype and the device mesh MAX would choose on its own, and pick_device takes the mesh’s first device: a GPU if MAX finds one, otherwise the CPU. Every notebook in this book uses that device, so on Colab or Kaggle with a GPU runtime, or on a Mac, your code runs on the GPU without changes.

Code cells that start with # export: become part of this book’s minillm package, so later sections import them instead of repeating them. pick_device is the first:

from __future__ import annotations

from max.driver import Device
from max.tensor import defaults


def pick_device() -> Device:
    """Returns the device MAX uses by default: a GPU if there is one,
    otherwise the CPU."""
    _, mesh = defaults()
    return mesh.devices[0]

Saved to minillm/device.py.

DEVICE = pick_device()
print("GPUs found:", accelerator_count(), " using:", DEVICE)
GPUs found: 1  using: Device(type=gpu,id=0)

Create tensors

A Tensor has a shape, a dtype, and a device. You can build one from Python lists, from a NumPy array, or with a constructor such as Tensor.ones. The examples name the dtype, float32, for a reason the next cells show:

x = Tensor(
    [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]], dtype=DType.float32, device=DEVICE
)
print(x)
print(x.shape, x.dtype, x.device)
Tensor([1 2 3
 4 5 6], dtype=DType.float32, device=Device(type=gpu,id=0))
[Dim(2), Dim(3)] DType.float32 Device(type=gpu,id=0)
print(Tensor(np.arange(6, dtype=np.float32).reshape(2, 3), device=DEVICE))
print(Tensor.zeros([2, 2], dtype=DType.float32, device=DEVICE))
print(Tensor.arange(5, dtype=DType.float32, device=DEVICE))
Tensor([0 1 2
 3 4 5], dtype=DType.float32, device=Device(type=gpu,id=0))
Tensor([0 0
 0 0], dtype=DType.float32, device=Device(type=gpu,id=0))
Tensor([0 1 2 3 4], dtype=DType.float32, device=Device(type=gpu,id=0))

Without device= or dtype=, MAX also picks the dtype: bfloat16 on a GPU, a 16-bit format most models ship in, and float32 on a CPU. This book computes in float32 everywhere, so its results can be compared exactly with reference implementations; the exercises in Section 4.2 measure what bfloat16 changes. NumPy also has no bfloat16 type, so a GPU tensor left at the default dtype can’t be read with to_numpy().

print("default placement:", Tensor.ones([2]).device, Tensor.ones([2]).dtype)
default placement: Device(type=gpu,id=0) DType.bfloat16

To change the defaults for a block of code, use the default_device and default_dtype context managers. Later sections build models inside them.

with default_device(DEVICE), default_dtype(DType.float32):
    y = Tensor.ones([2, 2])
print(y.device, y.dtype)
Device(type=gpu,id=0) DType.float32

Run operations

Python operators work as in NumPy, including @ for matrix multiplication and .T to transpose the last two axes. max.functional holds everything else.

print(x @ x.T)
print(x.sum(axis=-1))
print(x[:, 1:])
print(x.reshape([3, 2]))
print(F.softmax(x, axis=-1))
Tensor([14 32
 32 77], dtype=DType.float32, device=Device(type=gpu,id=0))
Tensor([ 6
 15], dtype=DType.float32, device=Device(type=gpu,id=0))
Tensor([2 3
 5 6], dtype=DType.float32, device=Device(type=gpu,id=0))
Tensor([1 2
 3 4
 5 6], dtype=DType.float32, device=Device(type=gpu,id=0))
Tensor([0.09003  0.2447  0.6652
 0.09003  0.2447  0.6652], dtype=DType.float32, device=Device(type=gpu,id=0))

Results are tensors on the same device as their inputs. To read one as a NumPy array, call to_numpy(), which copies it from the GPU first if needed. For a tensor already in CPU memory, np.from_dlpack() shares the memory instead of copying:

print(x.to_numpy())
on_cpu = x.to(CPU())
print(np.from_dlpack(on_cpu).dtype)
[[1. 2. 3.]
 [4. 5. 6.]]
float32

Eager execution

Each operation above ran immediately, so you could print its result. MAX builds a small graph for each operation, compiles it into a kernel the first time it sees that operation and shape, and caches the kernel. Compare the first call with the later ones:

z = Tensor(np.random.rand(256, 256).astype(np.float32), device=DEVICE)
call_ms = []
for i in range(4):
    start = time.perf_counter()
    F.gelu(z @ z).to_numpy()
    call_ms.append((time.perf_counter() - start) * 1000)
    print(f"call {i}: {call_ms[-1]:.0f} ms")
call 0: 4380 ms
call 1: 6 ms
call 2: 5 ms
call 3: 5 ms
Show the code 14 lines
fig = go.Figure(
    go.Bar(
        x=[f"call {i}" for i in range(4)],
        y=call_ms,
        marker_color=[viz.ORANGE] + [viz.BLUE] * 3,
        hovertemplate="%{x}: %{y:.1f} ms<extra></extra>",
    )
)
fig.update_layout(
    title="F.gelu(z @ z): the first call compiles, later calls reuse the kernels",
    yaxis=dict(title="ms (log scale)", type="log", dtick=1),
    height=360,
)
viz.show(fig)

The first call pays for compilation. On this page’s machine, an Apple-silicon Mac with a Metal GPU, it took several seconds, and the later calls took a few milliseconds. Treat these numbers as an example: yours depend on your hardware and vary from run to run. The cache persists across sessions, and max warm-interpreter-cache compiles the common kernels ahead of time.

Defer work with F.lazy

Inside F.lazy(), operations record a graph without running it. The tensors stay unrealized until you read a value:

with F.lazy():
    u = F.silu(x) * 2
print(u)
print(u.to_numpy())
Tensor(
    '<unrealized>',
    shape=[Dim(2), Dim(3)],
    dtype=float32,
    device=Device(type=gpu,id=0)
)
[[ 1.4621172  3.523188   5.7154446]
 [ 7.85611    9.933072  11.970329 ]]

Later sections build models inside F.lazy(), so the random initial weights are never computed: real weights from the checkpoint replace them.

Fuse operations with @F.functional

A function decorated with @F.functional runs as one graph, so MAX can fuse its operations into fewer kernels:

@F.functional
def silu_times_two(t: Tensor) -> Tensor:
    return F.silu(t) * 2


print(silu_times_two(x))
Tensor([1.462 3.523 5.715
 7.856 9.933 11.97], dtype=DType.float32, device=Device(type=gpu,id=0))

Modules and compilation

A Module groups parameters (tensors) and a forward method. Here is the feed-forward block SmolLM2 uses, at its real size: hidden size 576, intermediate size 1536. Section 1.4 explains the design; for now it’s a realistic workload.

class MLP(Module):
    def __init__(self, hidden: int, intermediate: int) -> None:
        self.gate_proj = Linear(hidden, intermediate, bias=False)
        self.up_proj = Linear(hidden, intermediate, bias=False)
        self.down_proj = Linear(intermediate, hidden, bias=False)

    def forward(self, x: Tensor) -> Tensor:
        return self.down_proj(F.silu(self.gate_proj(x)) * self.up_proj(x))


with default_device(DEVICE), default_dtype(DType.float32):
    mlp = MLP(576, 1536)
print(mlp)
print([name for name, _ in mlp.parameters])
MLP(
    gate_proj=Linear(in_dim=Dim(576), out_dim=Dim(1536), bias=False),
    up_proj=Linear(in_dim=Dim(576), out_dim=Dim(1536), bias=False),
    down_proj=Linear(in_dim=Dim(1536), out_dim=Dim(576), bias=False)
)
['gate_proj.weight', 'up_proj.weight', 'down_proj.weight']

Calling the module runs it eagerly, one operation at a time. A single timing is noisy, so median_ms calls the function 20 times, after one untimed call that compiles its kernels, and reports the median:

def median_ms(fn, x: Tensor, repeats: int = 20) -> float:
    fn(x).to_numpy()  # warm up: the first call compiles its kernels
    times = []
    for _ in range(repeats):
        start = time.perf_counter()
        fn(x).to_numpy()
        times.append((time.perf_counter() - start) * 1000)
    return float(np.median(times))


inputs = Tensor(np.random.rand(1, 64, 576).astype(np.float32), device=DEVICE)
eager_out = mlp(inputs).to_numpy()
print(f"eager:    {median_ms(mlp, inputs):.1f} ms")
eager:    49.0 ms

compile traces forward once into a whole-model graph and optimizes it: it fuses operations, plans memory, and removes the per-operation overhead. TensorType declares the input. Named dimensions such as "batch" and "seq_len" may take any size at run time.

start = time.perf_counter()
compiled_mlp = mlp.compile(
    TensorType(DType.float32, ("batch", "seq_len", 576), device=DEVICE)
)
print(f"compiled in {time.perf_counter() - start:.1f} s")

compiled_out = compiled_mlp(inputs).to_numpy()
print(f"compiled: {median_ms(compiled_mlp, inputs):.1f} ms")
relative = np.abs(compiled_out - eager_out).max() / np.abs(eager_out).max()
print(f"max difference, relative to the largest output: {relative:.1e}")
assert relative < 1e-5
compiled in 9.1 s
compiled: 0.6 ms
max difference, relative to the largest output: 1.2e-06

How much faster depends on the input size. Each eager operation pays a fixed cost to dispatch, which dominates small inputs; the compiled graph pays it once. Time both over a range of sequence lengths:

lengths = [1, 16, 64, 256, 1024]
timings = {"eager": [], "compiled": []}
for length in lengths:
    x_len = Tensor(
        np.random.rand(1, length, 576).astype(np.float32), device=DEVICE
    )
    timings["eager"].append(median_ms(mlp, x_len))
    timings["compiled"].append(median_ms(compiled_mlp, x_len))
    print(
        f"{length:5} tokens: eager {timings['eager'][-1]:6.1f} ms, "
        f"compiled {timings['compiled'][-1]:6.1f} ms"
    )
    1 tokens: eager   50.9 ms, compiled    0.7 ms
   16 tokens: eager   31.0 ms, compiled    1.0 ms
   64 tokens: eager   44.6 ms, compiled    1.8 ms
  256 tokens: eager   54.5 ms, compiled    4.1 ms
 1024 tokens: eager   59.0 ms, compiled    9.7 ms
Show the code 20 lines
fig = go.Figure(
    [
        go.Scatter(
            x=lengths,
            y=ms,
            name=name,
            mode="lines+markers",
            line_color=color,
            hovertemplate=name + ", %{x} tokens: %{y:.2f} ms<extra></extra>",
        )
        for (name, ms), color in zip(timings.items(), [viz.ORANGE, viz.BLUE])
    ]
)
fig.update_layout(
    title="The MLP block, eager and compiled (median of 20 calls)",
    height=420,
    xaxis=dict(title="sequence length (tokens)", type="log", dtick=1),
    yaxis=dict(title="ms (log scale)", type="log", dtick=1),
)
viz.show(fig)

The compiled MLP gives the same numbers up to float32 rounding (the compiled graph may order additions differently) and runs many times faster. In the run shown here, on an Apple-silicon Mac with a Metal GPU, an eager call took about 20 to 50 ms at every length, and a compiled one took about 1 ms up to 64 tokens and a few milliseconds at the longest lengths. The rest of the book works the same way: write and debug a model eagerly, printing anything you like, then compile it to run it for real. MAX’s docs cover eager execution and tensor realization in more detail.

The handbook’s CPU vs GPU vs TPU and GPU architecture fundamentals pages explain why matrix multiplications dominate the cost and why GPUs speed them up.

Exercises

  1. Create a [4, 4] tensor, apply F.softmax along each axis, and check with NumPy that the rows (or columns) sum to 1.
  2. Time the eager and compiled MLP with seq_len 1, 64, and 512. Where is the speedup largest? Why might small inputs benefit most?
  3. Print a tensor created inside F.lazy() before and after calling .to_numpy() on it.

Next

Section 1.3 loads SmolLM2’s real weights and turns token IDs into vectors.

1.3 Tokens, embeddings, and weights

Open in Colab Open in Kaggle View on GitHub

From this section on, you rebuild SmolLM2-135M-Instruct piece by piece. A model on Hugging Face is two things: a configuration that gives its sizes, and a checkpoint that holds its trained weights. This section loads both, then builds the model’s first layer, the token embedding.

As in Section 1.2, code cells that start with # export: go into the minillm package.

import json
import logging
from pathlib import Path

import numpy as np
import plotly.graph_objects as go
from max.driver import CPU
from max.dtype import DType
from max.nn import Embedding
from max.pipelines import AutoTokenizer
from max.tensor import Tensor, default_device, default_dtype

from minillm import viz
from minillm.device import pick_device

logging.getLogger("max").setLevel(logging.ERROR)
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU

The configuration

config.json lists the architecture and its sizes. These numbers decide the shape of every weight:

from huggingface_hub import hf_hub_download

MODEL_ID = "HuggingFaceTB/SmolLM2-135M-Instruct"
raw = json.loads(Path(hf_hub_download(MODEL_ID, "config.json")).read_text())
{
    k: raw[k]
    for k in [
        "architectures",
        "vocab_size",
        "hidden_size",
        "intermediate_size",
        "num_hidden_layers",
        "num_attention_heads",
        "num_key_value_heads",
        "rope_theta",
        "rms_norm_eps",
        "tie_word_embeddings",
    ]
}
{'architectures': ['LlamaForCausalLM'],
 'vocab_size': 49152,
 'hidden_size': 576,
 'intermediate_size': 1536,
 'num_hidden_layers': 30,
 'num_attention_heads': 9,
 'num_key_value_heads': 3,
 'rope_theta': 100000,
 'rms_norm_eps': 1e-05,
 'tie_word_embeddings': True}

Read it as: 30 identical layers (num_hidden_layers), each working on vectors of 576 numbers (hidden_size). Attention uses 9 query heads but only 3 key/value heads, so each head has 576 / 9 = 64 dimensions; Section 1.5 explains why the counts differ. LlamaForCausalLM is the architecture the code must implement.

A small dataclass keeps these fields typed. The field names match Hugging Face’s, so you can compare against any Llama-family config.json.

Show LlamaConfig 44 lines
from __future__ import annotations

import json
from dataclasses import dataclass, fields
from pathlib import Path

from huggingface_hub import hf_hub_download

MODEL_ID = "HuggingFaceTB/SmolLM2-135M-Instruct"


@dataclass(frozen=True)
class LlamaConfig:
    """The sizes of a Llama-family model, as in its ``config.json``."""

    vocab_size: int = 49152
    hidden_size: int = 576
    intermediate_size: int = 1536
    num_hidden_layers: int = 30
    num_attention_heads: int = 9
    num_key_value_heads: int = 3
    rope_theta: float = 100000.0
    rms_norm_eps: float = 1e-5
    max_position_embeddings: int = 8192
    tie_word_embeddings: bool = True
    eos_token_id: int = 2
    head_dim: int = 0  # 0 means hidden_size // num_attention_heads

    def __post_init__(self) -> None:
        # Newer configs, such as Qwen3's, give head_dim explicitly.
        if not self.head_dim:
            object.__setattr__(
                self, "head_dim", self.hidden_size // self.num_attention_heads
            )

    @classmethod
    def from_pretrained(cls, model_id: str = MODEL_ID) -> LlamaConfig:
        """Reads the config of a Hugging Face model."""
        raw = json.loads(
            Path(hf_hub_download(model_id, "config.json")).read_text()
        )
        return cls(
            **{f.name: raw[f.name] for f in fields(cls) if f.name in raw}
        )

Saved to minillm/config.py.

config = LlamaConfig.from_pretrained()
print(config)
print("head_dim:", config.head_dim)
LlamaConfig(vocab_size=49152, hidden_size=576, intermediate_size=1536, num_hidden_layers=30, num_attention_heads=9, num_key_value_heads=3, rope_theta=100000, rms_norm_eps=1e-05, max_position_embeddings=8192, tie_word_embeddings=True, eos_token_id=2, head_dim=64)
head_dim: 64

The checkpoint

The weights live in model.safetensors. The format is simple enough to read by hand: 8 bytes giving the length of a JSON header, the header itself, then the raw bytes of every tensor. The header maps each tensor’s name to its dtype, shape, and byte range:

path = hf_hub_download(MODEL_ID, "model.safetensors")
with open(path, "rb") as f:
    header_len = int.from_bytes(f.read(8), "little")
    header = json.loads(f.read(header_len))
header.pop("__metadata__", None)
print(len(header), "tensors")
first = list(header)[:8]
width = max(len(name) for name in first)
for name in first:
    print(f"{name:{width}} {header[name]}")
272 tensors
model.embed_tokens.weight                      {'dtype': 'BF16', 'shape': [49152, 576], 'data_offsets': [0, 56623104]}
model.layers.0.input_layernorm.weight          {'dtype': 'BF16', 'shape': [576], 'data_offsets': [56623104, 56624256]}
model.layers.0.mlp.down_proj.weight            {'dtype': 'BF16', 'shape': [576, 1536], 'data_offsets': [56624256, 58393728]}
model.layers.0.mlp.gate_proj.weight            {'dtype': 'BF16', 'shape': [1536, 576], 'data_offsets': [58393728, 60163200]}
model.layers.0.mlp.up_proj.weight              {'dtype': 'BF16', 'shape': [1536, 576], 'data_offsets': [60163200, 61932672]}
model.layers.0.post_attention_layernorm.weight {'dtype': 'BF16', 'shape': [576], 'data_offsets': [61932672, 61933824]}
model.layers.0.self_attn.k_proj.weight         {'dtype': 'BF16', 'shape': [192, 576], 'data_offsets': [61933824, 62155008]}
model.layers.0.self_attn.o_proj.weight         {'dtype': 'BF16', 'shape': [576, 576], 'data_offsets': [62155008, 62818560]}

The names follow the module tree of Hugging Face’s LlamaForCausalLM: an embedding, then model.layers.0 through model.layers.29, each with attention projections (q_proj, k_proj, v_proj, o_proj), an MLP (gate_proj, up_proj, down_proj), and two norms. Count the parameters:

n_params = sum(int(np.prod(t["shape"])) for t in header.values())
print(f"{n_params / 1e6:.1f}M parameters")
print(
    f"bfloat16: {n_params * 2 / 2**20:.0f} MiB, float32: {n_params * 4 / 2**20:.0f} MiB"
)
print("lm_head.weight in checkpoint:", "lm_head.weight" in header)
134.5M parameters
bfloat16: 257 MiB, float32: 513 MiB
lm_head.weight in checkpoint: False

Where do the parameters live? Click a box to zoom in, and click the bar at the top to zoom back out:

Show part, layer_number, order 71 lines
def part(name: str) -> str:
    if "embed_tokens" in name:
        return "embedding"
    if "self_attn" in name:
        return "attention"
    if "mlp" in name:
        return "MLP"
    return "norms"


color_of = {
    "embedding": viz.ORANGE,
    "attention": viz.BLUE,
    "MLP": viz.AQUA,
    "norms": viz.VIOLET,
}
# id -> [parent, label, total, color]
nodes = {"SmolLM2-135M": ["", "SmolLM2-135M", 0, viz.NEUTRAL]}


def layer_number(name: str) -> int:
    return int(name.split(".")[2]) if ".layers." in name else -1


def order(name: str) -> int:
    """Sorts the embedding first, then the layers in order, then the norm."""
    if "embed_tokens" in name:
        return -1
    return layer_number(name) if ".layers." in name else 99


for name in sorted(header, key=order):
    count = int(np.prod(header[name]["shape"]))
    kind = part(name)
    layer = layer_number(name)
    path = (
        ["SmolLM2-135M"]
        + ([f"layer {layer}"] if layer >= 0 else [])
        + [kind, name]
    )
    for depth in range(1, len(path)):
        node = "/".join(path[: depth + 1])
        label = name.split(".")[-2] if depth == len(path) - 1 else path[depth]
        color = color_of[kind] if path[depth] in (kind, name) else viz.NEUTRAL
        nodes.setdefault(node, ["/".join(path[:depth]), label, 0, color])
        nodes[node][2] += count
    nodes["SmolLM2-135M"][2] += count

fig = go.Figure(
    go.Treemap(
        ids=list(nodes),
        parents=[n[0] for n in nodes.values()],
        labels=[n[1] for n in nodes.values()],
        values=[n[2] for n in nodes.values()],
        branchvalues="total",
        sort=False,
        maxdepth=3,
        marker=dict(
            colors=[n[3] for n in nodes.values()],
            line=dict(color=viz.SURFACE, width=2),
        ),
        hovertemplate="%{label}<br>%{value:,} parameters, %{percentRoot:.1%} of the model<extra></extra>",
        texttemplate="%{label}<br>%{value:.3s}",
    )
)
fig.update_layout(
    title="SmolLM2-135M's parameters",
    height=520,
    margin=dict(t=56, l=8, r=8, b=8),
)
viz.show(fig)
  • Memory is parameters times bytes per parameter. The checkpoint stores bfloat16 (2 bytes); this book converts to float32 (4 bytes), so its results can be compared exactly with reference implementations. The handbook’s GPU memory calculation applies the same arithmetic to large models, plus the KV cache you build in Section 1.8.
  • There is no lm_head.weight. With tie_word_embeddings, the output layer reuses the embedding matrix; Section 1.6 uses that.

The loader below maps the file into memory and returns NumPy arrays. NumPy has no bfloat16 type, but the conversion is exact and cheap: a bfloat16 value is the upper 16 bits of a float32, so shifting the bits left by 16 gives the same number as a float32. MAX modules accept NumPy arrays directly, so this is all the loading code the model needs.

Show read_safetensors, load_checkpoint 44 lines
from __future__ import annotations

import json

import numpy as np
from huggingface_hub import hf_hub_download

from minillm.config import MODEL_ID

_NUMPY_DTYPES = {
    "F32": np.float32,
    "F16": np.float16,
    "I64": np.int64,
    "I32": np.int32,
}


def read_safetensors(path: str) -> dict[str, np.ndarray]:
    """Reads a safetensors file, converting floating-point tensors to float32."""
    with open(path, "rb") as f:
        header_len = int.from_bytes(f.read(8), "little")
        header = json.loads(f.read(header_len))
    data = np.memmap(path, dtype=np.uint8, mode="r", offset=8 + header_len)
    tensors = {}
    for name, info in header.items():
        if name == "__metadata__":
            continue
        start, end = info["data_offsets"]
        raw = data[start:end]
        if info["dtype"] == "BF16":
            array = (raw.view(np.uint16).astype(np.uint32) << 16).view(
                np.float32
            )
        else:
            array = raw.view(_NUMPY_DTYPES[info["dtype"]])
            if array.dtype == np.float16:
                array = array.astype(np.float32)
        tensors[name] = array.reshape(info["shape"])
    return tensors


def load_checkpoint(model_id: str = MODEL_ID) -> dict[str, np.ndarray]:
    """Downloads a model's safetensors checkpoint and returns its tensors."""
    return read_safetensors(hf_hub_download(model_id, "model.safetensors"))

Saved to minillm/weights.py.

weights = load_checkpoint()
w = weights["model.embed_tokens.weight"]
print(w.shape, w.dtype)
(49152, 576) float32

Token embeddings

The model’s first layer turns each token ID into a vector of 576 numbers. It is a lookup table: row i of a [49152, 576] matrix is the vector for token i. max.nn.Embedding implements it. load_state_dict replaces its random weight with the checkpoint’s:

with default_device(DEVICE), default_dtype(DType.float32):
    embed = Embedding(config.vocab_size, dim=config.hidden_size)
embed.load_state_dict({"weight": weights["model.embed_tokens.weight"]})

tokenizer = AutoTokenizer(MODEL_ID)
ids = tokenizer([" Paris France banana"])["input_ids"][0]
vectors = embed(Tensor(np.array([ids]), device=DEVICE))
print(ids, [tokenizer.decode([i]) for i in ids])
print(vectors.shape)
[7042, 4649, 21285] [' Paris', ' France', ' banana']
[Dim(1), Dim(3), Dim(576)]

Check it: each output vector must equal its row of the weight matrix.

table = embed.weight.to_numpy()
v = vectors.to_numpy()[0]
assert np.array_equal(v, table[ids])
print("embedding rows match the checkpoint")
embedding rows match the checkpoint

Training places related tokens near each other. Cosine similarity, the cosine of the angle between two vectors, shows it even before any layer runs:

\[\cos\theta = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert}\]

It is 1 for vectors pointing the same way, 0 for unrelated (orthogonal) ones, and \(-1\) for opposite ones, whatever their lengths.

def cosine(a: np.ndarray, b: np.ndarray) -> float:
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))


print(f"Paris  ~ France: {cosine(v[0], v[1]):.2f}")
print(f"Paris  ~ banana: {cosine(v[0], v[2]):.2f}")
print(f"France ~ banana: {cosine(v[1], v[2]):.2f}")
Paris  ~ France: 0.59
Paris  ~ banana: 0.36
France ~ banana: 0.37

Three numbers don’t show much. Project the embeddings of a few dozen words from three groups onto the two directions along which they vary most (principal component analysis, done here with NumPy’s SVD). Hover over a point for its word:

groups = {
    "countries": "France Italy Germany Spain Japan China Brazil Canada Egypt India",
    "fruits": "apple banana orange grape mango lemon cherry peach pear melon",
    "numbers": "one two three four five six seven eight nine ten",
}
words, group_of, rows = [], [], []
for group, members in groups.items():
    for word in members.split():
        token = tokenizer([" " + word])["input_ids"][0]
        if len(token) == 1:  # keep words that are a single token
            words.append(word)
            group_of.append(group)
            rows.append(table[token[0]])
points = np.array(rows) - np.mean(rows, axis=0)
_, _, axes = np.linalg.svd(points, full_matrices=False)
xy = points @ axes[:2].T
Show the code 30 lines
fig = go.Figure()
for (group, color), symbol in zip(
    zip(groups, viz.SERIES), ["circle", "square", "diamond"]
):
    index = [i for i, g in enumerate(group_of) if g == group]
    fig.add_trace(
        go.Scatter(
            x=xy[index, 0],
            y=xy[index, 1],
            mode="markers+text",
            name=group,
            text=[words[i] for i in index],
            textposition="top center",
            textfont=dict(size=11),
            marker=dict(
                color=color,
                symbol=symbol,
                size=10,
                line=dict(color=viz.SURFACE, width=2),
            ),
            hovertemplate="%{text}<extra>" + group + "</extra>",
        )
    )
fig.update_layout(
    title="Word embeddings projected to two dimensions",
    height=520,
    xaxis=dict(title="first principal component", showticklabels=False),
    yaxis=dict(title="second principal component", showticklabels=False),
)
viz.show(fig)

The groups separate without the model ever being told what a country or a fruit is: words used in similar contexts during training end up with similar vectors.

The layers refine these vectors using context: what Paris means depends on the tokens around it.

Exercises

  1. Find the 5 tokens whose embeddings are most similar to " Paris". (Hint: normalize the table’s rows, then take one matrix-vector product.)
  2. The treemap counts the embedding once because the output layer reuses it. How many parameters, and how many MiB in float32, would SmolLM2 need with a separate lm_head?

Next

Section 1.4 builds the layers that run inside each of the 30 blocks: RMSNorm, rotary position embeddings, and the SwiGLU MLP.

1.4 Building blocks

Open in Colab Open in Kaggle View on GitHub

Each of SmolLM2’s 30 layers has the same structure:

x = x + attention(rms_norm(x))     # mix information across positions
x = x + mlp(rms_norm(x))           # transform each position on its own

This section builds the three pieces besides attention: RMSNorm, the SwiGLU MLP, and rotary position embeddings (RoPE), which attention uses to know where each token is. Each piece is a few lines with max.nn and max.functional, and each is checked against a NumPy version of its formula, using the real weights of layer 0.

Show the setup 20 lines
import logging

import numpy as np
import plotly.graph_objects as go
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.nn import Linear, Module, RMSNorm
from max.tensor import Tensor, default_device, default_dtype

from minillm import viz
from minillm.device import pick_device
from minillm.config import LlamaConfig
from minillm.weights import load_checkpoint

logging.getLogger("max").setLevel(logging.ERROR)
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU

config = LlamaConfig.from_pretrained()
weights = load_checkpoint()

A layer’s weights are the checkpoint entries under its name prefix. This helper selects them and strips the prefix, so a module can load them with load_state_dict:

def select(
    weights: dict[str, np.ndarray], prefix: str
) -> dict[str, np.ndarray]:
    """Returns the entries under ``prefix``, with the prefix removed."""
    return {
        name.removeprefix(prefix): value
        for name, value in weights.items()
        if name.startswith(prefix)
    }

Saved to minillm/weights.py.

list(select(weights, "model.layers.0.mlp."))
['down_proj.weight', 'gate_proj.weight', 'up_proj.weight']

A random input stands in for the hidden states that reach layer 0. to_np reads a tensor back, and compare reports the largest difference from the NumPy result, absolute and relative to the largest value:

rng = np.random.default_rng(0)
x_np = rng.standard_normal((1, 6, config.hidden_size)).astype(np.float32)
x = Tensor(x_np, device=DEVICE)


def to_np(t: Tensor) -> np.ndarray:
    return t.to_numpy()


def compare(actual: np.ndarray, expected: np.ndarray) -> None:
    diff = np.abs(actual - expected).max()
    relative = diff / np.abs(expected).max()
    print(f"max difference {diff:.1e} (relative {relative:.1e})")
    assert relative < 1e-4, "doesn't match the NumPy reference"

RMSNorm

Deep networks train better when each layer sees inputs of a stable scale. RMSNorm divides each vector by its root mean square, then multiplies by a learned weight per dimension:

\[\mathrm{RMSNorm}(x) = \frac{x}{\sqrt{\mathrm{mean}(x^2) + \epsilon}} \cdot w\]

Unlike LayerNorm, it doesn’t subtract the mean or add a bias, which saves work and in practice trains as well. max.nn.RMSNorm implements it:

with default_device(DEVICE), default_dtype(DType.float32):
    norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
norm.load_state_dict(select(weights, "model.layers.0.input_layernorm."))

w = to_np(norm.weight)
expected = (
    x_np / np.sqrt((x_np**2).mean(-1, keepdims=True) + config.rms_norm_eps) * w
)
compare(to_np(norm(x)), expected)
max difference 3.0e-08 (relative 7.7e-08)

The learned weight is far from the all-ones it starts as. In layer 0, most dimensions are scaled close to zero, a few stand out several times larger, and nearly half are negative, so the norm also flips their sign. Hover over a bar to read its value:

Show the code 16 lines
fig = go.Figure(
    go.Bar(
        x=np.arange(config.hidden_size),
        y=w,
        marker_color=viz.BLUE,
        hovertemplate="dimension %{x}: %{y:.3f}<extra></extra>",
    )
)
fig.update_layout(
    title="Layer 0 input RMSNorm weight, per dimension",
    height=340,
    bargap=0,
    xaxis_title="dimension",
    yaxis_title="weight",
)
viz.show(fig)

The SwiGLU MLP

The MLP transforms each position independently. SmolLM2 uses a gated variant: one projection, passed through SiLU, decides how much of a second projection to let through, and a third projection maps back to the hidden size:

\[\mathrm{MLP}(x) = W_{\text{down}}\,\big(\mathrm{SiLU}(W_{\text{gate}}\,x) \odot W_{\text{up}}\,x\big)\]

where \(\mathrm{SiLU}(z) = z \cdot \sigma(z)\). The attribute names match the checkpoint (gate_proj, up_proj, down_proj), so load_state_dict needs no renaming.

Show the code 23 lines
z = np.linspace(-6, 6, 241)
fig = go.Figure(
    [
        go.Scatter(
            x=z,
            y=np.maximum(z, 0),
            name="ReLU",
            line=dict(color=viz.ORANGE, dash="dot"),
        ),
        go.Scatter(
            x=z, y=z / (1 + np.exp(-z)), name="SiLU", line_color=viz.BLUE
        ),
    ]
)
fig.update_traces(hovertemplate="%{x:.2f} -> %{y:.3f}")
fig.update_layout(
    title="SiLU is a smooth ReLU that lets small negative values through",
    height=360,
    xaxis_title="z",
    yaxis_title="activation",
    hovermode="x unified",
)
viz.show(fig)
from __future__ import annotations

from max import functional as F
from max.driver import Device
from max.dtype import DType
from max.graph import Dim
from max.nn import Linear, Module
from max.tensor import Tensor
from max.tensor.sharding import DeviceMesh

from minillm.config import LlamaConfig


class MLP(Module):
    """The SwiGLU feed-forward network of a Llama layer."""

    def __init__(self, config: LlamaConfig) -> None:
        hidden, inner = config.hidden_size, config.intermediate_size
        self.gate_proj = Linear(hidden, inner, bias=False)
        self.up_proj = Linear(hidden, inner, bias=False)
        self.down_proj = Linear(inner, hidden, bias=False)

    def forward(self, x: Tensor) -> Tensor:
        return self.down_proj(F.silu(self.gate_proj(x)) * self.up_proj(x))

Saved to minillm/layers.py.

with F.lazy(), default_device(DEVICE), default_dtype(DType.float32):
    mlp = MLP(config)
mlp.load_state_dict(select(weights, "model.layers.0.mlp."))

gate, up, down = (
    to_np(getattr(mlp, n).weight) for n in ("gate_proj", "up_proj", "down_proj")
)
g = x_np @ gate.T
expected = (g / (1 + np.exp(-g)) * (x_np @ up.T)) @ down.T
compare(to_np(mlp(x)), expected)
max difference 1.0e-03 (relative 8.2e-07)

Linear stores its weight as [out_features, in_features], the same layout as PyTorch, so the NumPy version multiplies by the transpose. The differences, about one part in a million, come from summing thousands of float32 products in a different order than NumPy does.

Rotary position embeddings

Attention compares tokens with dot products, which by themselves ignore order: “dog bites man” and “man bites dog” contain the same tokens. The model needs to know positions.

RoPE encodes position by rotating query and key vectors. It splits each 64-dimension head vector into 32 pairs and rotates pair i at position p by the angle \(p \cdot \theta_i\), where \(\theta_i = \text{base}^{-2i/d}\). Early pairs rotate quickly and late pairs slowly, like the hands of a clock. SmolLM2’s base is rope_theta = 100000.

Following the Hugging Face layout, the two halves of the vector form the pairs: element i pairs with element i + 32. Rotating every pair at once is then two multiplications:

\[\mathrm{rope}(x) = x \cdot \cos + \mathrm{rotate\_half}(x) \cdot \sin, \qquad \mathrm{rotate\_half}([x_1, x_2]) = [-x_2, x_1]\]

start offsets the positions; Section 1.8 needs it for the KV cache. The tables name their dtype: a Tensor built without one takes the device’s default, which is bfloat16 on a GPU, and bfloat16 can’t even represent every integer position above 256 exactly. Lengths are typed int | Dim because a compiled model sees symbolic dimensions such as seq_len instead of numbers; Section 1.2 shows where they come from.

Show rope_cos_sin, rotate_half, apply_rope 33 lines
def rope_cos_sin(
    head_dim: int,
    theta: float,
    start: int | Dim | Tensor,
    length: int | Dim,
    device: Device | DeviceMesh,
) -> tuple[Tensor, Tensor]:
    """Returns the RoPE cos and sin tables, ``[length, head_dim]``, for
    positions ``start`` to ``start + length - 1``."""
    inv_freq = Tensor(
        [theta ** (-i / head_dim) for i in range(0, head_dim, 2)],
        dtype=DType.float32,
        device=device,
    )
    positions = (
        Tensor.arange(length, dtype=DType.float32, device=device) + start
    )
    angles = F.outer(positions, inv_freq)
    angles = F.concat([angles, angles], axis=-1)
    return F.cos(angles), F.sin(angles)


def rotate_half(x: Tensor) -> Tensor:
    half = x.shape[-1] // 2
    return F.concat([-x[..., half:], x[..., :half]], axis=-1)


def apply_rope(x: Tensor, cos: Tensor, sin: Tensor) -> Tensor:
    """Rotates ``x``, shaped ``[batch, seq, heads, head_dim]``."""
    seq, dim = cos.shape
    cos = cos.reshape([1, seq, 1, dim])
    sin = sin.reshape([1, seq, 1, dim])
    return x * cos + rotate_half(x) * sin

Saved to minillm/layers.py.

Each line is one pair’s rotation at a position; move the slider to advance the position. Pair 0 turns a full radian per token, while pair 31 has barely moved after 24 tokens:

positions = 25
cos_t, sin_t = rope_cos_sin(
    config.head_dim, config.rope_theta, 0, positions, DEVICE
)
# One angle per pair and position: [position, head_dim].
angles = np.arctan2(to_np(sin_t), to_np(cos_t))
Show the code 33 lines
pairs = [0, 2, 8, 31]
steps = []
for pos in range(positions):
    group = []
    for pair, color in zip(pairs, viz.SERIES):
        degrees = float(np.degrees(angles[pos, pair]))
        group.append(
            go.Scatterpolar(
                r=[0, 1],
                theta=[0, degrees],
                mode="lines+markers",
                name=f"pair {pair}",
                line=dict(color=color, width=3),
                marker=dict(size=[0, 10], color=color),
                hovertemplate=f"pair {pair} at position {pos}: {degrees:.0f} degrees<extra></extra>",
            )
        )
    steps.append(group)
fig = go.Figure(
    layout=dict(
        title="RoPE rotates each pair at its own speed",
        height=520,
        polar=dict(
            radialaxis=dict(visible=False, range=[0, 1.05]),
            angularaxis=dict(gridcolor=viz.GRID, linecolor=viz.GRID),
        ),
    )
)
viz.show(
    viz.with_slider(
        fig, steps, [str(p) for p in range(positions)], "position ", active=3
    )
)

The whole cos table shows every pair at once. Each row is a position and each column a dimension. The left columns, the fast pairs, cycle every few positions; toward the middle the stripes widen, and the last pairs stay near 1 over all 256 positions. The right half repeats the left half, because element i and element i + 32 form one pair and share its angle:

table_cos, _ = rope_cos_sin(config.head_dim, config.rope_theta, 0, 256, DEVICE)
table_cos = to_np(table_cos)
half = config.head_dim // 2
assert np.array_equal(table_cos[:, :half], table_cos[:, half:])
Show the code 18 lines
fig = go.Figure(
    go.Heatmap(
        z=table_cos,
        zmin=-1,
        zmax=1,
        colorscale=viz.DIVERGING,
        colorbar=dict(title="cos", thickness=12),
        hovertemplate="position %{y}, dimension %{x}: %{z:.2f}<extra></extra>",
    )
)
fig.add_vline(x=half - 0.5, line=dict(color=viz.INK_SECONDARY, dash="dot"))
fig.update_layout(
    title="The RoPE cos table: positions by dimensions",
    height=520,
    xaxis=dict(title="dimension", showgrid=False),
    yaxis=dict(title="position", autorange="reversed", showgrid=False),
)
viz.show(fig)

RoPE relies on two properties, and both are easy to check. First, a rotation doesn’t change a vector’s length:

q = Tensor(
    rng.standard_normal((1, 8, 1, config.head_dim)).astype(np.float32),
    device=DEVICE,
)
cos, sin = rope_cos_sin(config.head_dim, config.rope_theta, 0, 8, DEVICE)
rotated = to_np(apply_rope(q, cos, sin))
print(np.linalg.norm(to_np(q), axis=-1).ravel()[:4])
print(np.linalg.norm(rotated, axis=-1).ravel()[:4])
assert np.allclose(
    np.linalg.norm(to_np(q), axis=-1),
    np.linalg.norm(rotated, axis=-1),
    rtol=1e-5,
)
[6.897215  6.9866242 7.8358827 6.884518 ]
[6.897215  6.9866247 7.8358827 6.884518 ]

Second, the dot product of a rotated query and key depends only on how far apart they are. Here a query at position 7 and a key at position 3 score the same as at positions 107 and 103:

def score(q_pos: int, k_pos: int) -> float:
    qv = Tensor(np.ones((1, 1, 1, config.head_dim), np.float32), device=DEVICE)
    kv = Tensor(
        np.linspace(-1, 1, config.head_dim, dtype=np.float32).reshape(
            1, 1, 1, -1
        ),
        device=DEVICE,
    )
    q_rot = apply_rope(
        qv, *rope_cos_sin(config.head_dim, config.rope_theta, q_pos, 1, DEVICE)
    )
    k_rot = apply_rope(
        kv, *rope_cos_sin(config.head_dim, config.rope_theta, k_pos, 1, DEVICE)
    )
    return float((to_np(q_rot) * to_np(k_rot)).sum())


print(f"{score(7, 3):.4f}  {score(107, 103):.4f}  {score(7, 5):.4f}")
assert (
    abs(score(7, 3) - score(107, 103)) < 1e-3
    and abs(score(7, 3) - score(7, 5)) > 1e-2
)
10.2645  10.2645  7.8206

The first two scores match. The third, for positions two apart instead of four, differs. The sweep below fixes the key at position 10 or at 110 and moves the query up to 48 positions after it. The two curves lie on top of each other:

q_vec = np.ones(config.head_dim, np.float32)
k_vec = np.linspace(-1, 1, config.head_dim, dtype=np.float32)
cos_t, sin_t = (
    to_np(t)
    for t in rope_cos_sin(config.head_dim, config.rope_theta, 0, 160, DEVICE)
)


def rope_np(v: np.ndarray, pos: int) -> np.ndarray:
    half = len(v) // 2
    return v * cos_t[pos] + np.concatenate([-v[half:], v[:half]]) * sin_t[pos]


distances = np.arange(49)
sweeps = {
    k_pos: [
        rope_np(q_vec, k_pos + d) @ rope_np(k_vec, k_pos) for d in distances
    ]
    for k_pos in (10, 110)
}
Show the code 22 lines
fig = go.Figure(
    [
        go.Scatter(
            x=distances,
            y=sweeps[k_pos],
            name=f"key at position {k_pos}",
            line=dict(color=color, dash=dash, width=width),
            hovertemplate="distance %{x}: %{y:.3f}<extra></extra>",
        )
        for k_pos, color, dash, width in [
            (10, viz.BLUE, "solid", 5),
            (110, viz.ORANGE, "dash", 2),
        ]
    ]
)
fig.update_layout(
    title="Query-key score after RoPE depends only on the distance",
    height=380,
    xaxis_title="query position minus key position",
    yaxis_title="q · k",
)
viz.show(fig)

Because scores depend only on relative position, RoPE generalizes well, and most open models use it.

Exercises

  1. Replace RMSNorm with a version you write from F.rsqrt, .mean, and *, and check that it gives the same output.
  2. Redraw the cos table with rope_theta=10000, the original RoPE base. Which pairs change, and why might a long-context model choose a larger base?
  3. Run the RMSNorm check with bfloat16 instead of float32 (cast the input and the module’s weight). How large is the error, and why do real models compute norms in float32?

Next

Section 1.5 builds attention, the part of the layer that mixes information between positions, and uses RoPE to do it.

1.5 Attention

Open in Colab Open in Kaggle View on GitHub

Attention is how a position gathers information from earlier positions. In “The capital of France is”, predicting the next token requires the last position to look back at “capital” and “France”. This section builds causal grouped-query attention, the variant SmolLM2 and most current models use, first from basic operations and then with MAX’s flash attention kernel.

Show the setup 28 lines
import logging
import math
import time
import warnings

import numpy as np
import plotly.graph_objects as go
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.nn import Embedding, RMSNorm
from max.pipelines import AutoTokenizer
from max.tensor import Tensor, default_device, default_dtype

from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.layers import apply_rope, rope_cos_sin
from minillm.weights import load_checkpoint, select

logging.getLogger("max").setLevel(logging.ERROR)
# MAX runs some kernels, such as flash attention, through a compiled
# fallback when executing eagerly, and warns each time it does.
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU

config = LlamaConfig.from_pretrained()
weights = load_checkpoint()

The computation

Each position projects its hidden vector into three vectors: a query for what the position is looking for, a key for what it offers, and a value for what it passes on.

A position’s output is a weighted average of the values of all positions it may see, weighted by how well its query matches their keys:

\[\mathrm{attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{Q K^\top}{\sqrt{d}} + M\right) V\]

Dividing by \(\sqrt{d}\), with \(d\) the head dimension, keeps the scores in a range where softmax still has a useful gradient. \(M\) is the causal mask.

The model runs several heads in parallel, each with its own projections, so different heads can track different relationships. SmolLM2 has 9 heads of 64 dimensions.

The causal mask

A language model predicts each token from the tokens before it, so position i must not see positions after i. The mask adds \(-\infty\) to those scores, and softmax turns them into zero weight.

The mask takes separate query and key lengths, so it also works when new queries attend to cached keys, as with the KV cache in Section 1.8. Here the lengths are equal.

Show causal_mask 30 lines
from __future__ import annotations

import math

from max import functional as F
from max.driver import Device
from max.dtype import DType
from max.graph import Dim
from max.nn import Linear, Module
from max.nn.kernels import flash_attention
from max.tensor import Tensor
from max.tensor.sharding import DeviceMesh

from minillm.config import LlamaConfig
from minillm.layers import apply_rope, rope_cos_sin


def causal_mask(
    q_len: int | Dim, kv_len: int | Dim, device: Device | DeviceMesh
) -> Tensor:
    """Returns a ``[q_len, kv_len]`` additive mask for queries at the last
    ``q_len`` of ``kv_len`` positions: 0 where a query may attend, -inf
    where it may not. The lengths may be ints or symbolic dimensions."""
    blocked = F.broadcast_to(
        Tensor(float("-inf"), dtype=DType.float32, device=device),
        [q_len, kv_len],
    )
    return F.band_part(
        blocked, num_lower=None, num_upper=kv_len - q_len, exclude=True
    )

Saved to minillm/attention.py.

causal_mask(4, 4, DEVICE)
Tensor([   0 -inf -inf -inf
    0    0 -inf -inf
    0    0    0 -inf
    0    0    0    0], dtype=DType.float32, device=Device(type=gpu,id=0))

Grouped-query attention

SmolLM2 has 9 query heads but only 3 key/value heads: each key/value head is shared by a group of 3 query heads. This is grouped-query attention (GQA). It costs little quality, and it makes the keys and values the model must store per token 3 times smaller, which matters once you cache them (Section 1.8).

Grouped-query attention in SmolLM2: 9 query heads in three colored groups of three, and each group points at the one key/value head it shares, so 3 key/value heads serve all 9 query heads.Grouped-query attention in SmolLM2: 9 query heads in three colored groups of three, and each group points at the one key/value head it shares, so 3 key/value heads serve all 9 query heads.

To compute it with plain matrix products, repeat each key/value head 3 times with broadcast_to:

def repeat_kv(x: Tensor, n_rep: int) -> Tensor:
    """Repeats each head of ``[batch, seq, kv_heads, dim]`` ``n_rep`` times."""
    batch, seq, kv_heads, dim = x.shape
    x = x.reshape([batch, seq, kv_heads, 1, dim])
    x = F.broadcast_to(x, [batch, seq, kv_heads, n_rep, dim])
    return x.reshape([batch, seq, kv_heads * n_rep, dim])


def naive_attention(q: Tensor, k: Tensor, v: Tensor, mask: Tensor) -> Tensor:
    """Computes causal attention with basic operations.

    ``q`` is ``[batch, seq, heads, dim]``; ``k`` and ``v`` may have fewer
    heads, which are repeated to match.
    """
    k = repeat_kv(k, q.shape[2] // k.shape[2])
    v = repeat_kv(v, q.shape[2] // v.shape[2])
    q, k, v = (t.permute([0, 2, 1, 3]) for t in (q, k, v))  # heads first
    scores = (q @ k.transpose(-1, -2)) / math.sqrt(int(q.shape[-1])) + mask
    return (F.softmax(scores) @ v).permute([0, 2, 1, 3])

Saved to minillm/attention.py.

Look at real attention

Run layer 0’s projections on a real sentence: embed the tokens, normalize, project to queries and keys, apply RoPE, and look at the attention weights.

tokenizer = AutoTokenizer(MODEL_ID)
ids = tokenizer(
    ["The capital of France is Paris, and the capital of Italy is"]
)["input_ids"][0]
tokens = [tokenizer.decode([i]) for i in ids]
seq = len(ids)

with default_device(DEVICE), default_dtype(DType.float32):
    embed = Embedding(config.vocab_size, dim=config.hidden_size)
    norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
embed.load_state_dict({"weight": weights["model.embed_tokens.weight"]})
norm.load_state_dict(select(weights, "model.layers.0.input_layernorm."))

h = norm(embed(Tensor(np.array([ids]), device=DEVICE)))
wq = Tensor(weights["model.layers.0.self_attn.q_proj.weight"], device=DEVICE)
wk = Tensor(weights["model.layers.0.self_attn.k_proj.weight"], device=DEVICE)
q = (h @ wq.T).reshape([1, seq, config.num_attention_heads, config.head_dim])
k = (h @ wk.T).reshape([1, seq, config.num_key_value_heads, config.head_dim])
cos, sin = rope_cos_sin(config.head_dim, config.rope_theta, 0, seq, DEVICE)
q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
mask = causal_mask(seq, seq, DEVICE)
k_full = repeat_kv(k, 3).permute([0, 2, 1, 3])
scores = (q.permute([0, 2, 1, 3]) @ k_full.transpose(-1, -2)) / math.sqrt(
    config.head_dim
) + mask
probs = F.softmax(scores).to_numpy()[0]  # [heads, seq, seq]
Show the code 47 lines
# Number the labels so that they stay unique when a token repeats.
labels = [f"{i}: {t!r}" for i, t in enumerate(tokens)]
pairs = [[[labels[i], labels[j]] for j in range(seq)] for i in range(seq)]
visible = np.tri(seq, dtype=bool)  # masked cells are left blank, not drawn as 0
heads = [
    [
        go.Heatmap(
            z=np.where(visible, probs[head], np.nan),
            customdata=pairs,
            zmin=0,
            zmax=1,
            coloraxis="coloraxis",
            hovertemplate="query %{customdata[0]}<br>key %{customdata[1]}<br>weight %{z:.2f}<extra></extra>",
        )
    ]
    for head in range(config.num_attention_heads)
]
fig = go.Figure(
    layout=dict(
        title="Layer 0 attention weights: rows are queries, columns are the keys they attend to",
        height=640,
        coloraxis=dict(
            colorscale=viz.SEQUENTIAL,
            cmin=0,
            cmax=1,
            colorbar=dict(title="weight", thickness=12),
        ),
        xaxis=dict(
            tickvals=list(range(seq)),
            ticktext=tokens,
            tickangle=-45,
            showgrid=False,
        ),
        yaxis=dict(
            tickvals=list(range(seq)),
            ticktext=tokens,
            autorange="reversed",
            scaleanchor="x",
            showgrid=False,
        ),
    )
)
viz.show(
    viz.with_slider(
        fig, heads, [str(h) for h in range(config.num_attention_heads)], "head "
    )
)

Each row is a query position; each column is a key it attends to. Hover over a cell to read its weight, and move the slider to change heads. The upper triangle is empty: that is the causal mask. The heads differ. Heads 7 and 8 put the most weight on the first token, which models use as a resting place when nothing else is relevant. Others, such as heads 2 and 5, put more weight on the current token.

The attention module

The module projects to queries, keys, and values, applies RoPE, attends, and projects the heads back to the hidden size with o_proj. Splitting project from forward lets Section 1.8 reuse it with a cache.

It creates the RoPE tables and the mask on x.mesh, the set of devices that hold x. Here that is one device; in Chapter 2 the same code runs on several devices at once.

For the attention itself it uses flash_attention from max.nn.kernels, which takes grouped heads directly. Flash attention computes the same result as naive_attention but never builds the per-head [seq, seq] score matrices: it processes keys in blocks and keeps a running softmax. This module still passes a dense [seq, seq] mask, one for the whole batch rather than one per head; production kernels take a causal flag instead, so their memory grows with the sequence length rather than its square.

Show Attention 51 lines
class Attention(Module):
    """Causal grouped-query attention with rotary position embeddings."""

    def __init__(self, config: LlamaConfig) -> None:
        self.num_heads = config.num_attention_heads
        self.num_kv_heads = config.num_key_value_heads
        self.head_dim = config.head_dim
        self.rope_theta = config.rope_theta
        hidden = config.hidden_size
        self.q_proj = Linear(hidden, self.num_heads * self.head_dim, bias=False)
        self.k_proj = Linear(
            hidden, self.num_kv_heads * self.head_dim, bias=False
        )
        self.v_proj = Linear(
            hidden, self.num_kv_heads * self.head_dim, bias=False
        )
        self.o_proj = Linear(self.num_heads * self.head_dim, hidden, bias=False)

    def project(
        self, x: Tensor, start: int | Dim | Tensor
    ) -> tuple[Tensor, Tensor, Tensor]:
        """Returns RoPE-rotated queries and keys, and values, for ``x``,
        whose first position is ``start``."""
        batch, seq, _ = x.shape
        q = self.q_proj(x).reshape([batch, seq, self.num_heads, self.head_dim])
        k = self.k_proj(x).reshape(
            [batch, seq, self.num_kv_heads, self.head_dim]
        )
        v = self.v_proj(x).reshape(
            [batch, seq, self.num_kv_heads, self.head_dim]
        )
        cos, sin = rope_cos_sin(
            self.head_dim, self.rope_theta, start, seq, x.mesh
        )
        return apply_rope(q, cos, sin), apply_rope(k, cos, sin), v

    def attend(self, q: Tensor, k: Tensor, v: Tensor) -> Tensor:
        """Attends from ``q`` to ``k`` and ``v`` and projects the heads back."""
        batch, q_len = q.shape[0], q.shape[1]
        mask = causal_mask(q_len, k.shape[1], q.mesh)
        mask = F.broadcast_to(mask, [batch, q_len, k.shape[1]])
        out = flash_attention(q, k, v, 1 / math.sqrt(self.head_dim), mask)
        # Attention never mixes heads, so the output is placed like the
        # queries. Chapter 2 relies on this when heads span several devices.
        out = out.rebind_mapping(q.mapping)
        return self.o_proj(
            out.reshape([batch, q_len, self.num_heads * self.head_dim])
        )

    def forward(self, x: Tensor) -> Tensor:
        return self.attend(*self.project(x, 0))

Saved to minillm/attention.py.

Load layer 0’s attention and check the module against naive_attention on the same normalized hidden states:

with F.lazy(), default_device(DEVICE), default_dtype(DType.float32):
    attn = Attention(config)
attn.load_state_dict(select(weights, "model.layers.0.self_attn."))

q, k, v = attn.project(h, 0)
reference = attn.o_proj(
    naive_attention(q, k, v, causal_mask(seq, seq, DEVICE)).reshape(
        [1, seq, config.hidden_size]
    )
)
diff = np.abs(attn(h).to_numpy() - reference.to_numpy()).max()
print(f"max difference {diff:.1e}")
assert diff < 1e-4
max difference 3.0e-07

Now compare their speed on a longer input, 1024 positions:

long = Tensor(
    np.random.default_rng(0)
    .standard_normal((1, 1024, config.hidden_size))
    .astype(np.float32),
    device=DEVICE,
)
q, k, v = attn.project(long, 0)
long_mask = causal_mask(1024, 1024, DEVICE)


def timed(fn) -> float:
    fn().to_numpy()  # compile the kernels first
    start = time.perf_counter()
    fn().to_numpy()
    return (time.perf_counter() - start) * 1000


print(f"naive: {timed(lambda: naive_attention(q, k, v, long_mask)):.0f} ms")
print(f"flash: {timed(lambda: attn.attend(q, k, v)):.0f} ms  (includes o_proj)")
naive: 93 ms
flash: 41 ms  (includes o_proj)

In the run shown here, on an Apple-silicon Mac with a Metal GPU, flash attention took about 55 ms including o_proj, against about 80 ms for the naive version. Treat these numbers as an example: yours depend on your hardware and vary from run to run. The handbook’s FlashAttention page explains the algorithm and why it matters most on GPUs, where moving data between memory levels costs more than the arithmetic.

Exercises

  1. Find the head in the plot that puts the most weight on the previous token. Does the pattern hold on a different sentence?
  2. Change repeat_kv to use 1 key/value head for all 9 query heads (multi-query attention). What would it save, and what might it cost?
  3. Time both versions at 256, 1024, and 2048 positions. How does each grow with sequence length?

Next

Section 1.6 combines attention with the Section 1.4 building blocks into complete layers, stacks 30 of them, and checks the result against MAX’s own implementation of SmolLM2.

1.6 The full model

Open in Colab Open in Kaggle View on GitHub

You now have every part of SmolLM2. This section assembles them into a decoder layer, stacks 30 layers into the model, adds the output head, loads all 272 weights, and compiles the result. Then it checks that your model’s logits match MAX’s own implementation of SmolLM2, the one AutoModel loads.

Show the setup 27 lines
import logging
import time
import warnings

import numpy as np
import plotly.graph_objects as go
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.graph import TensorType
from max.nn import Embedding, Linear, Module, ModuleList, RMSNorm
from max.pipelines import AutoModel, AutoTokenizer
from max.tensor import Tensor, default_device, default_dtype

from minillm import viz
from minillm.attention import Attention
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.layers import MLP
from minillm.weights import load_checkpoint, select

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU

config = LlamaConfig.from_pretrained()
weights = load_checkpoint()

The decoder layer

A layer normalizes its input, runs attention, and adds the result back to the input; then it does the same with the MLP:

\[\begin{aligned} h &= x + \mathrm{Attention}(\mathrm{RMSNorm}(x)) \\ x’ &= h + \mathrm{MLP}(\mathrm{RMSNorm}(h)) \end{aligned}\]

Adding back (the residual connection) means each layer only refines the hidden state instead of replacing it, which helps a stack of 30 layers train well.

The attribute names match the checkpoint once more, so a layer loads model.layers.N.* directly.

Show DecoderLayer 27 lines
from __future__ import annotations

from max import functional as F
from max.driver import Device
from max.dtype import DType
from max.nn import Embedding, Linear, Module, ModuleList, RMSNorm
from max.tensor import Tensor, default_device, default_dtype
from numpy.typing import NDArray

from minillm.attention import Attention
from minillm.config import LlamaConfig
from minillm.layers import MLP


class DecoderLayer(Module):
    """One transformer layer: attention and an MLP, each with a residual."""

    def __init__(self, config: LlamaConfig) -> None:
        eps = config.rms_norm_eps
        self.input_layernorm = RMSNorm(config.hidden_size, eps=eps)
        self.self_attn = Attention(config)
        self.post_attention_layernorm = RMSNorm(config.hidden_size, eps=eps)
        self.mlp = MLP(config)

    def forward(self, x: Tensor) -> Tensor:
        x = x + self.self_attn(self.input_layernorm(x))
        return x + self.mlp(self.post_attention_layernorm(x))

Saved to minillm/model.py.

Run layer 0 on real token embeddings and compare the size of each token’s input with the size of the update the layer adds to it:

tokenizer = AutoTokenizer(MODEL_ID)
ids = tokenizer(["The capital of France is"])["input_ids"][0]

with F.lazy(), default_device(DEVICE), default_dtype(DType.float32):
    layer = DecoderLayer(config)
layer.load_state_dict(select(weights, "model.layers.0."))

x = Tensor(weights["model.embed_tokens.weight"][ids][None], device=DEVICE)
y = layer(x)
x_np, y_np = x.to_numpy(), y.to_numpy()
print("input norm per token: ", np.linalg.norm(x_np, axis=-1).round(2))
print("update norm per token:", np.linalg.norm(y_np - x_np, axis=-1).round(2))
input norm per token:  [[2.   2.54 2.91 2.28 2.6 ]]
update norm per token: [[47.39 51.99 31.03 68.12 35.3 ]]

The update is 10 to 30 times larger than the input, because embeddings start as small vectors. The residual sum still carries the original embedding forward. Later in this section, a plot follows this residual stream through all 30 layers.

The model

The model embeds the tokens, runs the 30 layers in order, and normalizes the result. The causal LM wrapper adds lm_head, a Linear from 576 dimensions to one logit per vocabulary entry. The module tree mirrors Hugging Face’s LlamaForCausalLM, so every checkpoint name is also a parameter name.

load_model builds the model inside F.lazy(), so the random initialization never runs, and then loads the weights. SmolLM2 ties lm_head to the embedding table, so the loader uses the same array for both.

Show LlamaModel, LlamaForCausalLM, load_model 44 lines
class LlamaModel(Module):
    """The embedding, the decoder layers, and the final norm."""

    def __init__(self, config: LlamaConfig) -> None:
        self.embed_tokens = Embedding(config.vocab_size, dim=config.hidden_size)
        self.layers = ModuleList(
            [DecoderLayer(config) for _ in range(config.num_hidden_layers)]
        )
        self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)

    def forward(self, input_ids: Tensor) -> Tensor:
        x = self.embed_tokens(input_ids)
        for layer in self.layers:
            x = layer(x)
        return self.norm(x)


class LlamaForCausalLM(Module):
    """A Llama model with its language-model head: token IDs in, logits out."""

    def __init__(self, config: LlamaConfig) -> None:
        self.model = LlamaModel(config)
        self.lm_head = Linear(config.hidden_size, config.vocab_size, bias=False)

    def forward(self, input_ids: Tensor) -> Tensor:
        return self.lm_head(self.model(input_ids))


def load_model(
    config: LlamaConfig,
    weights: dict[str, NDArray],
    device: Device,
    model_class: type[Module] = LlamaForCausalLM,
) -> Module:
    """Builds ``model_class`` on ``device`` and loads ``weights`` into it."""
    with F.lazy(), default_device(device), default_dtype(DType.float32):
        model = model_class(config)
    if config.tie_word_embeddings:
        weights = {
            **weights,
            "lm_head.weight": weights["model.embed_tokens.weight"],
        }
    model.load_state_dict(weights)
    return model

Saved to minillm/model.py.

Here is the whole model as a diagram, built from the config. The shaded block repeats 30 times, and the curved lines are the residual connections. Hover over a box for the shape of the tensor it produces:

Show the code 121 lines
H, V, I = config.hidden_size, config.vocab_size, config.intermediate_size
stages = [  # (label, kind, output shape)
    ("token IDs", "io", "[batch, seq]"),
    (f"Embedding: {V:,} × {H}", "embedding", f"[batch, seq, {H}]"),
    ("RMSNorm", "norm", f"[batch, seq, {H}]"),
    (
        f"Attention: {config.num_attention_heads} query heads, {config.num_key_value_heads} key/value heads, RoPE",
        "attention",
        f"[batch, seq, {H}]",
    ),
    ("+", "add", f"[batch, seq, {H}]"),
    ("RMSNorm", "norm", f"[batch, seq, {H}]"),
    (f"SwiGLU MLP: {H} → {I} → {H}", "mlp", f"[batch, seq, {H}]"),
    ("+", "add", f"[batch, seq, {H}]"),
    ("final RMSNorm", "norm", f"[batch, seq, {H}]"),
    (
        f"LM head: {H} → {V:,} (shares the embedding)",
        "embedding",
        f"[batch, seq, {V}]",
    ),
    ("logits", "io", f"[batch, seq, {V}]"),
]
colors = {
    "io": viz.NEUTRAL,
    "embedding": viz.ORANGE,
    "norm": viz.VIOLET,
    "attention": viz.BLUE,
    "mlp": viz.AQUA,
    "add": viz.SURFACE,
}
ys = [-i for i in range(len(stages))]

fig = go.Figure()
fig.add_shape(
    type="rect",
    x0=-2.6,
    x1=2.6,
    y0=ys[7] - 0.45,
    y1=ys[2] + 0.75,
    line=dict(color=viz.GRID),
    fillcolor=viz.NEUTRAL,
    layer="below",
)
fig.add_annotation(
    x=2.5,
    y=ys[2] + 0.55,
    text=f"× {config.num_hidden_layers} layers",
    showarrow=False,
    xanchor="right",
    font=dict(color=viz.INK_SECONDARY),
)
for (label, kind, _), y in zip(stages, ys):
    width = 0.25 if kind == "add" else 2.1
    fig.add_shape(
        type="rect",
        x0=-width,
        x1=width,
        y0=y - 0.3,
        y1=y + 0.3,
        fillcolor=colors[kind],
        line=dict(
            color=viz.INK_SECONDARY if kind in ("add", "io") else colors[kind]
        ),
        layer="above" if kind == "add" else "below",
    )
    fig.add_annotation(
        x=0,
        y=y,
        text=label,
        showarrow=False,
        font=dict(
            color=viz.INK if kind in ("io", "add") else viz.SURFACE, size=13
        ),
    )
for upper, lower in zip(ys, ys[1:]):  # the main path
    fig.add_annotation(
        x=0,
        y=lower + 0.3,
        ax=0,
        ay=upper - 0.3,
        xref="x",
        yref="y",
        axref="x",
        ayref="y",
        showarrow=True,
        arrowhead=2,
        arrowcolor=viz.INK_SECONDARY,
        text="",
    )
# The residual connections skip the norm and the sublayer.
for start, add in [(1, 4), (4, 7)]:
    y0, y1 = ys[start] - 0.5, ys[add]
    fig.add_trace(
        go.Scatter(
            x=[0, -2.35, -2.35, -0.25],
            y=[y0, y0, y1, y1],
            mode="lines",
            hoverinfo="skip",
            line=dict(color=viz.INK_SECONDARY, dash="dot"),
            showlegend=False,
        )
    )
fig.add_trace(
    go.Scatter(
        x=[0] * len(stages),
        y=ys,
        mode="markers",
        marker=dict(size=40, opacity=0),
        customdata=[[label, shape] for label, _, shape in stages],
        showlegend=False,
        hovertemplate="%{customdata[0]}<br>output: %{customdata[1]}<extra></extra>",
    )
)
fig.update_layout(
    title="SmolLM2-135M, from token IDs to logits",
    height=640,
    xaxis=dict(visible=False, range=[-2.8, 2.8]),
    yaxis=dict(visible=False, range=[ys[-1] - 0.6, 0.6]),
    margin=dict(l=8, r=8),
)
viz.show(fig)
model = load_model(config, weights, DEVICE)
n_params = sum(p.num_elements() for _, p in model.parameters)
print(
    f"{len(list(model.parameters))} parameter tensors, {n_params / 1e6:.1f}M values"
)
273 parameter tensors, 162.8M values

Your model has 273 parameter tensors and 162.8M values. That is 28M more than the checkpoint’s 134.5M because lm_head counts as its own 49152 × 576 parameter, loaded from the same array as the embedding table. load_state_dict is strict: a missing or misnamed weight raises an error instead of silently keeping a random value, so a successful load already proves the names line up.

Inside the forward pass

Before compiling, run the layers one at a time and keep each one’s output. The size of each token’s hidden state grows as layers add their updates:

x = model.model.embed_tokens(Tensor(np.array([ids]), device=DEVICE))
states = [x.to_numpy()[0]]
for layer in model.model.layers:
    x = layer(x)
    states.append(x.to_numpy()[0])
states = np.array(states)  # [layer + 1, token, hidden]
Show the code 21 lines
labels = [repr(tokenizer.decode([i])) for i in ids]
fig = go.Figure(
    [
        go.Scatter(
            x=np.arange(31),
            y=np.linalg.norm(states[:, t], axis=-1),
            name=label,
            line_color=viz.SERIES[t],
            mode="lines",
            hovertemplate=label + " after layer %{x}: %{y:.1f}<extra></extra>",
        )
        for t, label in enumerate(labels)
    ]
)
fig.update_layout(
    title="Hidden-state norm through the layers (0 is the embedding)",
    height=420,
    xaxis_title="layer",
    yaxis=dict(title="norm (log scale)", type="log", dtick=1),
)
viz.show(fig)

The first token behaves differently. Its state jumps to about 24,000 at layer 12, over a hundred times the other tokens’ (about 120 to 165), and stays there until it drops in the last two layers. Models use the first position as a resting place for attention (Section 1.5), and this large state is part of how they do it.

Every layer writes into the same stream, so you can ask what the model would predict if it stopped early: apply the final norm and lm_head to each layer’s output. For the hidden state \(h_\ell\) after layer \(\ell\), the logit lens reads

\[p_\ell = \mathrm{softmax}\big(E \, \mathrm{RMSNorm}(h_\ell)\big)\]

where \(E\) is the embedding table that lm_head shares. Apply it at every position and after every layer:

final_norm = weights["model.norm.weight"]
table = weights["model.embed_tokens.weight"]  # lm_head shares it
normed = (
    states
    / np.sqrt((states**2).mean(-1, keepdims=True) + config.rms_norm_eps)
    * final_norm
)
lens = normed @ table.T  # [layer + 1, token, vocab]
lens_probs = np.exp(lens - lens.max(-1, keepdims=True))
lens_probs /= lens_probs.sum(-1, keepdims=True)

positions = np.arange(len(ids))
final_top = lens[-1].argmax(-1)  # each position's prediction after layer 30
p_final = lens_probs[:, positions, final_top]  # [layer + 1, token]
paris = tokenizer([" Paris"])["input_ids"][0][0]
assert final_top[-1] == paris
Show the code 29 lines
top_tokens = [
    [repr(tokenizer.decode([int(i)])) for i in row] for row in lens.argmax(-1)
]
finals = [repr(tokenizer.decode([int(i)])) for i in final_top]
fig = go.Figure(
    go.Heatmap(
        z=p_final,
        x=[f"{t}: {label}" for t, label in enumerate(labels)],
        y=np.arange(31),
        text=top_tokens,
        texttemplate="%{text}",
        textfont=dict(size=10),
        customdata=[finals] * 31,
        zmin=0,
        zmax=1,
        colorscale=viz.SEQUENTIAL,
        colorbar=dict(title="P(final)", thickness=12),
        xgap=1,
        ygap=1,
        hovertemplate="after %{x}, layer %{y}<br>top token %{text}<br>P(%{customdata}) = %{z:.3f}<extra></extra>",
    )
)
fig.update_layout(
    title="Logit lens: the top next token at each position and layer",
    height=820,
    xaxis=dict(title="position: the token read last", showgrid=False),
    yaxis=dict(title="layer", dtick=2, showgrid=False),
)
viz.show(fig)

Each column is a position, labeled with the token read last; each cell shows the layer’s top guess for the next token, shaded by the probability of the position’s final prediction. Read the last column from the bottom up: the top token is filler, a period or a space and then “ the“ from layer 23, and “ Paris“ becomes the top token only after layer 30, the last. Hover to read its probability: near zero through layer 28, about 0.06 after layer 29 and about 0.47 after layer 30. The middle positions settle earlier: “ of“ after “ capital“ from layer 25, and “ the“ after “ of“ from layer 23. Their probability peaks a few layers before the end, and the last layers spread some of it to other tokens.

Compile

Running 30 layers eagerly works but executes hundreds of operations one at a time. Compiling turns the whole forward pass into one optimized graph. The input is a batch of token-ID sequences of any length:

start = time.perf_counter()
compiled = model.compile(
    TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)
print(f"compiled in {time.perf_counter() - start:.0f} s")

start = time.perf_counter()
logits = compiled(Tensor(np.array([ids]), device=DEVICE)).to_numpy()
print(
    f"forward pass: {(time.perf_counter() - start) * 1000:.0f} ms, logits {logits.shape}"
)
compiled in 13 s
forward pass: 115 ms, logits (1, 5, 49152)

Check against MAX’s SmolLM2

AutoModel loads SmolLM2 with MAX’s own implementation of the architecture that config.json names LlamaForCausalLM, and can return the logits for each token it generates. The first generated token’s logits must match your model’s logits at the last prompt position:

reference_model = AutoModel(MODEL_ID, quantization_encoding="float32")
reference = reference_model.generate(
    input_ids=[ids], max_new_tokens=1, output_logits=True
)
expected = reference.logits[0][0]
reference_model.close()

ours = logits[0, -1]
print(
    f"max difference {np.abs(ours - expected).max():.1e} (logits range up to {np.abs(expected).max():.0f})"
)
print("top token, yours:", repr(tokenizer.decode([int(ours.argmax())])))
print("top token, MAX's:", repr(tokenizer.decode([int(expected.argmax())])))
assert (
    ours.argmax() == expected.argmax() and np.abs(ours - expected).max() < 1e-2
)
max difference 4.0e-05 (logits range up to 21)
top token, yours: ' Paris'
top token, MAX's: ' Paris'

The check covers every one of the 49,152 logits, not only the top one. Count how far each of your logits is from MAX’s; the dashed line marks the tolerance the check allows, and every logit sits orders of magnitude below it:

Show the code 23 lines
# Exact matches go in the lowest bin, since a log scale has no zero.
log_error = np.log10(np.maximum(np.abs(ours - expected), 1e-9))
fig = go.Figure(
    go.Histogram(
        x=log_error,
        xbins=dict(start=-9, end=0, size=0.25),
        marker_color=viz.BLUE,
        hovertemplate="%{y:,} logits<extra></extra>",
    )
)
fig.add_vline(x=-2, line=dict(color=viz.ORANGE, dash="dash"))
fig.update_layout(
    title="Difference between your logits and MAX's, over the whole vocabulary",
    height=380,
    bargap=0.05,
    xaxis=dict(
        title="absolute difference",
        tickvals=list(range(-9, 1)),
        ticktext=[f"1e{p}" for p in range(-9, 1)],
    ),
    yaxis_title="logits",
)
viz.show(fig)

In the run shown here, on an Apple-silicon Mac with a Metal GPU, the largest difference between your logits and MAX’s was about 4e-5, and both models predict the same next token. Treat these numbers as an example: yours depend on your hardware and vary from run to run. An error in any layer, such as a wrong RoPE layout or a missing scale, would change the logits by whole units, so the match shows that every layer you wrote is correct.

The same code runs the larger SmolLM2 models by changing MODEL_ID. Other Llama-family models need small additions: Llama 3.1, for example, rescales RoPE’s frequencies (rope_scaling in its config), and Qwen2 adds biases to the query, key, and value projections. The handbook’s Choosing the right model discusses how size, architecture, and license trade off when you pick one.

Exercises

  1. Run the logit lens on a longer prompt of your own. At which layer does each position settle on its final prediction?
  2. Load HuggingFaceTB/SmolLM2-360M-Instruct with the same code. What changes in config, and how do compile and forward times scale?
  3. Break something on purpose, such as dropping rotate_half from apply_rope, and see how far the logits move.

Next

Section 1.7 turns logits into text: the generation loop, and the sampling strategies behind temperature, top_k, and top_p.

1.7 Generation and sampling

Open in Colab Open in Kaggle View on GitHub

Your model turns tokens into next-token logits. This section turns logits into text: the generation loop, and the sampling strategies behind the temperature, top_k, and top_p settings from Section 1.1. Everything runs on your own model from Section 1.6.

Show the setup 32 lines
import logging
import time
import warnings

import numpy as np
import plotly.graph_objects as go
from IPython.display import Markdown, display
from plotly.subplots import make_subplots
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.functional import random
from max.graph import TensorType
from max.pipelines import AutoModel, AutoTokenizer
from max.tensor import Tensor

from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.model import load_model
from minillm.weights import load_checkpoint

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU

config = LlamaConfig.from_pretrained()
tokenizer = AutoTokenizer(MODEL_ID)
model = load_model(config, load_checkpoint(), DEVICE)
compiled = model.compile(
    TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)

The generation loop needs one thing from the model: the logits for the next token, given the tokens so far.

def next_logits(ids: list[int]) -> Tensor:
    logits = compiled(Tensor(np.array([ids]), device=DEVICE))
    return logits[0, -1]  # [vocab_size]

Greedy decoding

The simplest strategy picks the most likely token every time:

prompt = tokenizer(["The capital of France is"])["input_ids"][0]
ids = list(prompt)
for _ in range(24):
    ids.append(int(F.argmax(next_logits(ids)).item()))
print(tokenizer.decode(ids))
The capital of France is Paris. Paris is the largest city in France and the capital of the French department of the Espace de Paris.

Greedy decoding is deterministic, so it is the one strategy you can compare token for token with MAX’s own pipeline. top_k=1 makes AutoModel greedy too:

reference = AutoModel(MODEL_ID, quantization_encoding="float32")
expected = reference.generate(
    input_ids=[prompt], max_new_tokens=24, top_k=1
).sequences[0]
reference.close()
print("same tokens as AutoModel:", list(expected[-24:]) == ids[-24:])
assert list(expected[-24:]) == ids[-24:]
same tokens as AutoModel: True

Greedy text is coherent but tends to repeat itself and to pick the blandest continuation. Sampling picks tokens at random, in proportion to their probability, to get more varied text.

Temperature

Softmax turns logits \(z\) into probabilities. Dividing the logits by a temperature \(T\) first reshapes the distribution:

\[p_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}\]

Below 1, the likely tokens get likelier; above 1, the distribution flattens toward uniform. As \(T \to 0\), sampling becomes greedy. Move the slider to see it on the twelve likeliest next tokens of a story:

story = tokenizer(["Once upon a time, there was a"])["input_ids"][0]
logits = next_logits(story)
top = np.argsort(logits.to_numpy())[::-1][:12]
labels = [repr(tokenizer.decode([int(i)])) for i in top]
temperatures = [0.1, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5, 2.0]
Show the code 26 lines
steps = []
for temperature in temperatures:
    probs = F.softmax(logits / temperature).to_numpy()
    steps.append(
        [
            go.Bar(
                x=labels,
                y=probs[top],
                marker_color=viz.BLUE,
                hovertemplate="%{x}: %{y:.3f}<extra></extra>",
            )
        ]
    )
fig = go.Figure(
    layout=dict(
        title="Next-token probabilities after 'Once upon a time, there was a'",
        yaxis=dict(title="probability", range=[0, 1]),
        height=460,
        bargap=0.25,
    )
)
viz.show(
    viz.with_slider(
        fig, steps, [str(t) for t in temperatures], "temperature ", active=4
    )
)

Sampling with MAX operations

To sample from a distribution, you could compute the softmax and its cumulative sum, then search it with a random number. The Gumbel-max trick does it with operations you already have: add Gumbel noise \(-\log(-\log u)\), with \(u\) uniform in \((0, 1)\), to the scaled logits, and take the argmax:

\[\text{token} = \arg\max_i \left( \frac{z_i}{T} - \log(-\log u_i) \right), \qquad u_i \sim \mathrm{Uniform}(0, 1)\]

The sampled tokens follow the softmax distribution exactly, without computing the softmax. A quick check against the probabilities:

def gumbel_argmax(scores: Tensor) -> Tensor:
    u = random.uniform(
        scores.shape,
        range=(1e-6, 1.0),
        dtype=scores.dtype,
        device=scores.device,
    )
    return F.argmax(scores - F.log(-F.log(u)))


toy = Tensor(np.array([2.0, 1.0, 0.5, 0.1, -1.0], np.float32), device=DEVICE)
counts = np.bincount(
    [int(gumbel_argmax(toy).item()) for _ in range(2000)], minlength=5
)
print("sampled:", (counts / 2000).round(3))
print("softmax:", F.softmax(toy).to_numpy().round(3))
assert np.abs(counts / 2000 - F.softmax(toy).to_numpy()).max() < 0.05
sampled: [0.546 0.209 0.138 0.079 0.028]
softmax: [0.559 0.205 0.125 0.084 0.028]

Top-k and top-p

Even at a sensible temperature, the long tail of unlikely tokens adds up, and one bad draw can derail the text. Two filters cut the tail before sampling:

  • Top-k keeps the \(k\) most likely tokens.
  • Top-p (nucleus sampling) keeps the smallest set of tokens whose probabilities add up to at least \(p\), so it keeps few tokens when the model is confident and many when it isn’t.

Both filters work on tokens sorted from most to least likely. F.top_k returns the \(k\) largest scores already sorted, with their token IDs. For top-p alone, F.argsort sorts the whole vocabulary and F.gather puts the scores in that order. F.cumsum over the sorted probabilities then gives the running total, and F.where drops every token that the total had already passed \(p\) before reaching. The filtered scores go through the Gumbel-max trick, and the ID map turns the winning position back into a token. Without a filter, nothing needs sorting.

Show sample, generate 57 lines
from __future__ import annotations

from collections.abc import Callable

from max import functional as F
from max.functional import random
from max.tensor import Tensor


def sample(
    logits: Tensor,
    temperature: float = 1.0,
    top_k: int | None = None,
    top_p: float | None = None,
) -> int:
    """Picks a token ID from 1-D ``logits``; ``temperature=0`` is greedy."""
    if temperature == 0:
        return int(F.argmax(logits).item())
    scores, token_ids = logits / temperature, None
    if top_k is not None:
        scores, token_ids = F.top_k(scores, top_k)  # sorted, largest first
    if top_p is not None:
        if token_ids is None:
            token_ids = F.argsort(scores, ascending=False)
            scores = F.gather(scores, token_ids, axis=0)
        probs = F.softmax(scores)
        before = F.cumsum(probs, axis=-1) - probs  # mass of likelier tokens
        scores = F.where(before < top_p, scores, float("-inf"))
    u = random.uniform(
        scores.shape,
        range=(1e-6, 1.0),
        dtype=scores.dtype,
        device=scores.device,
    )
    choice = int(F.argmax(scores - F.log(-F.log(u))).item())
    return choice if token_ids is None else int(token_ids.to_numpy()[choice])


def generate(
    next_logits: Callable[[list[int]], Tensor],
    prompt: list[int],
    max_new_tokens: int,
    eos_token_id: int | None = None,
    **sampling: float | None,
) -> list[int]:
    """Generates up to ``max_new_tokens`` tokens after ``prompt``.

    ``next_logits`` maps the tokens so far to the next token's logits.
    Generation stops early at ``eos_token_id``. Returns only the new tokens.
    """
    ids = list(prompt)
    for _ in range(max_new_tokens):
        token = sample(next_logits(ids), **sampling)
        ids.append(token)
        if token == eos_token_id:
            break
    return ids[len(prompt) :]

Saved to minillm/sampling.py.

Top-p adapts to the model’s confidence. After “The capital of France is”, two tokens hold about three quarters of the probability; after “Once upon a time, there was a”, it spreads over many. Move \(p\) and compare how many tokens each prompt keeps (blue) and cuts (gray). The bars show the likeliest 40; the count covers the whole vocabulary:

prompts = ["The capital of France is", "Once upon a time, there was a"]
shown = 40
sorted_probs = []
for text in prompts:
    probs = F.softmax(next_logits(tokenizer([text])["input_ids"][0])).to_numpy()
    sorted_probs.append(np.sort(probs)[::-1])
Show the code 58 lines
ps = [0.5, 0.75, 0.9, 0.95, 0.99]
rank = np.arange(1, shown + 1)
steps = []
for p in ps:
    group = []
    for axis, probs in zip(["", "2"], sorted_probs):
        before = np.cumsum(probs) - probs  # the same test as `sample`
        kept = before < p
        count = int(kept.sum())
        kept, top = kept[:shown], probs[:shown]
        for mask, color, name in [
            (kept, viz.BLUE, "kept"),
            (~kept, viz.MUTED, "cut"),
        ]:
            group.append(
                go.Bar(
                    x=rank[mask],
                    y=top[mask],
                    marker_color=color,
                    name=name,
                    showlegend=False,
                    xaxis=f"x{axis}",
                    yaxis=f"y{axis}",
                    hovertemplate="rank %{x}: %{y:.3f}<extra>"
                    + name
                    + "</extra>",
                )
            )
        group.append(
            go.Scatter(
                x=[shown],
                y=[0.9],
                mode="text",
                text=[f"{count:,} kept"],
                textposition="middle left",
                showlegend=False,
                hoverinfo="skip",
                xaxis=f"x{axis}",
                yaxis=f"y{axis}",
            )
        )
    steps.append(group)

fig = make_subplots(
    rows=2,
    cols=1,
    shared_xaxes=True,
    vertical_spacing=0.12,
    subplot_titles=[repr(text) for text in prompts],
)
fig.update_layout(
    height=560,
    bargap=0.15,
    title="The likeliest 40 next tokens, sorted, at temperature 1",
)
fig.update_yaxes(range=[0, 1], title="probability")
fig.update_xaxes(title="rank", row=2, col=1)
viz.show(viz.with_slider(fig, steps, [str(p) for p in ps], "top_p ", active=2))

generate stops at the model’s end-of-sequence token, which a chat model emits when its answer is complete. Try a few settings. random.set_seed makes a sampled run repeatable:

for settings in [
    dict(temperature=0),
    dict(temperature=0.7, top_p=0.9),
    dict(temperature=1.0, top_k=50),
    dict(temperature=1.5),
]:
    random.set_seed(0)
    new = generate(next_logits, story, 24, config.eos_token_id, **settings)
    print(f"{settings}\n  {tokenizer.decode(new)!r}\n")
{'temperature': 0}
  ' young woman named Sarah who was struggling to find her place in the world. She had always been a bit of a rebel'

{'temperature': 0.7, 'top_p': 0.9}
  ' young girl named Lily who was determined to be a better version of herself. She had always been the "perfect" girl'

{'temperature': 1.0, 'top_k': 50}
  ' young musician who had spent their life playing and composing. He was known as "The Weaver" due to his exceptional skill'

{'temperature': 1.5}
  ' neighboring realm on your planet, mist. Even then; Unlocking beastellandsis deathshire istsubbralgenaki'

Why does a high temperature drift? Color each generated token by the probability it had in the distribution it was drawn from (temperature 1 for the greedy run), and hover over a chip for the five likeliest alternatives:

def generate_with_probs(
    prompt: list[int], n: int, temperature: float
) -> tuple[list[int], list[np.ndarray]]:
    """Generates like `generate`, keeping each step's sampling distribution."""
    ids, dists = list(prompt), []
    for _ in range(n):
        logits = next_logits(ids)
        dists.append(F.softmax(logits / (temperature or 1.0)).to_numpy())
        token = sample(logits, temperature=temperature)
        ids.append(token)
        if token == config.eos_token_id:
            break
    return ids[len(prompt) :], dists


runs = {}
for temperature in [0, 1.5]:
    random.set_seed(0)
    runs[temperature] = generate_with_probs(story, 24, temperature)
Show chip_title 20 lines
def chip_title(token: int, dist: np.ndarray) -> str:
    likeliest = np.argsort(dist)[::-1][:5]
    alternatives = ", ".join(
        f"{tokenizer.decode([int(i)])!r} {dist[i]:.2f}" for i in likeliest
    )
    return f"p = {dist[token]:.2f}; likeliest: {alternatives}"


for temperature, (new, dists) in runs.items():
    chosen = [float(d[t]) for t, d in zip(new, dists)]
    mean = np.mean(chosen)
    display(Markdown(f"`temperature={temperature}`, mean p = {mean:.2f}:"))
    display(
        viz.tokens(
            [tokenizer.decode([t]) for t in new],
            new,
            colors=[f"rgba(42,120,214,{0.08 + 0.72 * p:.2f})" for p in chosen],
            titles=[chip_title(t, d) for t, d in zip(new, dists)],
        )
    )

temperature=0, mean p = 0.56:

·young·woman·named·Sarah·who·was·struggling·to·find·her·place·in·the·world.·She·had·always·been·a·bit·of·a·rebel

temperature=1.5, mean p = 0.01:

·neighboring·realm·on·your·planet,·mist.·Even·then;·Unlocking·beastellandsis·deathshire·istsubbralgenaki

Higher temperatures read less predictably. The greedy run picks tokens with probabilities from about 0.1 to over 0.9. At 1.5 without a filter, nearly every chip is pale: a flattened distribution gives the long tail of unlikely tokens enough combined weight that most picks are tokens the model gave almost no chance, and the text turns to gibberish within a few words. Many chat applications use settings near temperature=0.7 and top_p=0.9. The handbook’s inference parameters page covers these and the other settings servers accept, such as stop sequences and repetition penalties.

The cost of each token

Generating the token after \(n\) tokens runs the model over all \(n\) of them. Time that step at several lengths, with random token IDs since only the length matters:

rng = np.random.default_rng(0)
lengths = [16, 64, 256, 512, 1024, 2048]
step_ms = []
for length in lengths:
    ids = rng.integers(0, config.vocab_size, length).tolist()
    next_logits(ids).to_numpy()  # warm up this length
    times = []
    for _ in range(5):
        start = time.perf_counter()
        next_logits(ids).to_numpy()
        times.append((time.perf_counter() - start) * 1000)
    step_ms.append(float(np.median(times)))
    print(f"{length:5} tokens: {step_ms[-1]:6.1f} ms per new token")
   16 tokens:   57.7 ms per new token
   64 tokens:   60.1 ms per new token
  256 tokens:   70.9 ms per new token
  512 tokens:  179.8 ms per new token
 1024 tokens:  563.3 ms per new token
 2048 tokens: 1839.3 ms per new token
Show the code 17 lines
fig = go.Figure(
    go.Scatter(
        x=lengths,
        y=step_ms,
        mode="lines+markers",
        line_color=viz.ORANGE,
        hovertemplate="%{x} tokens: %{y:.1f} ms<extra></extra>",
    )
)
fig.update_layout(
    title="Time to generate one token, recomputing the whole sequence",
    xaxis_title="sequence length (tokens)",
    yaxis_title="ms per token",
    yaxis_rangemode="tozero",
    height=380,
)
viz.show(fig)

In the run shown here, on an Apple-silicon Mac with a Metal GPU, a step took about 70 to 110 ms up to 256 tokens, about 560 ms at 1,024 tokens, and about 1.8 s at 2,048. Treat these numbers as an example: yours depend on your hardware and vary from run to run. For short sequences the step costs about the same at any length, because fixed work, such as reading every weight and launching each kernel, dominates. Past a few hundred tokens the cost climbs faster than the length itself, because attention compares every position with every earlier one. Other devices give other numbers with the same shape.

The keys and values of earlier positions never change, so all the work of recomputing them is wasted. Section 1.8 caches them. The handbook’s LLM inference metrics page explains why this per-token time, the TPOT from Section 1.1, is the number users feel while a reply streams in.

Exercises

  1. Sample the same prompt 5 times at temperature=1.0 with and without top_p=0.9. How often does each run go off topic?
  2. Implement min-p sampling: keep tokens whose probability is at least min_p times the top token’s probability. Which MAX operations do you need?
  3. Plot how many tokens top-p keeps at each step of a generation. When is the model confident?

Next

Section 1.8 adds a KV cache so each step processes one token instead of the whole sequence.

1.8 The KV cache

Open in Colab Open in Kaggle View on GitHub

Section 1.7 ended with a problem: each generated token reruns the model over the whole sequence, so the cost per token grows with the text. This section fixes it the way inference engines do, with a KV cache, and measures the difference.

Why caching works

To compute attention for a new token, the model needs the keys and values of every earlier position, in every layer. Those keys and values depend only on the tokens up to their own position, so they never change once computed. Recomputing them every step is wasted work. The KV cache stores them instead, which splits generation into two phases:

  • Prefill runs the whole prompt at once, like before, and saves every layer’s keys and values.
  • Decode runs one new token per step. Each layer computes the new token’s query, key, and value, appends the key and value to its cache, and attends over the whole cache.

The handbook’s How does LLM inference work? page describes the same two phases from the serving side.

Show the setup 27 lines
import logging
import time
import warnings

import numpy as np
import plotly.graph_objects as go
from max.driver import CPU
from plotly.subplots import make_subplots
from max.dtype import DType
from max.graph import TensorType
from max.pipelines import AutoTokenizer
from max.tensor import Tensor

from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.model import load_model
from minillm.sampling import generate
from minillm.weights import load_checkpoint

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU

config = LlamaConfig.from_pretrained()
weights = load_checkpoint()
tokenizer = AutoTokenizer(MODEL_ID)

The difference in work is easy to see. Each row below is one generation step for a 6-token prompt; each column is a position whose key and value that step computes. Without a cache, every step recomputes every earlier position. With one, prefill computes the prompt once and each decode step computes one new position:

prompt_len, steps = 6, 10
step_labels = ["prefill"] + [f"decode {i}" for i in range(1, steps)]
recompute_work = np.zeros((steps, prompt_len + steps - 1))
cached_work = np.zeros_like(recompute_work)
for step in range(steps):
    end = prompt_len + step
    recompute_work[step, :end] = 1
    cached_work[step, :end] = 1 if step == 0 else 0
    cached_work[step, end - 1] = 1
Show the code 28 lines
fig = make_subplots(
    rows=1,
    cols=2,
    shared_yaxes=True,
    horizontal_spacing=0.06,
    subplot_titles=[
        f"Recompute: {int(recompute_work.sum())} key/value computations",
        f"KV cache: {int(cached_work.sum())} key/value computations",
    ],
)
for col, work in enumerate([recompute_work, cached_work], start=1):
    fig.add_trace(
        go.Heatmap(
            z=np.where(work > 0, 1, np.nan),
            y=step_labels,
            xgap=2,
            ygap=2,
            colorscale=[[0, viz.BLUE], [1, viz.BLUE]],
            showscale=False,
            hovertemplate="%{y}: computes position %{x}<extra></extra>",
        ),
        row=1,
        col=col,
    )
fig.update_xaxes(title="position", showgrid=False, dtick=2)
fig.update_yaxes(autorange="reversed", showgrid=False)
fig.update_layout(height=380, title="Which keys and values each step computes")
viz.show(fig)

How big is the cache?

Each token stores one key and one value vector per key/value head, per layer. With \(L\) layers, \(H_{kv}\) key/value heads of dimension \(d\), and \(b\) bytes per number:

\[\text{bytes per token} = 2 \cdot L \cdot H_{kv} \cdot d \cdot b = 2 \cdot 30 \cdot 3 \cdot 64 \cdot 4\]

per_token = (
    2
    * config.num_hidden_layers
    * config.num_key_value_heads
    * config.head_dim
    * 4
)
print(f"{per_token / 1024:.0f} KiB per token in float32")
print(
    f"{per_token * 8192 / 2**20:.0f} MiB for SmolLM2's full 8192-token context"
)
print(
    f"{per_token * 3 * 8192 / 2**20:.0f} MiB if every query head had its own key/value head"
)
45 KiB per token in float32
360 MiB for SmolLM2's full 8192-token context
1080 MiB if every query head had its own key/value head

The cache grows with the context and with the number of sequences served at once. Compare it with the model’s weights:

weights_mib = sum(w.nbytes for w in weights.values()) / 2**20
two_sequences_mib = per_token * 8192 * 2 / 2**20
print(f"weights: {weights_mib:.0f} MiB")
print(f"cache for two sequences of 8192 tokens: {two_sequences_mib:.0f} MiB")
weights: 513 MiB
cache for two sequences of 8192 tokens: 720 MiB

Move the slider to serve more sequences at once:

Show the code 43 lines
lengths = np.arange(256, 8193, 256)
batches = [1, 2, 4, 16, 64]
steps = []
for batch in batches:
    steps.append(
        [
            go.Scatter(
                x=lengths,
                y=per_token * lengths * batch / 2**20,
                name="KV cache, 3 key/value heads",
                line_color=viz.BLUE,
                hovertemplate="%{x} tokens: %{y:,.0f} MiB<extra></extra>",
            ),
            go.Scatter(
                x=lengths,
                y=3 * per_token * lengths * batch / 2**20,
                name="KV cache, 9 key/value heads",
                line=dict(color=viz.ORANGE, dash="dot"),
                hovertemplate="%{x} tokens: %{y:,.0f} MiB<extra></extra>",
            ),
        ]
    )
fig = go.Figure(
    go.Scatter(
        x=[lengths[0], lengths[-1]],
        y=[weights_mib] * 2,
        name="weights",
        mode="lines",
        line=dict(color=viz.INK_SECONDARY, dash="dash"),
        hovertemplate="weights: %{y:,.0f} MiB<extra></extra>",
    )
)
fig.update_layout(
    title="float32 KV cache memory against context length",
    height=460,
    xaxis_title="context length (tokens)",
    yaxis=dict(title="MiB", type="log", dtick=1),
)
viz.show(
    viz.with_slider(
        fig, steps, [str(b) for b in batches], "sequences served at once: "
    )
)

This is where grouped-query attention (Section 1.5) pays off: 3 key/value heads instead of 9 make the cache 3 times smaller. Even for SmolLM2 in float32, two sequences at the full 8,192-token context need a 720 MiB cache, more than the 513 MiB of weights. For large models serving many users, the cache often takes more GPU memory than the weights. The handbook’s GPU memory calculation adds it to the weights when sizing a deployment.

Attention with a cache

CachedAttention extends Section 1.5’s Attention. Its forward takes the layer’s cached keys and values, if any, and returns them extended by the new positions. start is the position of the first new token, so RoPE rotates the new queries and keys for where they really are.

Prefill uses the causal mask as before. Decode has one query, the newest token, which may attend to every cached position, so it needs no mask.

Show CachedAttention 37 lines
from __future__ import annotations

import math

import numpy as np
from max import functional as F
from max.driver import Device
from max.dtype import DType
from max.graph import TensorType
from max.nn import ModuleList
from max.nn.kernels import flash_attention
from max.tensor import Tensor

from minillm.attention import Attention
from minillm.config import LlamaConfig
from minillm.model import DecoderLayer, LlamaForCausalLM, LlamaModel

KVPair = tuple[Tensor, Tensor]


class CachedAttention(Attention):
    """Attention that reads and extends a key/value cache."""

    def forward(
        self, x: Tensor, start: Tensor, past: KVPair | None = None
    ) -> tuple[Tensor, KVPair]:
        q, k, v = self.project(x, start)
        if past is None:  # prefill: causal attention over the prompt
            return self.attend(q, k, v), (k, v)
        k = F.concat([past[0], k], axis=1)
        v = F.concat([past[1], v], axis=1)
        # The one new token may attend to every cached position: no mask.
        out = flash_attention(q, k, v, 1 / math.sqrt(self.head_dim))
        out = out.rebind_mapping(q.mapping)
        batch, seq = q.shape[0], q.shape[1]
        out = out.reshape([batch, seq, self.num_heads * self.head_dim])
        return self.o_proj(out), (k, v)

Saved to minillm/kv_cache.py.

The layer and model pass each layer’s cache through. The model returns logits for the last position only: generation never needs the others, and skipping them saves most of the LM head’s work during prefill.

Show CachedDecoderLayer, CachedLlamaModel, CachedLlamaForCausalLM 52 lines
class CachedDecoderLayer(DecoderLayer):
    def __init__(self, config: LlamaConfig) -> None:
        super().__init__(config)
        self.self_attn = CachedAttention(config)

    def forward(
        self, x: Tensor, start: Tensor, past: KVPair | None = None
    ) -> tuple[Tensor, KVPair]:
        attn, kv = self.self_attn(self.input_layernorm(x), start, past)
        x = x + attn
        return x + self.mlp(self.post_attention_layernorm(x)), kv


class CachedLlamaModel(LlamaModel):
    def __init__(self, config: LlamaConfig) -> None:
        super().__init__(config)
        self.layers = ModuleList(
            [
                CachedDecoderLayer(config)
                for _ in range(config.num_hidden_layers)
            ]
        )

    def forward(
        self,
        input_ids: Tensor,
        start: Tensor,
        cache: list[KVPair] | None = None,
    ) -> tuple[Tensor, list[KVPair]]:
        x = self.embed_tokens(input_ids)
        new_cache = []
        for i, layer in enumerate(self.layers):
            x, kv = layer(x, start, None if cache is None else cache[i])
            new_cache.append(kv)
        return self.norm(x), new_cache


class CachedLlamaForCausalLM(LlamaForCausalLM):
    """Returns the last position's logits and the updated cache."""

    def __init__(self, config: LlamaConfig) -> None:
        super().__init__(config)
        self.model = CachedLlamaModel(config)

    def forward(
        self,
        input_ids: Tensor,
        start: Tensor,
        cache: list[KVPair] | None = None,
    ) -> tuple[Tensor, list[KVPair]]:
        hidden, new_cache = self.model(input_ids, start, cache)
        return self.lm_head(hidden[:, -1]), new_cache

Saved to minillm/kv_cache.py.

Two compiled graphs

Prefill and decode have different inputs, so they compile into two graphs that share the same weights. The cache is a list of 30 (key, value) pairs. Its length, cache_len, is a named dimension, so one decode graph serves every step. start is a one-element tensor rather than a Python number, so that it is an input of the graph instead of a constant baked into it. It is float32 because RoPE computes its angles in float32, and float32 holds every position up to 2^24 exactly.

KVCacheDecoder keeps the cache between calls. It has the same interface as Section 1.7’s next_logits: given the tokens so far, it returns the next token’s logits, running a prefill for a new prompt and a decode step when exactly one token was added. It plugs straight into generate.

Show KVCacheDecoder 41 lines
class KVCacheDecoder:
    """Returns next-token logits like a full forward pass, using a cache."""

    def __init__(
        self, model: CachedLlamaForCausalLM, config: LlamaConfig, device: Device
    ) -> None:
        self.device = device
        tokens = TensorType(DType.int64, ("batch", "seq_len"), device=device)
        one_token = TensorType(DType.int64, ("batch", 1), device=device)
        start = TensorType(DType.float32, [1], device=device)
        kv = TensorType(
            DType.float32,
            ("batch", "cache_len", config.num_key_value_heads, config.head_dim),
            device=device,
        )
        self.prefill = model.compile(tokens, start)
        self.decode = model.compile(
            one_token, start, [(kv, kv)] * config.num_hidden_layers
        )
        self.cache: list[KVPair] | None = None
        self.length = 0

    def __call__(self, ids: list[int]) -> Tensor:
        if self.cache is not None and len(ids) == self.length + 1:
            logits, self.cache = self.decode(
                self._tokens(ids[-1:]), self._position(self.length), self.cache
            )
        else:
            logits, self.cache = self.prefill(
                self._tokens(ids), self._position(0)
            )
        self.length = len(ids)
        return logits[0]

    def _tokens(self, ids: list[int]) -> Tensor:
        return Tensor(np.array([ids], dtype=np.int64), device=self.device)

    def _position(self, position: int) -> Tensor:
        return Tensor(
            np.array([position], dtype=np.float32), device=self.device
        )

Saved to minillm/kv_cache.py.

start = time.perf_counter()
cached = KVCacheDecoder(
    load_model(config, weights, DEVICE, CachedLlamaForCausalLM), config, DEVICE
)
print(f"compiled prefill and decode in {time.perf_counter() - start:.0f} s")
compiled prefill and decode in 30 s

Check it

The cache must not change the result. Generate greedily with Section 1.7’s full recompute and with the cache, and compare the tokens:

full = load_model(config, weights, DEVICE).compile(
    TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)


def recompute(ids: list[int]) -> Tensor:
    return full(Tensor(np.array([ids]), device=DEVICE))[0, -1]


prompt = tokenizer(["The capital of France is"])["input_ids"][0]
slow = generate(recompute, prompt, 48, temperature=0)
fast = generate(cached, prompt, 48, temperature=0)
print("identical tokens:", slow == fast)
assert slow == fast
print(tokenizer.decode(prompt + fast))
print("cached keys per layer:", [int(d) for d in cached.cache[0][0].shape])
identical tokens: True
The capital of France is Paris. Paris is the largest city in France and the capital of the French department of the Espace de Paris.

The Eiffel Tower, also known as the Eiffel Tower, is a monumental structure that stands 324
cached keys per layer: [1, 52, 3, 64]

Measure it

Time one decode step at several context lengths, with random prompts so that only the length matters:

def ms_per_token(next_logits, prompt: list[int], steps: int = 24) -> float:
    ids = list(prompt)
    ids.append(int(np.argmax(next_logits(ids).to_numpy())))  # prefill, untimed
    start = time.perf_counter()
    for _ in range(steps):
        ids.append(int(np.argmax(next_logits(ids).to_numpy())))
    return (time.perf_counter() - start) / steps * 1000
rng = np.random.default_rng(0)
lengths = [32, 128, 256, 512, 1024]
timings = {"recompute": [], "KV cache": []}
for length in lengths:
    prompt = rng.integers(0, config.vocab_size, length).tolist()
    timings["recompute"].append(ms_per_token(recompute, prompt))
    timings["KV cache"].append(ms_per_token(cached, prompt))
for name, ms in timings.items():
    print(f"{name:10}", "  ".join(f"{t:6.1f}" for t in ms), "ms")
recompute   100.9   130.0   134.0   209.5   595.8 ms
KV cache    112.9   136.1   110.8   135.5   169.9 ms
Show the code 21 lines
fig = go.Figure(
    [
        go.Scatter(
            x=lengths,
            y=ms,
            name=name,
            mode="lines+markers",
            line_color=color,
            hovertemplate=name + ", %{x} tokens: %{y:.1f} ms<extra></extra>",
        )
        for (name, ms), color in zip(timings.items(), [viz.ORANGE, viz.BLUE])
    ]
)
fig.update_layout(
    title="Time per decode step",
    height=420,
    xaxis_title="context length (tokens)",
    yaxis_title="ms per token",
    yaxis_rangemode="tozero",
)
viz.show(fig)

Hover over the points to read the timings; yours depend on your hardware and vary from run to run. Recompute grows with the context, because every step reruns attention over the whole sequence. The cached step grows much more slowly. Below a few hundred tokens the cache doesn’t pay off, and can even be slower, because fixed work, such as launching kernels and reading every weight, dominates a step of a model this small.

MAX’s own decode step is far faster: the TPOT measured in Section 1.1 was about 10 ms on the same machine. Part of the gap is this section’s simple design: every decode call passes the 60 cache tensors in and gets 60 new ones back, F.concat copies each layer’s whole cache into a new tensor, and the loop reads the logits back to NumPy to pick each token on the host. MAX’s engine writes new keys and values into preallocated cache blocks that its attention kernel reads in place, and it samples on the GPU. Section 5.3 builds a block-based cache and explains how production engines avoid the copy.

This simple cache has limits that production engines address:

  • Memory is allocated per sequence, as it grows. A server doesn’t know how long each reply will be, and contiguous per-sequence buffers fragment memory. Paged attention stores the cache in fixed-size blocks instead, like virtual memory pages.
  • It serves one sequence at a time. A decode step for one token barely uses the hardware. Engines batch many sequences per step, which needs a cache that holds sequences of different lengths.
  • Shared prompts are recomputed. Requests that start with the same system prompt could share those cache blocks (prefix caching).

Chapter 5 builds these into a small serving engine: Section 5.2 adds continuous batching, Section 5.3 a paged KV cache, and Section 5.4 prefix caching. The handbook covers each one: PagedAttention, batching, and prefix caching.

Exercises

  1. Extend the timing chart to context lengths from 16 to 2048. Below what length does the cache stop paying off on your device, and why?
  2. Make KVCacheDecoder keep the cache across two turns of a conversation: prefill only the new turn’s tokens after the first reply.
  3. Store the cache in bfloat16 instead of float32. How much memory does it save, and do the greedy tokens change?

What you built

Chapter 1 is complete. You have a working LLM inference stack written with MAX: a tokenizer and weight loader, a Llama-architecture model that matches MAX’s own, sampling, and a KV cache. Chapter 2 splits this model across several devices with MAX’s distributed tensors.

2. Distributed tensors

Large models don’t fit on one device, and even small ones serve faster when several devices share the work. In MAX, a tensor that spans several devices behaves like any other tensor: you describe how it is laid out, and MAX runs each operation on every device and inserts the communication it needs.

Distributed code is hard to develop when every test needs a multi-GPU machine. MAX can simulate a mesh of devices on one CPU, with the same code and the same results, so this whole chapter runs on a laptop or in Colab.

SectionYou learn
2.1 Distributed tensorsDevice meshes, placements, sharded matmuls, and the collectives they imply.
2.2 A tensor-parallel modelSplitting SmolLM2 across three devices and checking that its logits don’t change.

The handbook’s parallelism strategies and distributed inference pages explain when each strategy pays off in production.

2.1 Distributed tensors

Open in Colab Open in Kaggle View on GitHub

A distributed tensor is one logical array whose data lives on several devices. In MAX it is an ordinary Tensor, the same type you used from Section 1.2 on: operators, max.functional, and max.nn layers accept it, run on every device that holds a piece, and add the communication between devices that the math requires.

This section runs everything on a simulated mesh: four “devices” that are all the same CPU. The program, the per-device pieces, and the results are the same as on four GPUs, so you can develop and test distributed code on a laptop and run it unchanged on real hardware.

Show the setup 23 lines
import logging
import re
import warnings

import numpy as np
import plotly.graph_objects as go
from max import functional as F
from max.driver import CPU
from max.tensor import Tensor
from max.tensor.sharding import (
    DeviceMapping,
    DeviceMesh,
    Partial,
    Replicated,
    Sharded,
    auto_reshard,
    visualize,
)
from plotly.subplots import make_subplots

from minillm import viz

logging.getLogger("max").setLevel(logging.ERROR)

Meshes and placements

A DeviceMesh arranges devices on named axes. This one has four devices on one axis, named tp for tensor parallelism:

mesh = DeviceMesh((CPU(),) * 4, (4,), ("tp",))


def on(*placements) -> DeviceMapping:
    """Places a tensor on the mesh, with one placement per mesh axis."""
    return DeviceMapping(mesh, placements)

A DeviceMapping says how a tensor’s data relates to the devices on each mesh axis. That relation is its placement:

  • Replicated(): every device holds the whole tensor.
  • Sharded(axis): the tensor is split along axis, one piece per device.
  • Partial(): every device holds a full-size tensor, and the real value is their sum.

Split an 8 × 4 matrix by rows:

a = np.arange(32, dtype=np.float32).reshape(8, 4)
x = Tensor(a, device=on(Sharded(0)))
visualize(x)
DType.float32 [8, 4] with DeviceMapping(DeviceMesh(tp=4), [Sharded(axis=0)])
┌──────────┐   tensor[:, :]
│ device 0 │   ├── tp=0  device 0 - tensor[:2, :]
│ [:2, :]  │   ├── tp=1  device 1 - tensor[2:4, :]
├──────────┤   ├── tp=2  device 2 - tensor[4:6, :]
│ device 1 │   └── tp=3  device 3 - tensor[6:, :]
│ [2:4, :] │
├──────────┤
│ device 2 │
│ [4:6, :] │
├──────────┤
│ device 3 │
│ [6:, :]  │
└──────────┘

x.shape is the global shape, [8, 4], as if the tensor lived on one device. Each device’s piece, its shard, is in local_shards:

print("global shape:", [int(d) for d in x.shape])
for i, shard in enumerate(x.local_shards):
    print(f"device {i}:", [int(d) for d in shard.shape], shard.to_numpy()[:, 0])
global shape: [8, 4]
device 0: [2, 4] [0. 4.]
device 1: [2, 4] [ 8. 12.]
device 2: [2, 4] [16. 20.]
device 3: [2, 4] [24. 28.]

The next helper turns a tensor’s placement into a map of which device holds each element, using only mapping.placements and the shard shapes. The figures in this section draw it: one color per device, gray where every device holds the element, and a darker gray for a Partial tensor, whose every device holds a term of the sum.

Show owners, owner_heatmap, device_legend 73 lines
ALL, PARTIAL = -1, -2


def owners(t: Tensor) -> np.ndarray:
    """Returns the device that holds each element of ``t`` on a 1-D mesh."""
    grid = np.full([int(d) for d in t.shape], ALL)
    placement = t.mapping.placements[0]
    if isinstance(placement, Partial):
        grid[:] = PARTIAL
    elif isinstance(placement, Sharded):
        start = 0
        for device, shard in enumerate(t.local_shards):
            size = int(shard.shape[placement.axis])
            index = [slice(None)] * grid.ndim
            index[placement.axis] = slice(start, start + size)
            grid[tuple(index)] = device
            start += size
    return grid


HOLDER = {ALL: "every device", PARTIAL: "every device, one term each"} | {
    d: f"device {d}" for d in range(4)
}
# One band per value from -2 to 3: partial, all, then devices 0 to 3.
BANDS = [
    viz.INK_SECONDARY,
    viz.GRID,
    viz.BLUE,
    viz.ORANGE,
    viz.AQUA,
    viz.YELLOW,
]
OWNER_SCALE = [
    [edge, color]
    for i, color in enumerate(BANDS)
    for edge in (i / 6, (i + 1) / 6)
]


def owner_heatmap(t: Tensor, show_values: bool = True, **trace) -> go.Heatmap:
    grid = owners(t)
    values = t.to_numpy()
    rows, cols = grid.shape
    holder = [
        [HOLDER[int(grid[r, c])] for c in range(cols)] for r in range(rows)
    ]
    return go.Heatmap(
        z=grid,
        zmin=-2.5,
        zmax=3.5,
        colorscale=OWNER_SCALE,
        showscale=False,
        xgap=2,
        ygap=2,
        customdata=holder,
        text=values.round(1) if show_values else None,
        texttemplate="%{text}" if show_values else None,
        hovertemplate="row %{y}, column %{x}<br>held by %{customdata}<extra></extra>",
        **trace,
    )


def device_legend(fig: go.Figure) -> None:
    for d in range(4):
        fig.add_trace(
            go.Scatter(
                x=[None],
                y=[None],
                mode="markers",
                name=f"device {d}",
                marker=dict(symbol="square", size=12, color=BANDS[d + 2]),
            )
        )

Move the slider to compare placements of the same 8 × 4 matrix:

placements = {
    "Sharded(0)": Sharded(0),
    "Sharded(1)": Sharded(1),
    "Replicated()": Replicated(),
}
steps = [[owner_heatmap(Tensor(a, device=on(p)))] for p in placements.values()]
fig = go.Figure(
    layout=dict(
        title="Which device holds each element of an 8 × 4 matrix",
        height=520,
        xaxis=dict(showgrid=False, dtick=1),
        yaxis=dict(showgrid=False, autorange="reversed", dtick=1),
    )
)
device_legend(fig)
viz.show(viz.with_slider(fig, steps, list(placements), "placement "))

to_numpy() gathers the shards back into one array, so checking a distributed result against NumPy is one line:

print(np.array_equal(x.to_numpy(), a))
True

Sharded matrix multiplication

Models spend most of their time in matrix multiplications, and there are two natural ways to split \(y = x W\) across devices. Every tensor-parallel transformer uses both.

In a column-parallel multiplication, every device holds all of \(x\) and a slice of \(W\)’s columns, and computes the matching slice of \(y\). It needs no communication:

w = np.random.default_rng(0).standard_normal((4, 6)).astype(np.float32)

y = Tensor(a, device=on(Replicated())) @ Tensor(w, device=on(Sharded(1)))
print(y.mapping)
print("correct:", np.allclose(y.to_numpy(), a @ w, atol=1e-5))
assert np.allclose(y.to_numpy(), a @ w, atol=1e-5)
DeviceMapping(DeviceMesh(tp=4), [Sharded(axis=1)])
correct: True

In a row-parallel multiplication, each device holds a slice of \(x\)’s columns and the matching slice of \(W\)’s rows. Each device’s product is one term of the full sum over the inner dimension, so the result is Partial:

z = Tensor(a, device=on(Sharded(1))) @ Tensor(w, device=on(Sharded(0)))
print(z.mapping)
print("correct:", np.allclose(z.to_numpy(), a @ w, atol=1e-5))
assert np.allclose(z.to_numpy(), a @ w, atol=1e-5)
DeviceMapping(DeviceMesh(tp=4), [Partial()])
correct: True

A Partial tensor is correct only once its terms are summed. Most operations, such as an activation, need the real value, so MAX must first combine the shards with a collective, an operation that communicates between devices. Inside auto_reshard(mode="warn"), MAX reports every collective it inserts and the reason for it:

with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    with auto_reshard(mode="warn"):
        activated = F.relu(z)
report = str(next(w.message for w in caught if "reshards" in str(w.message)))
print(report)
print("result:", activated.mapping)

relu(x) reshards x
  x      : float32[8, 6]@mesh(tp=4){"tp":P}
  region : auto_reshard(DEFAULT_TRANSITIONS, mode="warn")

  decided per mesh axis
  axis    x          -> out       collective         bytes   status
  "tp"    x{P->S0}   -> S0        reduce_scatter(x)     144  not allowed
          x{P->S1}   -> S1        reduce_scatter(x)     144  not allowed
          x{P->R}    -> R         allreduce(x)          288  chosen, reshards x
  result  x{"tp":R}  -> {"tp":R}

  x = transfer_to(x, DeviceMapping(mesh, (R,)))
  with auto_reshard(mode="silent"):
  with auto_reshard(ALL_TRANSITIONS, mode="silent"):   # also allows the cheaper rows above
  at <cell>:4 in <module>()
result: DeviceMapping(DeviceMesh(tp=4), [Replicated()])

MAX chose an all-reduce: every device sends its partial sum and receives the total. It also considered a reduce-scatter, which sums and leaves each device one slice, but that transition isn’t in the default set. Outside an auto_reshard block, MAX raises an error instead of inserting a collective, so a model can’t add communication you didn’t plan for.

To move a tensor to a placement yourself, use F.transfer_to:

replicated = F.transfer_to(z, on(Replicated()))
print(replicated.mapping)
DeviceMapping(DeviceMesh(tp=4), [Replicated()])

The figure shows what the two collectives do to z, built from its real shards. Each device in the top row holds a different full-size term, and the four terms add up to a @ w. Move the slider to see what each device holds afterward: an all-reduce gives every device the whole sum; a reduce-scatter gives each device only its two rows of the sum, for half the bytes. The byte counts come from MAX’s report above.

Show cells 66 lines
terms = [shard.to_numpy() for shard in z.local_shards]
assert np.allclose(sum(terms), a @ w, atol=1e-4)
after = {
    "all-reduce": F.transfer_to(z, on(Replicated())),
    "reduce-scatter": F.transfer_to(z, on(Sharded(0))),
}
cost = {
    name: int(re.search(rf"{op}\(x\)\s+(\d+)", report).group(1))
    for name, op in [
        ("all-reduce", "allreduce"),
        ("reduce-scatter", "reduce_scatter"),
    ]
}
limit = np.abs(a @ w).max()


def cells(values: np.ndarray, rows: list[int], **trace) -> go.Heatmap:
    return go.Heatmap(
        z=values,
        y=rows,
        zmin=-limit,
        zmax=limit,
        colorscale=viz.DIVERGING,
        showscale=False,
        xgap=1,
        ygap=1,
        texttemplate="%{z:.0f}",
        textfont=dict(size=9),
        hovertemplate="row %{y}, column %{x}: %{z:.2f}<extra></extra>",
        **trace,
    )


fig = make_subplots(
    rows=2,
    cols=4,
    vertical_spacing=0.14,
    horizontal_spacing=0.03,
    subplot_titles=[f"device {d}: its term" for d in range(4)]
    + [f"device {d} afterward" for d in range(4)],
)
for d, term in enumerate(terms):
    fig.add_trace(cells(term, list(range(8))), row=1, col=d + 1)
steps = []
for name, tensor in after.items():
    group = []
    start = 0
    for d, shard in enumerate(tensor.local_shards):
        values = shard.to_numpy()
        rows = (
            list(range(start, start + len(values)))
            if len(values) < 8
            else list(range(8))
        )
        assert np.allclose(values, (a @ w)[rows], atol=1e-4)
        start += len(values)
        group.append(cells(values, rows, xaxis=f"x{d + 5}", yaxis=f"y{d + 5}"))
    steps.append(group)
fig.update_xaxes(showgrid=False, showticklabels=False)
fig.update_yaxes(showgrid=False, showticklabels=False, range=[7.5, -0.5])
fig.update_layout(
    title="What a collective does to a Partial tensor on 4 devices",
    height=560,
)
labels = [f"{name}, {cost[name]} bytes per device" for name in after]
viz.show(viz.with_slider(fig, steps, labels, ""))

Column then row: one all-reduce per block

Chaining the two patterns is the core of tensor parallelism, as introduced by the Megatron-LM paper. A column-parallel layer leaves its output split by columns, which is the input a row-parallel layer needs, so the pair runs with no communication between them and a single all-reduce at the end. Here is a small MLP block, checked against NumPy:

rng = np.random.default_rng(1)
w_up = rng.standard_normal((4, 16)).astype(np.float32)
w_down = rng.standard_normal((16, 4)).astype(np.float32)

with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    with auto_reshard(mode="warn"):
        h = F.relu(
            Tensor(a, device=on(Replicated()))
            @ Tensor(w_up, device=on(Sharded(1)))
        )
        partial_out = h @ Tensor(w_down, device=on(Sharded(0)))
reshards = [w for w in caught if "reshards" in str(w.message)]
out = F.transfer_to(partial_out, on(Replicated()))  # the one all-reduce

print("hidden:", h.mapping, " block output:", partial_out.mapping)
print("collectives MAX had to insert:", len(reshards))
print(
    "correct:",
    np.allclose(out.to_numpy(), np.maximum(a @ w_up, 0) @ w_down, atol=1e-3),
)
assert not reshards and np.allclose(
    out.to_numpy(), np.maximum(a @ w_up, 0) @ w_down, atol=1e-3
)
hidden: DeviceMapping(DeviceMesh(tp=4), [Sharded(axis=1)])  block output: DeviceMapping(DeviceMesh(tp=4), [Partial()])
collectives MAX had to insert: 0
correct: True

Here is the whole block as placements. Every device holds all of x. The column-parallel weight and the hidden activation are split by columns in the same way, so each device multiplies only its own slice. The row-parallel weight is split by rows to match, and the block’s output, h W_down, is a partial sum on every device (dark gray) until the all-reduce sums it:

Show the code 28 lines
panels = {
    "x": Tensor(a, device=on(Replicated())),
    "W_up": Tensor(w_up, device=on(Sharded(1))),
    "h": h,
    "W_down": Tensor(w_down, device=on(Sharded(0))),
    "h W_down": partial_out,
    "summed": out,
}
widths = [int(t.shape[1]) for t in panels.values()]
fig = make_subplots(
    rows=1,
    cols=len(panels),
    column_widths=widths,
    horizontal_spacing=0.03,
    subplot_titles=list(panels),
)
for col, t in enumerate(panels.values(), start=1):
    fig.add_trace(owner_heatmap(t, show_values=False), row=1, col=col)
device_legend(fig)
fig.update_xaxes(showgrid=False, showticklabels=False)
fig.update_yaxes(showgrid=False, showticklabels=False, autorange="reversed")
fig.update_layout(
    title="A column-parallel, row-parallel MLP block on 4 devices",
    height=420,
    legend=dict(y=-0.04, yanchor="top"),
    margin=dict(t=96),
)
viz.show(fig)

The hidden activation stayed sharded, relu ran on each shard independently, and the only communication was the explicit all-reduce at the end. Section 2.2 applies this pattern to every attention and MLP block of SmolLM2.

Two-dimensional meshes

Real deployments combine strategies. A mesh with two axes can split a batch of requests across dp (data parallelism) while splitting each layer’s weights across tp. Each mesh axis gets its own placement:

mesh_2d = DeviceMesh((CPU(),) * 4, (2, 2), ("dp", "tp"))
batch = Tensor(
    np.ones((4, 6), np.float32),
    device=DeviceMapping(mesh_2d, (Sharded(0), Sharded(1))),
)
visualize(batch)
DType.float32 [4, 6] with DeviceMapping(DeviceMesh(dp=2, tp=2), [Sharded(axis=0), Sharded(axis=1)])
┌──────────┬──────────┐   tensor[:, :]
│ device 0 │ device 1 │   ├── dp=0 - tensor[:2, :]
│ [:2, :3] │ [:2, 3:] │   │   ├── tp=0  device 0 - tensor[:2, :3]
├──────────┼──────────┤   │   └── tp=1  device 1 - tensor[:2, 3:]
│ device 2 │ device 3 │   └── dp=1 - tensor[2:, :]
│ [2:, :3] │ [2:, 3:] │       ├── tp=0  device 2 - tensor[2:, :3]
└──────────┴──────────┘       └── tp=1  device 3 - tensor[2:, 3:]

On real hardware, the cost of each collective depends on how much data it moves and how fast the devices are connected, and that decides the best layout for a given model and cluster. The handbook’s parallelism strategies page compares data, tensor, pipeline, and expert parallelism. MAX’s docs cover distributed tensors in more detail.

Exercises

  1. Shard x with Sharded(1) instead of Sharded(0) and visualize it. What happens with 3 devices instead of 4, which don’t divide 4 columns evenly?
  2. Swap the order in the MLP block: row-parallel first, then column-parallel. Which collectives does MAX insert now?
  3. Count the bytes each device sends in the all-reduce of the MLP block. How does it scale with the batch size and the hidden size? The bytes column in MAX’s reports is this per-device volume for a ring algorithm, so you can check your count against one.

Next

Section 2.2 splits SmolLM2 across devices and checks that its logits don’t change.

2.2 A tensor-parallel model

Open in Colab Open in Kaggle View on GitHub

This section splits your SmolLM2 from Chapter 1 across three devices with tensor parallelism, the strategy servers use to run a model too big or too slow for one GPU. The model code doesn’t change. You change where its weights live, and MAX runs every layer on all three devices and adds the communication. Then you check that the logits match the single-device model’s.

Show the setup 31 lines
import logging
import time
import warnings

import numpy as np
import plotly.graph_objects as go
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.graph import TensorType
from max.nn.common_layers.linear import col_parallel, row_parallel
from max.pipelines import AutoTokenizer
from max.tensor import Tensor, default_device, default_dtype
from max.tensor.sharding import (
    DeviceMapping,
    DeviceMesh,
    Replicated,
    auto_reshard,
    visualize,
)

from minillm import viz
from minillm.config import MODEL_ID, LlamaConfig
from minillm.model import LlamaForCausalLM, load_model
from minillm.weights import load_checkpoint

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")

config = LlamaConfig.from_pretrained()
weights = load_checkpoint()

The plan

Section 2.1 showed the pattern: a column-parallel layer followed by a row-parallel layer needs one all-reduce. Each SmolLM2 layer has two such pairs:

BlockColumn parallel (split outputs)Row parallel (split inputs)
Attentionq_proj, k_proj, v_projo_proj
MLPgate_proj, up_projdown_proj

Tensor-parallel MLP: the input x is replicated on every device; gate and up projections are split by column, so each device holds a slice of the intermediate activation; the down projection is split by row, so each device holds a partial sum; an all-reduce adds the partial sums into the output y.Tensor-parallel MLP: the input x is replicated on every device; gate and up projections are split by column, so each device holds a slice of the intermediate activation; the down projection is split by row, so each device holds a partial sum; an all-reduce adds the partial sums into the output y.

Splitting q_proj’s outputs three ways gives each device 3 of the 9 query heads, and splitting k_proj and v_proj gives each device 1 of the 3 key/value heads, the one those queries share under grouped-query attention (Section 1.5). So attention runs on each device independently, on its own heads. The embedding, the norms, and the LM head stay whole on every device (replicated).

max.nn provides col_parallel and row_parallel, which change a Linear layer’s weight placement on the mesh axis you name:

from __future__ import annotations

from max.nn.common_layers.linear import col_parallel, row_parallel

from minillm.model import LlamaForCausalLM


def tensor_parallel(
    model: LlamaForCausalLM, axis: str = "tp"
) -> LlamaForCausalLM:
    """Shards every layer's projections across mesh axis ``axis``."""
    for layer in model.model.layers:
        attn, mlp = layer.self_attn, layer.mlp
        for linear in (
            attn.q_proj,
            attn.k_proj,
            attn.v_proj,
            mlp.gate_proj,
            mlp.up_proj,
        ):
            col_parallel(linear, axis)
        for linear in (attn.o_proj, mlp.down_proj):
            row_parallel(linear, axis)
    return model

Saved to minillm/parallel.py.

Build the model on a mesh

Build the model inside default_device(mesh) so its weights are created on the mesh, apply the plan, and load the checkpoint. load_state_dict cuts each weight into the shards its placement asks for:

mesh = DeviceMesh((CPU(),) * 3, (3,), ("tp",))

with F.lazy(), default_device(mesh), default_dtype(DType.float32):
    tp_model = tensor_parallel(LlamaForCausalLM(config))
tp_model.load_state_dict(
    {**weights, "lm_head.weight": weights["model.embed_tokens.weight"]}
)

visualize(tp_model.model.layers[0].self_attn.q_proj.weight)
visualize(tp_model.model.layers[0].self_attn.o_proj.weight)
DType.float32 [576, 576] with DeviceMapping(DeviceMesh(tp=3), [Sharded(axis=0)])
┌──────────────┐   tensor[:, :]
│   device 0   │   ├── tp=0  device 0 - tensor[:192, :]
│  [:192, :]   │   ├── tp=1  device 1 - tensor[192:384, :]
├──────────────┤   └── tp=2  device 2 - tensor[384:, :]
│   device 1   │
│ [192:384, :] │
├──────────────┤
│   device 2   │
│  [384:, :]   │
└──────────────┘
DType.float32 [576, 576] with DeviceMapping(DeviceMesh(tp=3), [Sharded(axis=1)])
┌──────────────┬──────────────┬──────────────┐   tensor[:, :]
│   device 0   │   device 1   │   device 2   │   ├── tp=0  device 0 - tensor[:, :192]
│  [:, :192]   │ [:, 192:384] │  [:, 384:]   │   ├── tp=1  device 1 - tensor[:, 192:384]
└──────────────┴──────────────┴──────────────┘   └── tp=2  device 2 - tensor[:, 384:]

Read the head assignment straight from the shards: each device’s slice of q_proj holds whole query heads, and its slice of k_proj holds the key/value head those queries share. Hover over a box to see its device:

Show the code 43 lines
attn = tp_model.model.layers[0].self_attn
rows = {
    "query heads": attn.q_proj.weight,
    "key/value heads": attn.k_proj.weight,
}
device_colors = [viz.BLUE, viz.ORANGE, viz.AQUA]
fig = go.Figure()
for y, (name, weight) in enumerate(rows.items()):
    head = 0
    for device, shard in enumerate(weight.local_shards):
        for _ in range(int(shard.shape[0]) // config.head_dim):
            width = (
                9 / int(weight.shape[0]) * config.head_dim
            )  # key/value heads span their group
            fig.add_trace(
                go.Bar(
                    x=[width],
                    base=[head * width],
                    y=[name],
                    orientation="h",
                    width=0.6,
                    marker=dict(
                        color=device_colors[device],
                        line=dict(color=viz.SURFACE, width=3),
                    ),
                    text=f"{'q' if y == 0 else 'kv'}{head}",
                    textposition="inside",
                    insidetextanchor="middle",
                    name=f"device {device}",
                    legendgroup=str(device),
                    showlegend=y == 0 and head % 3 == 0,
                    hovertemplate=f"{name[:-1]} {head} on device {device}<extra></extra>",
                )
            )
            head += 1
fig.update_layout(
    title="Layer 0's attention heads on a 3-device mesh",
    height=260,
    barmode="overlay",
    xaxis=dict(visible=False),
    yaxis=dict(autorange="reversed", showgrid=False),
)
viz.show(fig)

Linear stores its weight as [out, in], so column parallel splits axis 0 and row parallel splits axis 1. Each device now holds a third of every projection, which is most of the model:

def elements_on_device_0(model) -> int:
    total = 0
    for _, p in model.parameters:
        shard = p.local_shards[0] if p.is_distributed else p
        total += int(np.prod([int(d) for d in shard.shape]))
    return total


total = sum(p.num_elements() for _, p in tp_model.parameters)
print(
    f"whole model: {total / 1e6:.0f}M values, device 0 holds {elements_on_device_0(tp_model) / 1e6:.0f}M"
)
whole model: 163M values, device 0 holds 92M

The whole-model count is above SmolLM2’s 135M because the model ties its LM head to the embedding, and the count includes that 28M-value weight twice. Broken down by part, for the single-device model and for each device of the mesh:

Show part 43 lines
def part(name: str) -> str:
    for key, label in [
        ("embed_tokens", "embedding"),
        ("lm_head", "LM head"),
        ("self_attn", "attention"),
        ("mlp", "MLP"),
    ]:
        if key in name:
            return label
    return "norms"


parts = ["embedding", "LM head", "attention", "MLP", "norms"]
columns = ["one device", "mesh device 0", "mesh device 1", "mesh device 2"]
counts = {p: [0] * 4 for p in parts}
for name, param in tp_model.parameters:
    counts[part(name)][0] += param.num_elements()
    shards = param.local_shards if param.is_distributed else [param] * 3
    for device, shard in enumerate(shards):
        counts[part(name)][device + 1] += int(
            np.prod([int(d) for d in shard.shape])
        )

fig = go.Figure(
    [
        go.Bar(
            x=columns,
            y=np.array(values) / 1e6,
            name=p,
            marker_color=color,
            hovertemplate=p + ": %{y:.1f}M values<extra>%{x}</extra>",
        )
        for (p, values), color in zip(counts.items(), viz.SERIES)
    ]
)
fig.update_layout(
    title="Parameter values per device",
    barmode="stack",
    height=420,
    yaxis_title="millions of values",
    legend_traceorder="normal",
)
viz.show(fig)

The replicated embedding and LM head (28M values each) keep device 0 well above a third. Large models also split those across the vocabulary.

Run it

The input token IDs are replicated: every device sees the whole prompt. Run the model with auto_reshard(mode="warn") to see each collective MAX inserts:

tokenizer = AutoTokenizer(MODEL_ID)
ids = tokenizer(["The capital of France is"])["input_ids"][0]
tokens = Tensor(np.array([ids]), device=DeviceMapping(mesh, (Replicated(),)))

start = time.perf_counter()
with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    with auto_reshard(mode="warn"):
        tp_logits = tp_model(tokens).to_numpy()
reshards = [str(w.message) for w in caught if "reshards" in str(w.message)]
print(
    f"forward pass in {time.perf_counter() - start:.0f} s, {len(reshards)} collectives inserted"
)
assert len(reshards) == 2 * config.num_hidden_layers
print(reshards[0])
forward pass in 108 s, 60 collectives inserted

add(lhs, rhs) reshards rhs
  lhs    : float32[1, 5, 576]@mesh(tp=3){"tp":R}
  rhs    : float32[1, 5, 576]@mesh(tp=3){"tp":P}
  region : auto_reshard(DEFAULT_TRANSITIONS, mode="warn")

  decided per mesh axis
  axis    lhs          rhs          -> out       collective                             bytes   status
  "tp"    lhs{R->P}    rhs{P}       -> P         keep_one_copy(lhs)                         60  not allowed
          lhs{R->S1}   rhs{P->S1}   -> S1        local_slice(lhs), reduce_scatter(rhs)    7740  not allowed
          lhs{R->S2}   rhs{P->S2}   -> S2        local_slice(lhs), reduce_scatter(rhs)    7740  not allowed
          lhs{R}       rhs{P->R}    -> R         allreduce(rhs)                          15360  chosen, reshards rhs
  result  lhs{"tp":R}  rhs{"tp":R}  -> {"tp":R}

  rhs = transfer_to(rhs, DeviceMapping(mesh, (R,)))
  with auto_reshard(mode="silent"):
  with auto_reshard(ALL_TRANSITIONS, mode="silent"):   # also allows the cheaper rows above
  at minillm/model.py:43 in forward()

That makes 60 collectives for 30 layers, two all-reduces per layer: one where the output of attention’s row-parallel o_proj meets the residual addition, and one where the MLP’s down_proj output does. Each residual addition needs the full value, so a column-then-row layout needs both. Everything else, including attention itself, ran on each device without communication. The forward pass is slow because the simulated mesh runs eagerly, one operation at a time, on one CPU; its time says nothing about real hardware.

Check it

The sharded model must produce the same logits as the single-device model from Section 1.6:

single = load_model(config, weights, CPU()).compile(
    TensorType(DType.int64, ("batch", "seq_len"), device=CPU())
)
reference = single(Tensor(np.array([ids]), device=CPU())).to_numpy()

print(f"max difference {np.abs(tp_logits - reference).max():.1e}")
print("next token:", repr(tokenizer.decode([int(tp_logits[0, -1].argmax())])))
assert (
    np.abs(tp_logits - reference).max() < 1e-3
    and tp_logits[0, -1].argmax() == reference[0, -1].argmax()
)
max difference 7.0e-05
next token: ' Paris'

The logits agree to within 1e-4, the size of a change in summation order: the all-reduce adds three partial sums where one device computed one full sum. On real hardware, replace the mesh’s CPU() entries with Accelerator(0), Accelerator(1), and Accelerator(2), and nothing else changes. MAX’s own models shard their layers with the same col_parallel and row_parallel helpers; Gemma 4’s attention, for example, uses the column-then-row layout you built here. max serve runs such a model across GPUs: pass --devices gpu:0,1,2 to use three.

Simulation answers is it correct? cheaply. Whether a layout is fast depends on the hardware: an all-reduce over NVLink costs microseconds, over PCIe or a network much more, and for a model this small on real GPUs the communication would cost more than it saves. The handbook’s distributed inference and parallelism strategies pages discuss when each layout pays off.

Exercises

  1. Run on a 9-device mesh, one query head per device. What happens to the key/value heads, which only split three ways?
  2. Shard the embedding by vocabulary (rows) and the LM head to match. Which collectives does MAX add, and how much memory does device 0 save?
  3. Add a dp axis for a 2 × 3 mesh and run a batch of two prompts with the batch split along dp. Check both rows against the single-device model.

Next

Chapter 3 goes below the operations: you write a kernel in Mojo and call it from your model.

3. Custom ops in Mojo

MAX’s operations are written in Mojo, and you can write your own the same way. This chapter writes a kernel in a notebook cell, calls it from your model, tests and times it, and then looks underneath compile at the graph API that MAX builds models with.

SectionYou learn
3.1 Your first custom opWriting a fused SwiGLU kernel in a %%mojo cell, calling it with F.custom, and measuring whether it helps.
3.2 Under the hood: the graph APIWhat compile builds, max.graph, and MAX’s weight loaders.

To go deeper into GPU programming in Mojo, work through the Mojo GPU puzzles. The handbook’s kernel optimization chapter explains why fused kernels matter.

3.1 Your first custom op

Open in Colab Open in Kaggle View on GitHub

MAX’s operations, from F.silu to flash attention, run kernels written in Mojo. You can write your own the same way, compile it from a notebook cell, and call it from Python like any other MAX operation. This section writes a fused SwiGLU kernel, the silu(gate) * up inside SmolLM2’s MLP from Section 1.4, tests it, runs it inside your full model, and measures whether it helps.

Show the setup 25 lines
import logging
import time
import warnings
from pathlib import Path

import numpy as np
import plotly.graph_objects as go
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.graph import TensorType
from max.pipelines import AutoTokenizer
from max.tensor import Tensor, default_device, default_dtype
from plotly.subplots import make_subplots

from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.layers import MLP
from minillm.model import load_model
from minillm.weights import load_checkpoint

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU

Write the kernel

import mojo.notebook adds a %%mojo cell magic to Jupyter. A cell that starts with %%mojo package -o kernels.mojoc compiles its Mojo code into a package that MAX can load.

import mojo.notebook

The simplest kind of custom op is elementwise: each output element depends only on the input elements at the same position. You write a struct with one static elementwise method that maps input vectors to an output vector, and MAX handles everything else: splitting the work across threads, loading and storing memory, and choosing CPU or GPU code.

  • @extensibility.register("swiglu") names the op; Python calls it by this name.
  • ElementwiseBinaryOp declares the shape of the method: two inputs, one output, all of one dtype.
  • SIMD[dtype, width] is a vector of width values. MAX picks the width that fills the hardware’s vector registers, so one call processes several elements at once.
  • comptime assert runs at compile time and rejects integer dtypes, for which exp is undefined.

SiLU is \(x \cdot \sigma(x) = x / (1 + e^{-x})\):

%%mojo package -o kernels.mojoc
import extensibility
from extensibility import ElementwiseBinaryOp
from std.math import exp


@extensibility.register("swiglu")
struct SwiGLU(ElementwiseBinaryOp):
    @staticmethod
    def elementwise[
        dtype: DType, width: SIMDLength
    ](gate: SIMD[dtype, width], up: SIMD[dtype, width]) -> SIMD[dtype, width]:
        comptime assert dtype.is_floating_point(), "SwiGLU needs floats"
        return gate / (1 + exp(-gate)) * up

Call it from Python

F.custom calls a custom op by name on eager tensors. It needs the device to run on, the inputs, the type of each output (here, the same as gate’s), and the package to load the kernel from:

KERNELS = Path("kernels.mojoc").resolve()


def swiglu(gate: Tensor, up: Tensor) -> Tensor:
    return F.custom(
        "swiglu",
        device=gate.device,
        values=[gate, up],
        out_types=[gate.type],
        custom_extensions=KERNELS,
    )[0]


rng = np.random.default_rng(0)
g = rng.standard_normal((4, 8)).astype(np.float32)
u = rng.standard_normal((4, 8)).astype(np.float32)
result = swiglu(Tensor(g, device=DEVICE), Tensor(u, device=DEVICE))
print(result.to_numpy()[0, :4])
print(F.silu(Tensor(g, device=DEVICE)).to_numpy()[0, :4] * u[0, :4])
[-0.01063813 -0.03336792  0.09002377  0.01961606]
[-0.01063813 -0.03336792  0.09002377  0.01961606]

Test it

A kernel needs the same tests as any other code, against a reference you trust. Check shapes that don’t divide evenly into vectors, a single element, and inputs extreme enough to overflow exp:

def reference(g: np.ndarray, u: np.ndarray) -> np.ndarray:
    return g / (1 + np.exp(-g.astype(np.float64))) * u


cases = {
    "1 element": (1,),
    "odd length": (1, 7, 1001),
    "model size": (2, 32, 1536),
}
for name, shape in cases.items():
    g = rng.standard_normal(shape).astype(np.float32) * 4
    u = rng.standard_normal(shape).astype(np.float32)
    out = swiglu(Tensor(g, device=DEVICE), Tensor(u, device=DEVICE)).to_numpy()
    print(f"{name:12} max error {np.abs(out - reference(g, u)).max():.1e}")
    assert np.allclose(out, reference(g, u), rtol=1e-4, atol=1e-5)

extremes = np.array([-100.0, -20.0, 0.0, 20.0, 100.0], np.float32)
out = swiglu(
    Tensor(extremes, device=DEVICE),
    Tensor(np.ones(5, np.float32), device=DEVICE),
)
print("extremes:", out.to_numpy())
assert np.isfinite(out.to_numpy()).all()
1 element    max error 1.0e-08
odd length   max error 1.8e-06
model size   max error 3.0e-06
extremes: [-0.0000000e+00 -4.1223068e-08  0.0000000e+00  2.0000000e+01
  1.0000000e+02]

Sweep the input to see the whole function and its error. With up set to ones, the op is SiLU itself:

sweep = np.linspace(-12, 12, 481, dtype=np.float32)
ones = np.ones_like(sweep)
ours = swiglu(
    Tensor(sweep, device=DEVICE), Tensor(ones, device=DEVICE)
).to_numpy()
error = np.abs(ours - reference(sweep, ones))
Show the code 34 lines
fig = make_subplots(
    rows=2,
    cols=1,
    shared_xaxes=True,
    vertical_spacing=0.08,
    row_heights=[0.6, 0.4],
)
fig.add_trace(
    go.Scatter(
        x=sweep,
        y=ours,
        name="your kernel",
        line_color=viz.BLUE,
        hovertemplate="silu(%{x:.2f}) = %{y:.4f}<extra></extra>",
    ),
    row=1,
    col=1,
)
fig.add_trace(
    go.Scatter(
        x=sweep,
        y=np.maximum(error, 1e-12),
        name="error against float64 NumPy",
        line_color=viz.ORANGE,
        hovertemplate="%{x:.2f}: error %{y:.1e}<extra></extra>",
    ),
    row=2,
    col=1,
)
fig.update_yaxes(title_text="output", row=1, col=1)
fig.update_yaxes(title_text="absolute error", type="log", dtick=1, row=2, col=1)
fig.update_xaxes(title_text="gate", row=2, col=1)
fig.update_layout(title="The swiglu op with up = 1", height=520)
viz.show(fig)

The error stays at the scale of float32 rounding, about \(10^{-7}\) of the output or less, across the whole range.

At \(-100\), \(e^{100}\) overflows float32 to infinity, and \(-100 / \infty\) is \(-0\), the correct limit. A different but equivalent formula, such as gate * exp(gate) / (1 + exp(gate)), would compute \(\infty / \infty\) and return NaN at \(+100\). Tests with extreme inputs catch this kind of bug.

Use it in your model

FusedMLP is the MLP from Section 1.4 with the custom op in place of F.silu(...) *:

class FusedMLP(MLP):
    def forward(self, x: Tensor) -> Tensor:
        return self.down_proj(swiglu(self.gate_proj(x), self.up_proj(x)))


config = LlamaConfig.from_pretrained()
weights = load_checkpoint()
model = load_model(config, weights, DEVICE)
for layer in model.model.layers:
    with F.lazy(), default_device(DEVICE), default_dtype(DType.float32):
        fused = FusedMLP(config)
    fused.load_state_dict(dict(layer.mlp.parameters))
    layer.mlp = fused

Compile the model as before, with one addition: custom_extensions tells the compiler where to find the kernel. Then compare the logits with the model from Section 1.6:

token_type = TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
fused_model = model.compile(token_type, custom_extensions=[KERNELS])
plain_model = load_model(config, weights, DEVICE).compile(token_type)

tokenizer = AutoTokenizer(MODEL_ID)
ids = Tensor(
    np.array([tokenizer(["The capital of France is"])["input_ids"][0]]),
    device=DEVICE,
)
fused_logits = fused_model(ids).to_numpy()
plain_logits = plain_model(ids).to_numpy()
print(f"max difference {np.abs(fused_logits - plain_logits).max():.1e}")
print(
    "next token:", repr(tokenizer.decode([int(fused_logits[0, -1].argmax())]))
)
assert (
    np.abs(fused_logits - plain_logits).max() < 1e-2
    and fused_logits[0, -1].argmax() == plain_logits[0, -1].argmax()
)
max difference 4.9e-05
next token: ' Paris'

Is it faster?

Time a compiled MLP block with and without the custom op, on a batch of 512 positions:

Show the code 28 lines
with default_device(DEVICE), default_dtype(DType.float32):
    plain_mlp = MLP(config)
    fused_mlp = FusedMLP(config)
fused_mlp.load_state_dict(dict(plain_mlp.parameters))

block_type = TensorType(
    DType.float32, ("batch", "seq_len", config.hidden_size), device=DEVICE
)
runs = {
    "MAX ops": plain_mlp.compile(block_type),
    "custom op": fused_mlp.compile(block_type, custom_extensions=[KERNELS]),
}
x = Tensor(
    rng.standard_normal((1, 512, config.hidden_size)).astype(np.float32),
    device=DEVICE,
)
for run in runs.values():
    for _ in range(3):
        run(x).to_numpy()
times = {name: [] for name in runs}
# Alternate the two, so a change in clock speed or load hits both alike.
for _ in range(30):
    for name, run in runs.items():
        start = time.perf_counter()
        run(x).to_numpy()
        times[name].append((time.perf_counter() - start) * 1000)
for name, ms in times.items():
    print(f"{name:12} median {np.median(ms):.2f} ms")
MAX ops      median 3.58 ms
custom op    median 3.41 ms
Show the code 21 lines
fig = go.Figure(
    [
        go.Box(
            y=ms,
            name=name,
            marker_color=color,
            boxpoints="all",
            jitter=0.4,
            pointpos=0,
            hovertemplate="%{y:.2f} ms<extra>" + name + "</extra>",
        )
        for (name, ms), color in zip(times.items(), [viz.BLUE, viz.ORANGE])
    ]
)
fig.update_layout(
    title="A compiled MLP block on 512 positions, 30 runs each",
    height=400,
    yaxis=dict(title="ms", rangemode="tozero"),
    showlegend=False,
)
viz.show(fig)

In the runs behind this page, on an Apple-silicon Mac with a Metal GPU, the two boxes overlap almost completely, the medians land within about 10% of each other, and which one is faster changes from run to run. Treat the output above as one example: your hardware will give different numbers, so measure on the hardware you serve on. There is little to gain because the graph compiler already fuses elementwise operations into their neighbors: the kernel trace in Section 4.4 shows silu and the multiply applied in the epilogue of a matmul, so the plain version never writes the activation to memory either. Most of the time goes to the three matrix multiplications.

Write a custom op when MAX doesn’t have the operation you need, or when a better algorithm needs control that a chain of separate operations can’t express. Flash attention (Section 1.5) is the classic case: it never materializes the attention matrix, which no combination of separate operations can avoid. Elementwise ops written as above stay fusible, so the compiler can still merge them with their neighbors, and the same kernel runs on a GPU with no changes.

Kernels that need control over threads and memory, such as a matmul or attention, implement an execute method instead, written with Mojo’s GPU programming model. MAX’s guide to building custom ops shows the pattern, and the Mojo GPU puzzles teach GPU programming in Mojo from first principles. The handbook’s kernel optimization tools page compares Mojo with CUDA and Triton, and GPU architecture fundamentals explains what a kernel has to work with.

Exercises

  1. Write a gelu_tanh custom op as an ElementwiseUnaryOp (unary elementwise ops take one input) and test it against F.gelu with approximate="tanh".
  2. Add a float16 case to the tests. How large is the error, and why?
  3. Fuse more: write an op that takes gate, up, and a scale, and returns silu(gate) * up * scale. Does the timing change?

Next

Section 3.2 looks underneath compile: the graph MAX builds from your module, and the graph API you can use to build one directly.

3.2 Under the hood: the graph API

Open in Colab Open in Kaggle View on GitHub

Since Section 1.2 you have called compile on modules and gotten a fast callable back. This section opens that box. compile traces your module into a graph, a complete description of the computation, and MAX’s compiler turns the graph into optimized code. You can also build graphs directly with max.graph, which is how many of MAX’s serving pipelines build their models, and load checkpoints with MAX’s production weight loader.

Show the setup 25 lines
import logging
import re
from pathlib import Path

import numpy as np
import plotly.graph_objects as go
from huggingface_hub import hf_hub_download
from max.driver import CPU, Buffer
from max.dtype import DType
from max.engine import InferenceSession
from max.graph import DeviceRef, Graph, TensorType, Weight, ops
from max.graph.weights import load_weights
from max.tensor import Tensor, default_device, default_dtype

from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.layers import MLP
from minillm.weights import load_checkpoint, select

logging.getLogger("max").setLevel(logging.ERROR)
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU

config = LlamaConfig.from_pretrained()
weights = load_checkpoint()

What compile builds

trace runs the first half of compile: it calls forward with symbolic inputs and records every operation into a graph, without compiling it. Here is SmolLM2’s MLP from Section 1.4:

with default_device(DEVICE), default_dtype(DType.float32):
    mlp = MLP(config)
mlp.load_state_dict(select(weights, "model.layers.0.mlp."))

staged = mlp.trace(
    TensorType(
        DType.float32,
        ("seq_len", config.hidden_size),
        device=DeviceRef.from_device(DEVICE),
    )
)
print("\n".join(str(staged.graph).splitlines()[:12]))
mo.graph @MLP<seq_len>(%arg0: !mo.tensor<[seq_len, 576], f32, gpu:0>) -> !mo.tensor<[seq_len, 576], f32, gpu:0> attributes {_kernel_library_paths = [], argument_names = ["input0"], result_names = ["output0"]} {
  %0 = mo.chain.create()
  %1 = mo.constant.external [name = "gate_proj.weight", align = 1, device = #M.device_ref<"cpu", 0>] : !mo.tensor<[1536, 576], f32>
  %result, %outChain = mo.transfer[%0] %1 : !mo.tensor<[1536, 576], f32> to <"gpu", 0>
  %2 = mo.constant <#M.dense_array<1, 0> : tensor<2xsi64>> : !mo.tensor<[2], si64>
  %3 = mo.transpose(%result, %2) : (!mo.tensor<[1536, 576], f32, gpu:0>, !mo.tensor<[2], si64>) -> !mo.tensor<[576, 1536], f32, gpu:0>
  %4 = rmo.matmul(%arg0, %3) : (!mo.tensor<[seq_len, 576], f32, gpu:0>, !mo.tensor<[576, 1536], f32, gpu:0>) -> !mo.tensor<[seq_len, 1536], f32, gpu:0>
  %5 = mo.constant <#M.dense_array<0.000000e+00> : tensor<f32>> : !mo.tensor<[], f32, gpu:0>
  %6 = rmo.add(%4, %5) : (!mo.tensor<[seq_len, 1536], f32, gpu:0>, !mo.tensor<[], f32, gpu:0>) -> !mo.tensor<[seq_len, 1536], f32, gpu:0>
  %7 = mo.silu(%6) : !mo.tensor<[seq_len, 1536], f32, gpu:0>
  %8 = mo.constant.external [name = "up_proj.weight", align = 1, device = #M.device_ref<"cpu", 0>] : !mo.tensor<[1536, 576], f32>
  %result_0, %outChain_1 = mo.transfer[%0] %8 : !mo.tensor<[1536, 576], f32> to <"gpu", 0>

The text is a dataflow graph: each line defines one value (%4) from earlier ones. Parse it and draw it, with each operation placed one column after the latest value it reads. Hover over a node to see its line of IR:

Show kind 85 lines
LINE = re.compile(
    r"^\s*(%\w+)(?:, %\w+)* = ([\w.]+)(.*)$"
)  # some ops have two results
nodes, edges = {"%arg0": ("input", "input0: [seq_len, 576]")}, []
for line in str(staged.graph).splitlines():
    match = LINE.match(line)
    if not match or match.group(2) in ("mo.chain.create", "mo.constant"):
        continue  # skip bookkeeping and small constants
    value, op, rest = match.groups()
    weight = re.search(r'name = "([^"]+)"', rest)
    label = weight.group(1) if weight else op.split(".")[-1]
    nodes[value] = (label, line.strip())
    edges += [(src, value) for src in re.findall(r"%\w+", rest) if src in nodes]
output = re.search(r"mo.output (%\w+)", str(staged.graph)).group(1)
nodes["output"] = ("output", "output0: [seq_len, 576]")
edges.append((output, "output"))

depth = {}
for value in nodes:  # lines are in dataflow order
    depth[value] = max(
        (depth[src] + 1 for src, dst in edges if dst == value), default=0
    )
column = {}
position = {}
for value in nodes:
    row = column.get(depth[value], 0)
    column[depth[value]] = row + 1
    position[value] = (depth[value], -row)


def kind(label: str) -> str:
    if label in ("input", "output"):
        return "input or output"
    if label.endswith(".weight"):
        return "weight"
    if label == "transfer":
        return "copy to device"
    return "matmul" if label == "matmul" else "other operation"


colors = {
    "input or output": viz.VIOLET,
    "weight": viz.ORANGE,
    "copy to device": viz.YELLOW,
    "matmul": viz.BLUE,
    "other operation": viz.AQUA,
}
edge_x, edge_y = [], []
for src, dst in edges:
    edge_x += [position[src][0], position[dst][0], None]
    edge_y += [position[src][1], position[dst][1], None]
fig = go.Figure(
    go.Scatter(
        x=edge_x,
        y=edge_y,
        mode="lines",
        line=dict(color=viz.GRID, width=2),
        hoverinfo="skip",
        showlegend=False,
    )
)
for name, color in colors.items():
    members = [v for v in nodes if kind(nodes[v][0]) == name]
    fig.add_trace(
        go.Scatter(
            x=[position[v][0] for v in members],
            y=[position[v][1] for v in members],
            mode="markers+text",
            name=name,
            text=[nodes[v][0] for v in members],
            textposition="top center",
            marker=dict(
                size=18, color=color, line=dict(color=viz.SURFACE, width=2)
            ),
            customdata=[nodes[v][1][:110] for v in members],
            hovertemplate="%{customdata}<extra></extra>",
        )
    )
fig.update_layout(
    title="The MLP's graph, as compile sees it",
    height=480,
    xaxis=dict(visible=False, range=[-0.8, max(depth.values()) + 0.6]),
    yaxis=dict(visible=False, range=[-max(column.values()) + 0.5, 0.6]),
)
viz.show(fig)

The graph is MAX’s intermediate representation (in MLIR). A few things to read in it:

  • The input type is [seq_len, 576]: seq_len stays symbolic, which is why one compiled graph serves every sequence length.
  • Each weight is a mo.constant.external with a name such as gate_proj.weight. The graph refers to weights by name and gets their values when it is loaded, so the same graph works with any checkpoint.
  • On a GPU, each weight also passes through a mo.transfer that copies it from host memory to the device (yellow in the diagram); on a CPU there is nothing to copy.
  • Each projection’s rmo.matmul is followed by an rmo.add of a constant 0.0. MAX’s Linear built with bias=False stores its bias as the number 0 and still adds it. It costs nothing at run time: the kernel trace in Section 4.4 shows no separate add.
  • Every line is one operation (rmo.matmul, mo.silu, …) with fully known types. That is what lets the compiler fuse operations, plan memory for the whole computation, and generate code for the target device ahead of time.

Build a graph yourself

max.graph builds graphs directly. A Graph takes a name, the input types, and a forward function that combines max.graph.ops operations. Weights are declared as Weight objects with a name, dtype, shape, and device:

H, I = config.hidden_size, config.intermediate_size


def mlp_forward(x):
    def weight(name: str, shape: list[int]) -> Weight:
        return Weight(
            name, DType.float32, shape, device=DeviceRef.from_device(DEVICE)
        )

    gate = x @ weight("gate_proj.weight", [I, H]).T
    up = x @ weight("up_proj.weight", [I, H]).T
    return (ops.silu(gate) * up) @ weight("down_proj.weight", [H, I]).T


graph = Graph(
    "mlp",
    forward=mlp_forward,
    input_types=[
        TensorType(
            DType.float32, ("seq_len", H), device=DeviceRef.from_device(DEVICE)
        )
    ],
)

An InferenceSession compiles graphs for a set of devices. load compiles this one and binds each named weight from a registry; execute runs it on Buffers, the driver’s plain device memory:

session = InferenceSession(devices=[DEVICE])
compiled_mlp = session.load(
    graph, weights_registry=select(weights, "model.layers.0.mlp.")
)

x = np.random.default_rng(0).standard_normal((5, H)).astype(np.float32)
(out,) = compiled_mlp.execute(Buffer.from_numpy(x).to(DEVICE))
expected = mlp(Tensor(x, device=DEVICE)).to_numpy()
scale = np.abs(expected).max()
relative = np.abs(out.to_numpy() - expected).max() / scale
print(
    f"max difference from your module, relative to the largest output: {relative:.1e}"
)
assert relative < 1e-5
max difference from your module, relative to the largest output: 4.8e-07

The hand-built graph and your module agree because Module.compile builds this same graph for you, loads it in a session shared by the whole process, and wraps the result so it takes and returns Tensors.

Custom ops work in graphs too, through ops.custom. Compile the kernel from Section 3.1 again and use it in place of silu * up:

import mojo.notebook
Show SwiGLU 14 lines
%%mojo package -o kernels.mojoc
import extensibility
from extensibility import ElementwiseBinaryOp
from std.math import exp


@extensibility.register("swiglu")
struct SwiGLU(ElementwiseBinaryOp):
    @staticmethod
    def elementwise[
        dtype: DType, width: SIMDLength
    ](gate: SIMD[dtype, width], up: SIMD[dtype, width]) -> SIMD[dtype, width]:
        comptime assert dtype.is_floating_point(), "SwiGLU needs floats"
        return gate / (1 + exp(-gate)) * up
Show fused_forward 34 lines
def fused_forward(x):
    def weight(name: str, shape: list[int]) -> Weight:
        return Weight(
            name, DType.float32, shape, device=DeviceRef.from_device(DEVICE)
        )

    gate = x @ weight("gate_proj.weight", [I, H]).T
    up = x @ weight("up_proj.weight", [I, H]).T
    (h,) = ops.custom(
        "swiglu",
        device=DeviceRef.from_device(DEVICE),
        values=[gate, up],
        out_types=[gate.type],
    )
    return h.tensor @ weight("down_proj.weight", [H, I]).T


fused_graph = Graph(
    "fused_mlp",
    forward=fused_forward,
    input_types=[
        TensorType(
            DType.float32, ("seq_len", H), device=DeviceRef.from_device(DEVICE)
        )
    ],
    custom_extensions=[Path("kernels.mojoc")],
)
fused_mlp = session.load(
    fused_graph, weights_registry=select(weights, "model.layers.0.mlp.")
)
(fused_out,) = fused_mlp.execute(Buffer.from_numpy(x).to(DEVICE))
relative = np.abs(fused_out.to_numpy() - expected).max() / scale
print(f"max difference, relative to the largest output: {relative:.1e}")
assert relative < 1e-5
max difference, relative to the largest output: 4.8e-07

Production weight loading

The loader from Section 1.3 reads one safetensors file into memory as float32. MAX’s own loader, max.graph.weights.load_weights, reads safetensors (including checkpoints split across many files) and GGUF. It maps files into memory instead of reading them, so a tensor is read only when it’s used, and it keeps each tensor’s original dtype and, for GGUF, its quantization encoding. Tensors are reached by attribute path:

path = Path(hf_hub_download(MODEL_ID, "model.safetensors"))
checkpoint = load_weights([path])
gate = checkpoint.model.layers[0].mlp.gate_proj.weight
print(gate.name)
weight = gate.allocate()
print(weight.shape, weight.dtype)
model.layers.0.mlp.gate_proj.weight
[Dim(1536), Dim(576)] DType.bfloat16

allocate returns a Weight like the ones in your hand-built graph, with the checkpoint’s name, shape, and dtype filled in. A serving pipeline builds its graph from these, so the graph matches the checkpoint exactly, and the loader supplies the data when the graph is loaded.

Why it matters

Because the whole computation is known before it runs, MAX can:

  • fuse operations and plan memory across the entire model instead of one operation at a time;
  • compile once, ahead of time, and cache the result (the reason max serve starts faster the second time);
  • on GPUs, record a decode step’s kernel launches and replay them with almost no CPU overhead (device graph capture);
  • run the same graph on CPUs and on NVIDIA, AMD, and Apple GPUs.

max.nn modules with compile give you all of this while you write ordinary Python. Use max.graph directly when you need control over exactly what goes into a graph, as MAX’s serving pipelines do. MAX’s docs describe the graph API in full; the handbook’s kernel optimization page explains why whole-graph compilation and fusion matter for LLMs.

Exercises

  1. Print the full traced graph of one DecoderLayer from Section 1.6. Find the flash attention call and the residual additions.
  2. Build the MLP graph with a static input shape, [5, 576], instead of seq_len. What happens when you execute it with 6 positions?
  3. Load layer 0’s MLP weights through load_weights instead of load_checkpoint. What has to change for the float32 graph?

Next

Chapter 4 turns to correctness and speed: testing layers against references, verifying a whole model, and benchmarking and profiling.

4. Testing and benchmarking

A model has to be correct before it is worth making fast. This chapter applies the tools MAX uses to test and measure its own models to yours.

SectionYou learn
4.1 Testing layers against referencesTesting a decoder layer on many shapes against Hugging Face’s PyTorch Llama, with the pattern MAX’s layer benchmarks use.
4.2 Verifying a model’s logitsComparing a whole model’s logits with a reference, using the metrics MAX’s verification tools use.
4.3 Benchmarking kernels and layersA benchmark harness, and why decode is limited by memory and prefill by arithmetic.
4.4 ProfilingWhere the time goes, from a whole generation run down to the kernels and the parts of your model.

Related handbook pages: LLM inference metrics and performance benchmarks.

4.1 Testing layers against references

Open in Colab Open in Kaggle View on GitHub

Chapter 1 checked each layer once, on one input. Real tests check many shapes against an independent reference, with metrics that tell a rounding difference from a bug. This section builds the pattern MAX uses for its own layers, in max/python/layer_benchmarks, and applies it to your decoder layer, with Hugging Face’s PyTorch Llama as the reference.

Show the setup 29 lines
import logging
import warnings

import numpy as np
import plotly.graph_objects as go
import torch
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.graph import TensorType
from max.tensor import Tensor, default_device, default_dtype
from transformers import AutoConfig
from transformers.models.llama.modeling_llama import (
    LlamaDecoderLayer,
    LlamaRotaryEmbedding,
)

from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.model import DecoderLayer
from minillm.weights import load_checkpoint, select

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU

config = LlamaConfig.from_pretrained()
layer0 = select(load_checkpoint(), "model.layers.0.")

How to compare outputs

One number is not enough. MAX’s test harness reports three:

  • The maximum absolute error catches any single element that is far off.
  • The mean absolute error is the typical error, which tells a few outliers from a systematic drift.
  • The cosine distance, \(1 - \cos\theta\) between the flattened outputs, is near 0 when the outputs point the same way even if their scale differs slightly, and large for a structural bug such as a wrong head layout.

A test passes when the outputs are allclose, within an absolute and a relative tolerance, and the cosine distance is below a threshold. The defaults below, 1e-3 for both tolerances and 1e-5 for the cosine distance, are tighter than MAX’s (1e-2 and 1e-3) because every test here runs in float32; MAX’s bfloat16 layer tests loosen them further.

Show CorrectnessResult, compare_outputs 48 lines
from __future__ import annotations

from dataclasses import dataclass

import numpy as np


@dataclass
class CorrectnessResult:
    """How far an output is from its reference, as MAX's layer tests report it."""

    label: str
    mae: float
    max_abs_err: float
    cos_dist: float
    passed: bool

    def __str__(self) -> str:
        status = "pass" if self.passed else "FAIL"
        return (
            f"{self.label:16} mae {self.mae:.1e}  max {self.max_abs_err:.1e}  "
            f"cos dist {self.cos_dist:.1e}  {status}"
        )


def compare_outputs(
    actual: np.ndarray,
    expected: np.ndarray,
    label: str = "",
    atol: float = 1e-3,
    rtol: float = 1e-3,
    cos_threshold: float = 1e-5,
) -> CorrectnessResult:
    """Compares ``actual`` with ``expected``; passes when they are allclose
    and their cosine distance is below ``cos_threshold``."""
    a = actual.astype(np.float64).ravel()
    e = expected.astype(np.float64).ravel()
    cos_dist = 1.0 - float(
        a @ e / (np.linalg.norm(a) * np.linalg.norm(e) + 1e-12)
    )
    allclose = bool(np.allclose(actual, expected, atol=atol, rtol=rtol))
    return CorrectnessResult(
        label=label,
        mae=float(np.mean(np.abs(a - e))),
        max_abs_err=float(np.max(np.abs(a - e))),
        cos_dist=cos_dist,
        passed=allclose and cos_dist < cos_threshold,
    )

Saved to minillm/testing.py.

The reference

Hugging Face’s LlamaDecoderLayer is an independent implementation of the same layer, in PyTorch. Its parameter names match yours, so the same checkpoint entries load into both. "eager" selects its plain attention code instead of a fused kernel, so the reference stays simple.

hf_config = AutoConfig.from_pretrained(MODEL_ID, attn_implementation="eager")
reference = LlamaDecoderLayer(hf_config, layer_idx=0).float().eval()
reference.load_state_dict(
    {name: torch.from_numpy(w.copy()) for name, w in layer0.items()}
)
rotary = LlamaRotaryEmbedding(hf_config)


def reference_layer(x: np.ndarray) -> np.ndarray:
    batch, seq, _ = x.shape
    hidden = torch.from_numpy(x)
    positions = torch.arange(seq).expand(batch, seq)
    mask = torch.full((seq, seq), float("-inf")).triu(1)[None, None]
    with torch.no_grad():
        out = reference(
            hidden,
            attention_mask=mask,
            position_ids=positions,
            position_embeddings=rotary(hidden, positions),
        )
    return out.numpy()

Compile once, test many shapes

MAX’s harness separates static parameters, which fix the graph (here, the layer’s sizes), from dynamic ones, which vary per run (batch size and sequence length). Compile once with symbolic dimensions, then run every shape through the same compiled layer. The shapes cover the edges: a single token (a decode step), odd lengths, a batch, and a long prefill.

Show compiled_layer, check 34 lines
def compiled_layer(layer_class=DecoderLayer, weights=layer0, cfg=config):
    with F.lazy(), default_device(DEVICE), default_dtype(DType.float32):
        layer = layer_class(cfg)
    layer.load_state_dict(weights)
    return layer.compile(
        TensorType(
            DType.float32, ("batch", "seq_len", cfg.hidden_size), device=DEVICE
        )
    )


SHAPES = [(1, 1), (1, 7), (2, 33), (1, 512)]


def check(run, shapes=SHAPES) -> list:
    rng = np.random.default_rng(0)
    results = []
    for batch, seq in shapes:
        x = rng.standard_normal((batch, seq, config.hidden_size)).astype(
            np.float32
        )
        actual = run(Tensor(x, device=DEVICE)).to_numpy()
        results.append(
            compare_outputs(
                actual, reference_layer(x), label=f"batch {batch}, seq {seq}"
            )
        )
    return results


correct = check(compiled_layer())
for result in correct:
    print(result)
assert all(r.passed for r in correct)
batch 1, seq 1   mae 2.7e-07  max 2.9e-06  cos dist 2.4e-14  pass
batch 1, seq 7   mae 6.5e-07  max 1.7e-05  cos dist 1.9e-13  pass
batch 2, seq 33  mae 7.0e-07  max 2.1e-05  cos dist 1.9e-13  pass
batch 1, seq 512 mae 7.1e-07  max 3.8e-05  cos dist 2.0e-13  pass

Every shape passes, with maximum errors of 4e-5 or less and cosine distances near zero.

A test must be able to fail

A test that never fails proves nothing, so check that this one catches a real bug. A plausible one is a config read wrong, with the base of RoPE (Section 1.4) at Llama 2’s 10000 instead of SmolLM2’s 100000:

from dataclasses import replace

buggy = compiled_layer(cfg=replace(config, rope_theta=10000.0))
buggy_results = check(buggy)
for result in buggy_results:
    print(result)
assert not all(r.passed for r in buggy_results)
batch 1, seq 1   mae 2.7e-07  max 2.9e-06  cos dist 2.4e-14  pass
batch 1, seq 7   mae 4.9e-04  max 6.6e-03  cos dist 1.2e-07  FAIL
batch 2, seq 33  mae 2.2e-03  max 1.3e-01  cos dist 2.3e-06  FAIL
batch 1, seq 512 mae 5.0e-03  max 2.3e-01  cos dist 1.1e-05  FAIL
Show the code 45 lines
ATOL, COS_THRESHOLD = 1e-3, 1e-5
fig = go.Figure(
    go.Scatter(
        x=[1e-7, ATOL, ATOL, 1e-7],
        y=[1e-15, 1e-15, COS_THRESHOLD, COS_THRESHOLD],
        fill="toself",
        fillcolor="rgba(27,175,122,0.15)",
        line=dict(color=viz.AQUA, width=1, dash="dot"),
        name="pass region (about)",
        hoverinfo="skip",
    )
)
for name, results, color in [
    ("correct layer", correct, viz.BLUE),
    ("rope_theta = 10000", buggy_results, viz.ORANGE),
]:
    fig.add_trace(
        go.Scatter(
            x=[r.max_abs_err for r in results],
            y=[max(r.cos_dist, 1e-15) for r in results],
            mode="markers+text",
            name=name,
            text=[
                r.label.replace("batch ", "b").replace(", seq ", "/s")
                for r in results
            ],
            textposition="middle right",
            marker=dict(
                color=color,
                size=11,
                symbol=["circle" if r.passed else "x" for r in results],
            ),
            hovertemplate="%{text}: max error %{x:.1e}, cos dist %{y:.1e}"
            + "<extra>"
            + name
            + "</extra>",
        )
    )
fig.update_layout(
    title="Each shape's max error against its cosine distance",
    height=440,
    xaxis=dict(title="max |error| (log scale)", type="log", dtick=1),
    yaxis=dict(title="cosine distance (log scale)", type="log", dtick=2),
)
viz.show(fig)

Each point is one shape, labeled batch/sequence length, and an x marks a failure. The shaded corner is where both checks pass; its right edge is only approximate, because allclose also allows a relative error of 1e-3 of each expected value.

The single-token case still passes: at position 0, RoPE doesn’t rotate at all, so the bug can’t show there. Every longer input fails. The maximum error grows by two to four orders of magnitude and breaks the allclose check. The cosine distance grows by six to eight orders of magnitude, but it crosses its 1e-5 threshold only on the 512-token input. The two checks catch different bugs, which is why the test uses both, and why the shape list must include sequences, not just one token.

Where in the sequence does the bug show? Run the 512-token input through both layers and plot the largest error at each position:

x = (
    np.random.default_rng(1)
    .standard_normal((1, 512, config.hidden_size))
    .astype(np.float32)
)
expected = reference_layer(x)[0]
per_position = {
    name: np.abs(run(Tensor(x, device=DEVICE)).to_numpy()[0] - expected).max(-1)
    for name, run in [
        ("correct layer", compiled_layer()),
        ("rope_theta = 10000", buggy),
    ]
}
Show the code 22 lines
fig = go.Figure(
    [
        go.Scatter(
            x=np.arange(512),
            y=np.maximum(err, 1e-9),
            name=name,
            mode="lines",
            line_color=color,
            hovertemplate="position %{x}: %{y:.1e}<extra>" + name + "</extra>",
        )
        for (name, err), color in zip(
            per_position.items(), [viz.BLUE, viz.ORANGE]
        )
    ]
)
fig.update_layout(
    title="Largest error at each position of a 512-token input",
    height=400,
    xaxis_title="position",
    yaxis=dict(title="max |error| (log scale)", type="log", dtick=1),
)
viz.show(fig)

At position 0 the two layers agree, because RoPE doesn’t rotate there. Within a few positions the buggy layer’s error is three to four orders of magnitude larger, and it stays there for the whole sequence, while the correct layer’s error stays near 1e-5. A test that only ran one token would have passed both.

As a pytest file

In a repository, these checks belong in a test file that runs on every change. %%writefile saves a cell as a file. The test compiles the layer once per module, as MAX’s own tests do, and checks every shape:

Show the code 45 lines
%%writefile test_decoder_layer.py
import numpy as np
import pytest
import torch
from max.dtype import DType
from max.graph import TensorType
from max.tensor import Tensor, default_device, default_dtype
from transformers import AutoConfig
from transformers.models.llama.modeling_llama import LlamaDecoderLayer, LlamaRotaryEmbedding

from minillm.config import MODEL_ID, LlamaConfig
from minillm.device import pick_device
from minillm.model import DecoderLayer
from minillm.testing import compare_outputs
from minillm.weights import load_checkpoint, select

CONFIG = LlamaConfig.from_pretrained()


@pytest.fixture(scope="module")
def layers():
    weights = select(load_checkpoint(), "model.layers.0.")
    with default_device(pick_device()), default_dtype(DType.float32):
        ours = DecoderLayer(CONFIG)
    ours.load_state_dict(weights)
    compiled = ours.compile(
        TensorType(DType.float32, ("batch", "seq_len", CONFIG.hidden_size), device=pick_device())
    )
    hf_config = AutoConfig.from_pretrained(MODEL_ID, attn_implementation="eager")
    reference = LlamaDecoderLayer(hf_config, layer_idx=0).float().eval()
    reference.load_state_dict({k: torch.from_numpy(v.copy()) for k, v in weights.items()})
    return compiled, reference, LlamaRotaryEmbedding(hf_config)


@pytest.mark.parametrize("batch, seq", [(1, 1), (1, 7), (2, 33)])
def test_decoder_layer_matches_reference(layers, batch, seq):
    compiled, reference, rotary = layers
    x = np.random.default_rng(seq).standard_normal((batch, seq, CONFIG.hidden_size)).astype(np.float32)
    hidden, positions = torch.from_numpy(x), torch.arange(seq).expand(batch, seq)
    mask = torch.full((seq, seq), float("-inf")).triu(1)[None, None]
    with torch.no_grad():
        expected = reference(hidden, attention_mask=mask, position_ids=positions,
                             position_embeddings=rotary(hidden, positions)).numpy()
    result = compare_outputs(compiled(Tensor(x, device=pick_device())).to_numpy(), expected)
    assert result.passed, result
Overwriting test_decoder_layer.py
!python -m pytest -q -p no:cacheprovider test_decoder_layer.py 2>&1 | tail -n 3
...                                                                      [100%]
3 passed in 4.76s

MAX’s own layer tests follow the same shape, with one more step: each harness also benchmarks the layer on the same shapes, so correctness and speed are measured together. Section 4.3 covers benchmarking.

Exercises

  1. Add a bfloat16 run: cast the input and the weights, and compare with the float32 reference. Which tolerances do you need, and which metric moves most?
  2. Test CachedAttention from Section 1.8: compare a prefill of 8 tokens followed by one decode step with a single 9-token pass.
  3. Plant a different bug, such as swapping gate_proj and up_proj. Which metric catches it first?

Next

Section 4.2 tests the whole model: its logits on real prompts, against the same reference.

4.2 Verifying a model’s logits

Open in Colab Open in Kaggle View on GitHub

Layer tests catch bugs in one layer. A whole model can still go wrong in ways no layer test sees: a weight loaded into the wrong layer, a tied head that isn’t tied, a mask that only fails on long prompts. So model bring-ups end with logit verification: run the same prompts through your model and a trusted reference, and compare the logits at every position.

This section verifies your SmolLM2 against Hugging Face’s PyTorch implementation, with the metrics MAX’s own verification tools use.

Show the setup 36 lines
import logging
import warnings

import numpy as np
import plotly.graph_objects as go
import torch
from max.driver import CPU
from max.dtype import DType
from max.graph import TensorType
from max.pipelines import AutoTokenizer
from max.tensor import Tensor
from transformers import AutoModelForCausalLM

from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.kv_cache import CachedLlamaForCausalLM, KVCacheDecoder
from minillm.model import load_model
from minillm.sampling import generate
from minillm.testing import compare_outputs
from minillm.weights import load_checkpoint

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU

config = LlamaConfig.from_pretrained()
weights = load_checkpoint()
tokenizer = AutoTokenizer(MODEL_ID)

ours = load_model(config, weights, DEVICE).compile(
    TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)
reference = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, torch_dtype=torch.float32
).eval()

Prompts

Verification prompts should exercise different behavior: plain prose, code, numbers, another language, and a prompt long enough to stress attention and RoPE at larger positions.

long_prompt = " ".join(["The quick brown fox jumps over the lazy dog."] * 60)
prompts = {
    "prose": "The capital of France is Paris, and the capital of Italy is",
    "code": "def fibonacci(n):\n    if n < 2:\n        return n\n    return",
    "numbers": "1, 1, 2, 3, 5, 8, 13, 21,",
    "french": "Le chat est assis sur",
    "long": long_prompt,
}


def logits_pair(text: str) -> tuple[np.ndarray, np.ndarray]:
    ids = tokenizer([text])["input_ids"][0]
    mine = ours(Tensor(np.array([ids]), device=DEVICE)).to_numpy()[0]
    with torch.no_grad():
        theirs = reference(torch.tensor([ids])).logits[0].numpy()
    return mine, theirs

Metrics for logits

Logits feed a softmax, so on top of the elementwise metrics from Section 4.1, two measures matter for generation.

The KL divergence between the next-token distributions measures how much probability mass moved. It is the standard measure for comparing two models’ predictions, and it ignores differences in logits that softmax removes, such as a constant offset. For the reference distribution \(P\) and yours, \(Q\), at one position:

\[D_{\mathrm{KL}}(P \,\Vert\, Q) = \sum_i P_i \log \frac{P_i}{Q_i}\]

It is 0 only when the two distributions are identical.

Top-1 agreement is the fraction of positions at which both models predict the same most likely token. It is all that greedy decoding sees.

Show softmax, kl_divergence 26 lines
def softmax(z: np.ndarray) -> np.ndarray:
    z = z.astype(np.float64) - z.max(-1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(-1, keepdims=True)


def kl_divergence(p_logits: np.ndarray, q_logits: np.ndarray) -> float:
    """Mean KL(P || Q) over positions, in nats."""
    p, q = softmax(p_logits), softmax(q_logits)
    return float((p * (np.log(p + 1e-30) - np.log(q + 1e-30))).sum(-1).mean())


print(
    f"{'prompt':8} {'positions':>9} {'max err':>8} {'cos dist':>9} {'KL':>9} {'top-1':>6}"
)
errors = {}
for name, text in prompts.items():
    mine, theirs = logits_pair(text)
    errors[name] = np.abs(mine - theirs).max(-1)
    r = compare_outputs(mine, theirs, atol=1e-3, rtol=1e-3)
    top1 = (mine.argmax(-1) == theirs.argmax(-1)).mean()
    print(
        f"{name:8} {len(mine):>9} {r.max_abs_err:>8.1e} {r.cos_dist:>9.1e} "
        f"{kl_divergence(theirs, mine):>9.1e} {top1:>6.0%}"
    )
    assert top1 >= 0.99 and kl_divergence(theirs, mine) < 1e-6
prompt   positions  max err  cos dist        KL  top-1
prose           13  8.2e-05   2.4e-12   3.2e-11   100%
code            18  8.3e-05   1.5e-12   4.0e-11   100%
numbers         25  1.1e-04   2.1e-12   2.2e-11   100%
french           6  1.0e-04   1.3e-12   8.6e-11   100%
long           600  1.1e-04   2.6e-12   2.5e-12   100%

A summary number can hide a trend. Plot the largest logit error at each position: if a bug in RoPE or the mask grew with position, the long prompt would show it as a rising line.

Show the code 22 lines
fig = go.Figure(
    [
        go.Scatter(
            x=np.arange(1, len(err) + 1),
            y=err,
            customdata=np.arange(len(err)),
            name=name,
            mode="lines",
            line_color=color,
            hovertemplate=name
            + ", position %{customdata}: %{y:.1e}<extra></extra>",
        )
        for (name, err), color in zip(errors.items(), viz.SERIES)
    ]
)
fig.update_layout(
    title="Largest logit error at each position, against Hugging Face",
    height=400,
    xaxis=dict(title="position + 1 (log scale)", type="log", dtick=1),
    yaxis=dict(title="max |logit difference|", type="log", dtick=1),
)
viz.show(fig)

Every prompt agrees at every position, including the 600-token one, with 100% top-1 agreement and KL divergences between about 1e-12 and 1e-10 nats. MAX verifies its own models the same way, comparing MAX’s logits with reference logits from PyTorch or vLLM by cosine distance, KL divergence, and tolerances. The model bring-up workflow guide covers the rest of bringing a new architecture to MAX.

Greedy generation

Matching logits at the prompt is necessary but not sufficient: generation feeds the model its own outputs, through your KV cache from Section 1.8. Compare 48 greedy tokens from your cached decoder with Hugging Face’s generate:

Show the code 27 lines
cached = KVCacheDecoder(
    load_model(config, weights, DEVICE, CachedLlamaForCausalLM), config, DEVICE
)

gaps = {}
for name in ["prose", "code", "numbers"]:
    ids = tokenizer([prompts[name]])["input_ids"][0]
    mine = generate(cached, ids, 48, temperature=0)
    with torch.no_grad():
        out = reference.generate(
            torch.tensor([ids]),
            max_new_tokens=48,
            do_sample=False,
            output_logits=True,
            return_dict_in_generate=True,
        )
    theirs = out.sequences[0, len(ids) :].tolist()
    gaps[name] = [
        float(np.diff(np.sort(step[0].numpy())[-2:])[0]) for step in out.logits
    ]
    same = next(
        (i for i, (a, b) in enumerate(zip(mine, theirs)) if a != b), len(mine)
    )
    print(f"{name:8} identical for {same} of {len(mine)} tokens")
    assert same >= 24, (
        "diverged early; check the gap between the top two logits"
    )
prose    identical for 48 of 48 tokens
code     identical for 48 of 48 tokens
numbers  identical for 48 of 48 tokens

At each greedy step, how far ahead was the winning token? output_logits=True returns the reference’s raw logits at every step; plot the gap between the top two, with the largest logit error from the first chart as a dotted line:

Show the code 32 lines
fig = go.Figure(
    [
        go.Scatter(
            x=np.arange(len(gap)),
            y=gap,
            name=name,
            mode="lines+markers",
            line_color=color,
            marker=dict(size=5),
            hovertemplate=name + ", step %{x}: gap %{y:.3f}<extra></extra>",
        )
        for (name, gap), color in zip(gaps.items(), viz.SERIES)
    ]
)
fig.update_layout(
    title="Gap between the top two logits at each greedy step",
    height=400,
    xaxis_title="generated token",
    yaxis=dict(title="logit gap (log scale)", type="log", dtick=1),
)
largest_error = max(err.max() for err in errors.values())
fig.add_trace(
    go.Scatter(
        x=[0, max(len(gap) for gap in gaps.values()) - 1],
        y=[largest_error] * 2,
        name=f"largest logit error ({largest_error:.0e})",
        mode="lines",
        line=dict(color=viz.INK_SECONDARY, dash="dot", width=1),
        hoverinfo="skip",
    )
)
viz.show(fig)

A rounding difference can change the winner only at a step where the gap is smaller than the error, that is, where a line dips below the dotted one. Here the smallest gap is above 0.1 and the largest error about 1e-4, so no step comes close.

If a run does diverge after many tokens, the cause is usually a near tie: at some step two tokens have almost the same logit, and a difference in the fourth or fifth decimal place, from summing in a different order, picks a different winner. From then on the two texts differ, though both are valid greedy continuations. When you see a divergence, look at the gap between the top two logits at that step: a tiny gap means rounding, a large one means a bug.

Beyond logits: accuracy evaluations

Logit verification compares two implementations. It can’t tell you whether the model is good at a task, or whether a change that alters numbers on purpose, such as quantization, costs accuracy. For that, serving teams run task evaluations such as GSM8K or MMLU against the server (Section 7.1 serves your model). The handbook’s quantization page covers how quantized models are evaluated.

Exercises

  1. Run the comparison in bfloat16 (cast the weights and cast the logits back to float32). How do KL divergence and top-1 agreement change?
  2. Find the step where two tokens’ logits are closest during a greedy run. How small is the gap?
  3. Load one layer’s weights into the wrong layer index and rerun the table. Which prompt and metric show it most clearly?

Next

Section 4.3 measures speed: how to benchmark kernels and layers so the numbers mean something.

4.3 Benchmarking kernels and layers

Open in Colab Open in Kaggle View on GitHub

A benchmark is an experiment, and most quick timings are bad experiments: they include compilation, measure the time to launch work instead of the time to finish it, or report one noisy sample. This section writes a small benchmark harness that avoids those mistakes, then uses it to show why decoding one token is limited by memory traffic and prefill by arithmetic.

Show the setup 23 lines
import logging
import time
import warnings

import numpy as np
import plotly.graph_objects as go
from max.driver import CPU
from max.dtype import DType
from max.graph import TensorType
from max.nn import Linear
from max.tensor import Tensor, default_device, default_dtype
from plotly.subplots import make_subplots

from minillm import viz
from minillm.device import pick_device
from minillm.config import LlamaConfig
from minillm.model import DecoderLayer
from minillm.weights import load_checkpoint, select

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU
config = LlamaConfig.from_pretrained()

A harness

The harness follows four rules, in the spirit of MAX’s layer benchmarks, which time on the device with events and report the median and p95:

  1. Warm up. The first calls compile kernels and fill caches, so don’t time them.
  2. Wait for the result. Work can run asynchronously, especially on a GPU. Reading the output (to_numpy()) forces it to finish.
  3. Repeat. One sample is mostly noise, so take many.
  4. Report the median and a tail percentile. Outliers pull the mean; the median shows the typical run and p90 shows how slow an unlucky one gets.
Show Timing, benchmark 38 lines
from __future__ import annotations

import time
from collections.abc import Callable
from dataclasses import dataclass

import numpy as np


@dataclass
class Timing:
    """Milliseconds per call over the measured iterations."""

    median_ms: float
    p90_ms: float
    min_ms: float

    def __str__(self) -> str:
        return f"median {self.median_ms:7.2f} ms  p90 {self.p90_ms:7.2f} ms"


def benchmark(
    fn: Callable[[], object], warmup: int = 3, iterations: int = 20
) -> Timing:
    """Times ``fn``, which must return a fully computed result (for example,
    a NumPy array read back from the device)."""
    for _ in range(warmup):
        fn()
    samples = []
    for _ in range(iterations):
        start = time.perf_counter()
        fn()
        samples.append((time.perf_counter() - start) * 1000)
    return Timing(
        median_ms=float(np.median(samples)),
        p90_ms=float(np.percentile(samples, 90)),
        min_ms=float(np.min(samples)),
    )

Saved to minillm/bench.py.

Memory-bound or compute-bound?

Every projection in the model is a matrix multiplication, x @ W.T, with x holding one row per token. Its cost has two parts: reading W from memory, which doesn’t depend on the number of tokens, and the arithmetic, which grows with it. For \(n\) tokens and a \(K \to N\) projection in float32 (4 bytes per number), the ratio of the two is the operation’s arithmetic intensity:

\[I = \frac{\text{FLOPs}}{\text{bytes moved}} = \frac{2nKN}{4\,(KN + nK + nN)}\]

A device can’t run faster than its compute peak \(P_{\text{peak}}\), nor faster than its memory bandwidth \(B\) can feed it, which gives the roofline:

\[P \le \min\left(P_{\text{peak}},\; B \cdot I\right)\]

With few tokens, \(I\) is small and memory sets the limit; with many, \(I\) grows until compute does. Measure SmolLM2’s gate_proj (\(K = 576\), \(N = 1536\)) for 1 to 2048 tokens:

Show the code 29 lines
with default_device(DEVICE), default_dtype(DType.float32):
    proj = Linear(config.hidden_size, config.intermediate_size, bias=False)
run = proj.compile(
    TensorType(DType.float32, ("tokens", config.hidden_size), device=DEVICE)
)

K, N = config.hidden_size, config.intermediate_size
rows = []
for tokens in [1, 4, 16, 64, 256, 1024, 2048]:
    x = Tensor(
        np.random.default_rng(0)
        .standard_normal((tokens, K))
        .astype(np.float32),
        device=DEVICE,
    )
    t = benchmark(lambda: run(x).to_numpy())
    flops = 2 * tokens * K * N
    bytes_moved = 4 * (K * N + tokens * (K + N))
    rows.append(
        (
            tokens,
            t.median_ms,
            flops / t.median_ms / 1e6,
            bytes_moved / t.median_ms / 1e6,
        )
    )
    print(
        f"{tokens:5} tokens  {t}  {rows[-1][2]:7.1f} GFLOP/s  {rows[-1][3]:6.1f} GB/s"
    )
    1 tokens  median    0.56 ms  p90    0.63 ms      3.1 GFLOP/s     6.3 GB/s
    4 tokens  median    0.35 ms  p90    0.66 ms     20.3 GFLOP/s    10.2 GB/s
   16 tokens  median    0.39 ms  p90    0.47 ms     72.0 GFLOP/s     9.3 GB/s
   64 tokens  median    0.86 ms  p90    1.25 ms    131.0 GFLOP/s     4.7 GB/s
  256 tokens  median    2.07 ms  p90    2.97 ms    218.5 GFLOP/s     2.8 GB/s
 1024 tokens  median    8.22 ms  p90   11.20 ms    220.3 GFLOP/s     1.5 GB/s
 2048 tokens  median   10.09 ms  p90   14.24 ms    359.1 GFLOP/s     2.1 GB/s
Show the code 82 lines
tokens, ms, gflops, gbps = zip(*rows)
intensity = [
    2 * t * K * N / (4 * (K * N + t * (K + N))) for t in tokens
]  # FLOP per byte moved

fig = make_subplots(
    rows=1,
    cols=2,
    horizontal_spacing=0.14,
    subplot_titles=["Time per call", "Arithmetic throughput against intensity"],
)
fig.add_trace(
    go.Scatter(
        x=tokens,
        y=ms,
        mode="lines+markers",
        line_color=viz.BLUE,
        showlegend=False,
        hovertemplate="%{x} tokens: %{y:.3f} ms<extra></extra>",
    ),
    row=1,
    col=1,
)
fig.add_trace(
    go.Scatter(
        x=intensity,
        y=gflops,
        mode="lines+markers+text",
        line_color=viz.ORANGE,
        text=[str(t) for t in tokens],
        textposition="top left",
        showlegend=False,
        customdata=tokens,
        hovertemplate="%{customdata} tokens<br>%{x:.1f} FLOP/byte<br>%{y:.0f} GFLOP/s<extra></extra>",
    ),
    row=1,
    col=2,
)
# The roof uses the best bandwidth and throughput this sweep reached, not the
# device's datasheet numbers, so every point sits on or under it.
peak, bandwidth = max(gflops), max(gbps)
roof_x = np.geomspace(min(intensity) / 2, max(intensity) * 2, 200)
fig.add_trace(
    go.Scatter(
        x=roof_x,
        y=np.minimum(peak, bandwidth * roof_x),
        mode="lines",
        line=dict(color=viz.INK_SECONDARY, dash="dash", width=1.5),
        name="measured roofline",
        hovertemplate="roof at %{x:.1f} FLOP/byte: %{y:.0f} GFLOP/s<extra></extra>",
    ),
    row=1,
    col=2,
)
fig.add_annotation(
    x=np.log10(peak / bandwidth),
    y=np.log10(peak),
    text=f"ridge: {peak / bandwidth:.0f} FLOP/byte",
    showarrow=True,
    arrowhead=0,
    ay=-30,
    row=1,
    col=2,
)
fig.update_xaxes(title_text="tokens", type="log", dtick=1, row=1, col=1)
fig.update_yaxes(title_text="ms (log scale)", type="log", dtick=1, row=1, col=1)
fig.update_xaxes(
    title_text="FLOP per byte moved (log scale)",
    type="log",
    dtick=1,
    row=1,
    col=2,
)
fig.update_yaxes(
    title_text="GFLOP/s (log scale)", type="log", dtick=1, row=1, col=2
)
fig.update_layout(
    title="gate_proj (576 to 1536) from 1 to 2048 tokens; labels are token counts",
    height=420,
    legend=dict(x=0.58, y=1.12),
)
viz.show(fig)

Read the two plots together, and hover over the points. The dashed line on the right is the roofline drawn from this sweep: its slope is the best bandwidth any point reached, its flat top the best throughput. Points left of the ridge sit under the slanted, memory-bound part of the roof; points to the right sit under the flat, compute-bound part. A device’s datasheet ceilings are higher than these measured ones.

With few tokens, the time barely grows: 1 to 16 tokens all take about half a millisecond on this page’s machine, an Apple-silicon Mac with a Metal GPU. Treat these numbers as an example: yours depend on your hardware and vary from run to run. Each weight is used for only a few multiply-adds after it is loaded, so arithmetic throughput is low and the operation is memory-bound. At a matrix this small, though, most of that time is a fixed cost per call, for launching the work and reading the result back: the single-token call moves W’s 3.5 MB at under 10 GB/s, far below the GPU’s memory bandwidth. In a large model, reading the weights dominates instead.

With many tokens, each weight, once loaded, is reused for every token, so arithmetic throughput climbs and the time grows with the number of tokens. The operation moves toward the compute-bound end, where the hardware’s arithmetic units set the limit.

Decoding one token per step runs every weight matrix at the memory-bound end. That is why per-token latency depends on memory bandwidth, and why serving engines batch many requests into each step (Chapter 5): here, 16 tokens cost barely more than one. Prefill processes the whole prompt at once and runs toward the compute-bound end. The handbook’s GPU memory page explains the same trade-off on GPUs.

Benchmark a layer

The same harness measures a whole decoder layer at the shapes that matter in serving: decode steps (one token) for several batch sizes, and prefills of several lengths.

with default_device(DEVICE), default_dtype(DType.float32):
    layer = DecoderLayer(config)
layer.load_state_dict(select(load_checkpoint(), "model.layers.0."))
run_layer = layer.compile(
    TensorType(
        DType.float32, ("batch", "seq_len", config.hidden_size), device=DEVICE
    )
)

print("shape (batch x seq)       time per call       tokens/s")
medians = {}
for batch, seq in [(1, 1), (8, 1), (32, 1), (1, 128), (1, 512)]:
    x = Tensor(
        np.random.default_rng(0)
        .standard_normal((batch, seq, config.hidden_size))
        .astype(np.float32),
        device=DEVICE,
    )
    t = benchmark(lambda: run_layer(x).to_numpy(), warmup=5, iterations=50)
    medians[batch, seq] = t.median_ms
    print(f"{batch:5} x {seq:<5}  {t}  {batch * seq / t.median_ms * 1000:9.0f}")
print(
    f"a decode step for 32 sequences costs {medians[32, 1] / medians[1, 1]:.1f}x one sequence's"
)
shape (batch x seq)       time per call       tokens/s
    1 x 1      median    1.42 ms  p90    3.23 ms        704
    8 x 1      median    1.47 ms  p90    2.30 ms       5429
   32 x 1      median    1.68 ms  p90    2.35 ms      19027
    1 x 128    median    2.65 ms  p90    3.15 ms      48343
    1 x 512    median    8.16 ms  p90    8.84 ms      62743
a decode step for 32 sequences costs 1.2x one sequence's
Show the code 32 lines
labels = [f"{b} x {s}" for b, s in medians]
fig = make_subplots(
    rows=1,
    cols=2,
    horizontal_spacing=0.12,
    subplot_titles=["Time per call (ms)", "Tokens per second"],
)
fig.add_trace(
    go.Bar(
        x=labels,
        y=list(medians.values()),
        marker_color=viz.BLUE,
        showlegend=False,
        hovertemplate="%{x}: %{y:.2f} ms<extra></extra>",
    ),
    row=1,
    col=1,
)
fig.add_trace(
    go.Bar(
        x=labels,
        y=[b * s / ms * 1000 for (b, s), ms in medians.items()],
        marker_color=viz.AQUA,
        showlegend=False,
        hovertemplate="%{x}: %{y:,.0f} tokens/s<extra></extra>",
    ),
    row=1,
    col=2,
)
fig.update_xaxes(title_text="batch x sequence length")
fig.update_layout(title="One decoder layer at serving shapes", height=400)
viz.show(fig)

On this machine, a decode step for 32 sequences costs well under twice a single sequence’s step, so throughput in tokens per second rises sharply with batch size. Timings on a laptop are noisy, which is why the harness reports a percentile alongside the median. A 512-token prefill runs through one layer at tens of thousands of tokens per second; the model has 30 layers, so a whole-model prefill would run at a thirtieth of that, under 2,000 tokens per second.

These numbers describe this machine’s GPU. On another CPU or GPU the curves have the same shape at different scales, although the time isn’t always monotonic in size: the kernels behind each operation choose different strategies for different shapes, so on some devices a smaller batch takes longer than a larger one (this page’s run doesn’t show it). Measure the shapes you serve. MAX’s own layer benchmarks run this kind of sweep for every layer, and its kernel benchmarks do the same for individual kernels. For whole-server measurements, Section 7.2 uses max benchmark. The handbook’s LLM performance benchmarks page explains how to read published numbers.

Exercises

  1. Run the matmul sweep for down_proj (1536 → 576). Where does it switch from memory-bound to compute-bound?
  2. Benchmark the same layer with iterations=3 five times. How much do the medians vary? How many iterations make the result stable?
  3. Compare the eager and compiled layer at batch 1, sequence 1. Where does eager execution spend its time?

Next

Section 4.4 looks inside a run: which operations take the time, using MAX’s profiling tools.

4.4 Profiling

Open in Colab Open in Kaggle View on GitHub

A benchmark tells you how long something takes. A profile tells you where the time goes, which is what you need before optimizing anything. This section profiles at three levels: a whole generation run, the kernels a compiled graph launches, and the parts of your own model.

Show the setup 21 lines
import logging
import warnings

import numpy as np
import plotly.graph_objects as go
from max.driver import CPU
from max.dtype import DType
from max.graph import TensorType
from max.tensor import Tensor, default_device, default_dtype

from minillm import viz
from minillm.bench import benchmark
from minillm.device import pick_device
from minillm.config import LlamaConfig
from minillm.model import load_model
from minillm.weights import load_checkpoint

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU
config = LlamaConfig.from_pretrained()

Profile a whole run

max generate takes a --profile flag. On an NVIDIA GPU with Nsight Systems installed, it records a kernel trace and prints the kernels that took the most GPU time. It always profiles the Python side with cProfile too, which answers the first question to ask of any run: is the time in your model, or in the code around it?

!max generate --model HuggingFaceTB/SmolLM2-135M-Instruct \
    --quantization-encoding float32 --prompt "hello" --max-new-tokens 16 \
    --num-warmups 1 --profile 2>&1 | grep -A 8 "=== Top"
=== Top 15 Python/CPU functions (94.4% of CPU time, 307.64 ms total) ===
     %       total     calls  function
 49.0%   150.85 ms        64  ~:0<built-in method numpy.from_dlpack>
 43.0%   132.25 ms        32  max/engine/api.py:118(_Model_execute)
  0.3%     1.04 ms        16  max/pipelines/lib/pipeline_variants/text_generation.py:597(execute)
  0.3%   970.84 us        16  max/pipelines/sampling/sampling_logits_processor.py:109(_packed_rows_to_device)
  0.3%   781.75 us        16  max/pipelines/lib/module_registration.py:763(prepare_initial_token_inputs)
  0.2%   589.67 us        16  max/pipelines/sampling/sampling_logits_processor.py:695(create)
  0.2%   586.50 us        16  max/pipelines/lib/pipeline_variants/text_generation.py:610(_execute)

The time is in the model: _Model_execute, the call into the compiled graph, and, on a GPU, numpy.from_dlpack, which reads the logits back to the host and so waits for the GPU to finish its work. That wait is model time too, a common trap when reading profiles of asynchronous devices. On this page’s machine, an Apple-silicon Mac with a Metal GPU, the two take over 90% of the profiled time. Treat these numbers as an example: yours depend on your hardware and vary from run to run. The pipeline around the model (tokenization, scheduling, sampling, and the KV cache bookkeeping) takes under a percent per function. For this model, any speedup has to come from the compiled graph itself.

See which kernels run

MODULAR_DEBUG=op-log-level=TRACE makes the runtime log every operation it launches and completes. Trace the MLP from Section 1.4. The setting must be in place when the process starts, so the code runs as a separate script:

%%writefile trace_mlp.py
import numpy as np
from max.dtype import DType
from max.graph import TensorType
from max.tensor import Tensor, default_device, default_dtype

from minillm.config import LlamaConfig
from minillm.device import pick_device
from minillm.layers import MLP

DEVICE = pick_device()
with default_device(DEVICE), default_dtype(DType.float32):
    mlp = MLP(LlamaConfig())
run = mlp.compile(TensorType(DType.float32, ("seq_len", 576), device=DEVICE))
print("--- run ---", flush=True)
run(Tensor(np.ones((4, 576), np.float32), device=DEVICE)).to_numpy()
Overwriting trace_mlp.py
!MODULAR_DEBUG=op-log-level=TRACE python trace_mlp.py 2>&1 | sed -n '/--- run ---/,$p' | grep LAUNCH | grep -v 'LAUNCH matmul[^ ]'
[OP] LAUNCH matmul [id=22] target=gpu:0
[OP] LAUNCH Epilogue(matmul, Elementwise(silu, mul), operandIndices=[0]) [id=24] target=gpu:0
[OP] LAUNCH matmul [id=26] target=gpu:0

Each matmul logs a second LAUNCH line from inside the kernel. That line names the kernel and, in parentheses, the graph operation it serves (matmul(mo.matmul), or matmul(Epilogue(mo.matmul, ...)) for the fused one); the grep -v drops it, which leaves one line per kernel the graph launches.

The MLP has five operations in your Python (three matmuls, silu, and a multiply), more in the traced graph you saw in Section 3.2, and three kernels at run time. The Epilogue entry is the second matmul with silu and mul fused into it: the activation is applied as each tile of the matmul’s output is produced, before it is written to memory. That fusion is why the hand-written SwiGLU kernel in Section 3.1 had little left to fuse, and the trace is how you’d find out. The handbook’s kernel optimization for LLM inference page explains fusion and why it matters most on GPUs.

Profile by decomposition

To find where time goes inside your own model, time its parts separately with the harness from Section 4.3. Compile one layer’s attention, MLP, and norm, and the LM head, and time each at two shapes: one decode step (1 token) and a prefill (256 tokens). Each time is multiplied by how often the part runs in the model: 30 attention and MLP blocks, and 61 norms (two per layer plus the final one).

Show the code 31 lines
model = load_model(config, load_checkpoint(), DEVICE)
hidden = TensorType(
    DType.float32, ("batch", "seq_len", config.hidden_size), device=DEVICE
)
layer = model.model.layers[0]
parts = {
    "attention (x30)": (layer.self_attn.compile(hidden), 30),
    "MLP (x30)": (layer.mlp.compile(hidden), 30),
    "norms (x61)": (layer.input_layernorm.compile(hidden), 61),
    "LM head": (model.lm_head.compile(hidden), 1),
}

shares, all_times = {}, {}
for tokens in [1, 256]:
    x = Tensor(
        np.random.default_rng(0)
        .standard_normal((1, tokens, config.hidden_size))
        .astype(np.float32),
        device=DEVICE,
    )
    times = {
        name: benchmark(lambda: run(x).to_numpy(), iterations=30).median_ms
        * count
        for name, (run, count) in parts.items()
    }
    total = sum(times.values())
    shares[tokens] = {name: t / total for name, t in times.items()}
    all_times[tokens] = times
    print(f"{tokens} token(s): {total:.1f} ms in total")
    for name, t in times.items():
        print(f"  {name:16} {t:7.2f} ms  {t / total:5.0%}")
1 token(s): 49.2 ms in total
  attention (x30)    23.31 ms    47%
  MLP (x30)          10.01 ms    20%
  norms (x61)        13.80 ms    28%
  LM head             2.05 ms     4%
256 token(s): 156.0 ms in total
  attention (x30)    73.32 ms    47%
  MLP (x30)          33.32 ms    21%
  norms (x61)        20.77 ms    13%
  LM head            28.61 ms    18%
Show the code 25 lines
rows = ["decode (1 token)", "prefill (256 tokens)"]
fig = go.Figure(
    [
        go.Bar(
            y=rows,
            x=[shares[1][name], shares[256][name]],
            name=name,
            orientation="h",
            marker=dict(color=color, line=dict(color=viz.SURFACE, width=2)),
            customdata=[all_times[1][name], all_times[256][name]],
            hovertemplate=name
            + ": %{x:.0%} (%{customdata:.2f} ms)<extra>%{y}</extra>",
        )
        for name, color in zip(parts, viz.SERIES)
    ]
)
fig.update_layout(
    title="Where the model's time goes",
    barmode="stack",
    height=300,
    legend_traceorder="normal",
    xaxis=dict(title="share of model time", tickformat=".0%", range=[0, 1]),
    yaxis=dict(autorange="reversed", showgrid=False),
)
viz.show(fig)

Compare the two bars. The shares below were measured on this page’s Metal GPU.

Attention takes the largest share at both shapes, about 40%. Its cost is the only one that grows with the square of the sequence length, so its share grows on longer prompts; at 256 tokens the prompt is still short enough that it barely moves.

The LM head’s share grows the most, from a few percent to over a tenth. It holds more weights than all 30 layers’ attention put together (49,152 × 576), and here it runs on every one of the 256 prompt positions. The cached model from Section 1.8 computes it for the last position only, which keeps it small during prefill.

The norms look expensive at decode, over a quarter of the time, but that is mostly an artifact of this method. Each separately compiled call pays a fixed cost to launch its work and read back the result, and the decomposition counts that cost 61 times for the norms; the whole compiled model runs as one graph and pays it once. Time the whole model as one compiled graph to see how large that overhead is:

whole = model.compile(
    TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)
for tokens in [1, 256]:
    ids = Tensor(np.zeros((1, tokens), np.int64), device=DEVICE)
    whole_ms = benchmark(lambda: whole(ids).to_numpy(), iterations=30).median_ms
    parts_ms = sum(all_times[tokens].values())
    print(
        f"{tokens:3} token(s): whole model {whole_ms:6.1f} ms,"
        f" sum of parts {parts_ms:6.1f} ms ({parts_ms / whole_ms:.1f}x)"
    )
  1 token(s): whole model   24.0 ms, sum of parts   49.2 ms (2.0x)
256 token(s): whole model   84.0 ms, sum of parts  156.0 ms (1.9x)

At both shapes the whole model takes less than half the time its parts add up to. The difference is what the 122 separate calls cost: each one launches its work and reads its result back to the host, and the whole graph does both once. So a decomposition ranks the parts, but its shares overstate small operations such as the norms. Check a surprising number with the kernel trace before acting on it.

Every optimization starts here: find the largest share, decide whether it is memory-bound or compute-bound (Section 4.3), then look at its kernels (the trace above).

With Nsight Systems

On NVIDIA GPUs, Nsight Systems answers the same questions: set MODULAR_ENABLE_PROFILING=detailed and run your program under nsys profile, or pass --profile to max generate. max benchmark --trace does the same for a server that already runs under nsys launch, started with --gpu-profiling detailed. MAX marks its operations as named ranges in the timeline, so you can match GPU kernels to the operations that launched them. MAX’s guide to GPU profiling with Nsight Systems covers the setup. The handbook’s kernel optimization tools page surveys the tools.

Exercises

  1. Trace a full DecoderLayer and count its kernels. Which operations are fused into which?
  2. Add a third shape, a decode step for a batch of 32, to the decomposition. Which part’s share changes most?
  3. Profile the decode step of the KVCacheDecoder from Section 1.8 with cProfile from Python. How much of the step is Python overhead at this model size?

Next

Chapter 5 uses what you measured here to build the core of a serving engine that batches many requests into each step.

5. A serving engine

The model from Chapter 1 generates for one prompt at a time. A serving engine takes requests from many users at once and keeps them all moving. This chapter builds the core of such an engine, one technique per section.

SectionYou learn
5.1 The life of a requestA batched model step for an engine, and a first engine that serves one request at a time.
5.2 Continuous batchingA scheduler, and three batching policies compared on the same workload.
5.3 A paged KV cacheBlocks and block tables, the idea behind MAX Serve’s paged KV cache, and the memory they save.
5.4 Prefix cachingSharing the cache blocks of a common prompt prefix between requests.
5.5 Speculative decodingDrafting tokens with SmolLM2-135M, verifying them with SmolLM2-360M, and when it pays off.

Related handbook pages: batching, PagedAttention, prefix caching, and speculative decoding.

5.1 The life of a request

Open in Colab Open in Kaggle View on GitHub

So far your model served one prompt at a time, from a notebook. A serving engine takes requests from many users at once and keeps them all moving: it admits new requests, runs the model, hands each request its next token, and stops each one when it is done. This chapter builds that engine over five sections, on top of a new model interface designed for it.

MAX Serve and other inference engines share this structure:

The serving loop: requests go to the scheduler, which forms a batch for the model step; the sampler turns the logits into a new token for each request, which goes back to the scheduler. The KV cache manager sits beside the loop: the scheduler allocates and releases its blocks, and the model step reads and writes keys and values in them.The serving loop: requests go to the scheduler, which forms a batch for the model step; the sampler turns the logits into a new token for each request, which goes back to the scheduler. The KV cache manager sits beside the loop: the scheduler allocates and releases its blocks, and the model step reads and writes keys and values in them.

The handbook’s How does LLM inference work? page describes the same loop.

Show the setup 24 lines
import logging
import time
import warnings

import numpy as np
import plotly.graph_objects as go
from max.dtype import DType
from max.graph import TensorType
from max.pipelines import AutoTokenizer
from max.tensor import Tensor
from plotly.subplots import make_subplots

from minillm import viz
from minillm.config import MODEL_ID, LlamaConfig
from minillm.device import pick_device
from minillm.model import load_model
from minillm.weights import load_checkpoint

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU

config = LlamaConfig.from_pretrained()
tokenizer = AutoTokenizer(MODEL_ID)
Show the setup 22 lines
from __future__ import annotations

import math
from collections.abc import Callable

from max import functional as F
from max.dtype import DType
from max.nn import ModuleList
from max.nn.kernels import flash_attention
from max.tensor import Tensor

from minillm.attention import Attention
from minillm.config import LlamaConfig
from minillm.layers import rotate_half
from minillm.model import DecoderLayer, LlamaForCausalLM

KVPair = tuple[Tensor, Tensor]
Step = Callable[..., tuple[Tensor, list[KVPair]]]
# Masked scores use a large finite value instead of -inf: attention kernels
# process keys in blocks, and a row whose first block is fully -inf computes
# exp(-inf - -inf) = NaN, which then spreads through the values to other rows.
MASKED = -1e30

Saved to minillm/step.py.

A model step for an engine

The cached model from Section 1.8 kept its cache inside and handled one sequence. An engine needs one call that advances several sequences, each with its own history, by any number of tokens. The step model does that. Each call takes, for every sequence in the batch:

  • the new tokens and their positions,
  • an attention mask saying which past and new positions each token may see,
  • the past keys and values,

and returns the next-token logits plus only the new keys and values. The model keeps no state. Where the cache lives and how it is laid out is up to the engine, and Section 5.3 and Section 5.4 change exactly that.

Positions now differ per sequence, so RoPE takes a position for each token instead of a start offset:

def rope_at(
    x: Tensor, positions: Tensor, head_dim: int, theta: float
) -> Tensor:
    """Rotates ``x``, ``[batch, q, heads, head_dim]``, for per-token
    ``positions``, ``[batch, q]``."""
    inv_freq = Tensor(
        [theta ** (-i / head_dim) for i in range(0, head_dim, 2)],
        dtype=DType.float32,
        device=x.mesh,
    )
    batch, q = positions.shape
    angles = positions.reshape([batch, q, 1]) * inv_freq
    angles = F.concat([angles, angles], axis=-1).reshape(
        [batch, q, 1, head_dim]
    )
    return x * F.cos(angles) + rotate_half(x) * F.sin(angles)

Saved to minillm/step.py.

The attention layer projects only the new tokens, appends their keys and values to the past ones, and attends over both. The mask has one column per past and new position:

Show StepAttention 27 lines
class StepAttention(Attention):
    """Attends from new tokens to the past keys and values plus their own."""

    def forward(
        self, x: Tensor, positions: Tensor, mask: Tensor, past: KVPair
    ) -> tuple[Tensor, KVPair]:
        batch, q, _ = x.shape
        query = self.q_proj(x).reshape(
            [batch, q, self.num_heads, self.head_dim]
        )
        key = self.k_proj(x).reshape(
            [batch, q, self.num_kv_heads, self.head_dim]
        )
        value = self.v_proj(x).reshape(
            [batch, q, self.num_kv_heads, self.head_dim]
        )
        query = rope_at(query, positions, self.head_dim, self.rope_theta)
        key = rope_at(key, positions, self.head_dim, self.rope_theta)
        all_keys = F.concat([past[0], key], axis=1)
        all_values = F.concat([past[1], value], axis=1)
        # The mask has one column per past and new position.
        mask = F.rebind(mask, [batch, q, all_keys.shape[1]])
        out = flash_attention(
            query, all_keys, all_values, 1 / math.sqrt(self.head_dim), mask
        ).rebind_mapping(query.mapping)
        out = out.reshape([batch, q, self.num_heads * self.head_dim])
        return self.o_proj(out), (key, value)

Saved to minillm/step.py.

The decoder layer and the model pass the new arguments through, and the model collects each layer’s new keys and values:

Show StepLayer, StepModel 38 lines
class StepLayer(DecoderLayer):
    def __init__(self, config: LlamaConfig) -> None:
        super().__init__(config)
        self.self_attn = StepAttention(config)

    def forward(
        self, x: Tensor, positions: Tensor, mask: Tensor, past: KVPair
    ) -> tuple[Tensor, KVPair]:
        attn, kv = self.self_attn(
            self.input_layernorm(x), positions, mask, past
        )
        x = x + attn
        return x + self.mlp(self.post_attention_layernorm(x)), kv


class StepModel(LlamaForCausalLM):
    """Advances a batch of sequences: returns each one's next-token logits
    and the new tokens' keys and values for every layer."""

    def __init__(self, config: LlamaConfig) -> None:
        super().__init__(config)
        self.model.layers = ModuleList(
            [StepLayer(config) for _ in range(config.num_hidden_layers)]
        )

    def forward(
        self,
        tokens: Tensor,
        positions: Tensor,
        mask: Tensor,
        past: list[KVPair],
    ) -> tuple[Tensor, list[KVPair]]:
        x = self.model.embed_tokens(tokens)
        new_kv = []
        for layer, layer_past in zip(self.model.layers, past, strict=True):
            x, kv = layer(x, positions, mask, layer_past)
            new_kv.append(kv)
        return self.lm_head(self.model.norm(x)[:, -1]), new_kv

Saved to minillm/step.py.

Compile it once. Every dimension that changes between calls is symbolic: the batch size, the number of new tokens q, and the past length:

import numpy as np
from max.driver import Device
from max.graph import TensorType


def compile_step(model: StepModel, config: LlamaConfig, device: Device) -> Step:
    """Compiles ``model`` for any batch size, number of new tokens, and past
    length."""
    kv = TensorType(
        DType.float32,
        ("batch", "past", config.num_key_value_heads, config.head_dim),
        device=device,
    )
    return model.compile(
        TensorType(DType.int64, ("batch", "q"), device=device),
        TensorType(DType.float32, ("batch", "q"), device=device),
        TensorType(DType.float32, ("batch", "q", "kv"), device=device),
        [(kv, kv)] * config.num_hidden_layers,
    )

Saved to minillm/step.py.

start = time.perf_counter()
weights = load_checkpoint()
step = compile_step(
    load_model(config, weights, DEVICE, StepModel), config, DEVICE
)
print(f"compiled the step model in {time.perf_counter() - start:.0f} s")
compiled the step model in 13 s

The cache and the batch

The engine stores each sequence’s keys and values in a cache object. The first version is the simplest: one growing array per layer. Every later cache has the same interface (length, num_layers, read, and append), so run_step and the step model work with all of them unchanged.

class ContiguousCache:
    """One sequence's keys and values: a growing array per layer."""

    def __init__(self, config: LlamaConfig) -> None:
        shape = (0, config.num_key_value_heads, config.head_dim)
        self.layers = [
            [np.zeros(shape, np.float32), np.zeros(shape, np.float32)]
            for _ in range(config.num_hidden_layers)
        ]

    @property
    def length(self) -> int:
        return len(self.layers[0][0])

    @property
    def num_layers(self) -> int:
        return len(self.layers)

    def read(self, layer: int) -> tuple[np.ndarray, np.ndarray]:
        return self.layers[layer][0], self.layers[layer][1]

    def append(self, layer: int, keys: np.ndarray, values: np.ndarray) -> None:
        entry = self.layers[layer]
        entry[0] = np.concatenate([entry[0], keys])
        entry[1] = np.concatenate([entry[1], values])

Saved to minillm/step.py.

To run several sequences in one step, the engine packs them into rectangular arrays. Sequences add different numbers of new tokens, so shorter ones are left-padded, and the mask is built per token: a new token may see its own sequence’s past and the new tokens up to itself, and nothing else. A padding token sees only itself, so its row of the softmax stays well defined; its output is discarded. Masked positions get a large negative number instead of \(-\infty\). If a padding row’s first block of keys were entirely \(-\infty\), a blockwise attention kernel would compute \(e^{-\infty + \infty}\), which is NaN. In the next layer that NaN would become the padding position’s keys and values, and the real tokens of the same sequence would pick it up even though the mask hides that position.

Show run_step 45 lines
def run_step(
    step: Step, caches: list, new_tokens: list[list[int]], device: Device
) -> np.ndarray:
    """Advances each sequence by its ``new_tokens``, appends their keys and
    values to its cache, and returns next-token logits, ``[batch, vocab]``."""
    batch, q = len(caches), max(len(t) for t in new_tokens)
    past = max(c.length for c in caches)
    tokens = np.zeros((batch, q), np.int64)
    positions = np.zeros((batch, q), np.float32)
    mask = np.full((batch, q, past + q), MASKED, np.float32)
    for i, (cache, new) in enumerate(zip(caches, new_tokens, strict=True)):
        pad, n = q - len(new), cache.length
        tokens[i, pad:] = new
        positions[i, pad:] = np.arange(n, n + len(new))
        mask[i, pad:, :n] = 0.0  # every new token sees the sequence's past
        mask[i, pad:, past + pad :] = np.triu(
            np.full((len(new), len(new)), MASKED), 1
        )
        mask[i, np.arange(pad), past + np.arange(pad)] = (
            0.0  # padding sees itself
        )
    past_kv = []
    for layer in range(caches[0].num_layers):
        keys = np.zeros(
            (batch, past, *caches[0].read(layer)[0].shape[1:]), np.float32
        )
        values = np.zeros_like(keys)
        for i, cache in enumerate(caches):
            k, v = cache.read(layer)
            keys[i, : len(k)], values[i, : len(v)] = k, v
        past_kv.append(
            (Tensor(keys, device=device), Tensor(values, device=device))
        )
    logits, new_kv = step(
        Tensor(tokens, device=device),
        Tensor(positions, device=device),
        Tensor(mask, device=device),
        past_kv,
    )
    for layer, (k, v) in enumerate(new_kv):
        k, v = k.to_numpy(), v.to_numpy()
        for i, (cache, new) in enumerate(zip(caches, new_tokens, strict=True)):
            pad = q - len(new)
            cache.append(layer, k[i, pad:], v[i, pad:])
    return logits.to_numpy()

Saved to minillm/step.py.

Check the step against the model from Section 1.6 by generating greedily for two prompts of different lengths in the same steps:

Show greedy_reference 12 lines
full = load_model(config, weights, DEVICE).compile(
    TensorType(DType.int64, ("batch", "seq"), device=DEVICE)
)


def greedy_reference(ids: list[int], n: int) -> list[int]:
    """Generates ``n`` tokens greedily with the full model, no cache."""
    ids = list(ids)
    for _ in range(n):
        logits = full(Tensor(np.array([ids]), device=DEVICE)).to_numpy()
        ids.append(int(logits[0, -1].argmax()))
    return ids[-n:]
prompts = [
    tokenizer([t])["input_ids"][0]
    for t in ["The capital of France is", "Once upon a time, there was a"]
]
caches = [ContiguousCache(config) for _ in prompts]
outputs = [[int(t)] for t in run_step(step, caches, prompts, DEVICE).argmax(-1)]
for _ in range(11):
    next_ids = run_step(
        step, caches, [[o[-1]] for o in outputs], DEVICE
    ).argmax(-1)
    for out, t in zip(outputs, next_ids):
        out.append(int(t))
for prompt, out in zip(prompts, outputs):
    print(out == greedy_reference(prompt, 12), repr(tokenizer.decode(out)))
    assert out == greedy_reference(prompt, 12)
True ' Paris. Paris is the largest city in France and the capital'
True ' young woman named Sarah who was struggling to find her place in'

Both sequences match the reference, even though they have different lengths and shared every step.

Here is the mask run_step builds for a mixed step: sequence 0 has 6 tokens cached and decodes 1 new token, while sequence 1 has nothing cached and prefills a 4-token prompt. A wrapper around step records the inputs it receives. Blue cells may attend; hover over any cell:

recorded = {}


def recording_step(tokens, positions, mask, past):
    recorded.update(
        tokens=tokens.to_numpy(),
        positions=positions.to_numpy(),
        mask=mask.to_numpy(),
    )
    return step(tokens, positions, mask, past)


caches = [ContiguousCache(config), ContiguousCache(config)]
run_step(
    step, caches[:1], [prompts[1][:6]], DEVICE
)  # sequence 0 now has 6 cached tokens
run_step(recording_step, caches, [[prompts[1][6]], prompts[0][:4]], DEVICE)
array([[ -2.6938362,  -9.929393 ,  -7.831539 , ...,  -7.389453 ,
         -0.1006913, -14.242568 ],
       [ 13.415854 ,   5.296691 ,  10.037319 , ...,   2.9147892,
          6.7043333,   4.4411554]], shape=(2, 49152), dtype=float32)
Show the code 60 lines
mask, positions = recorded["mask"], recorded["positions"]
past = mask.shape[2] - mask.shape[1]
columns = [f"cached position {j}" for j in range(past)] + [
    f"new position {j}" for j in range(mask.shape[1])
]
fig = make_subplots(
    rows=1,
    cols=2,
    horizontal_spacing=0.26,
    subplot_titles=[
        "sequence 0: 6 cached, decodes 1",
        "sequence 1: nothing cached, prefills 4",
    ],
)
for i, new_length in enumerate([1, 4]):
    pad = mask.shape[1] - new_length  # left padding, as in run_step
    rows = [
        f"padding row {r}"
        if r < pad
        else f"new token at position {int(positions[i, r])}"
        for r in range(mask.shape[1])
    ]
    text = [
        [
            f"{rows[r]} -> {columns[c]}: "
            + ("visible" if mask[i, r, c] == 0 else "masked")
            for c in range(mask.shape[2])
        ]
        for r in range(mask.shape[1])
    ]
    fig.add_trace(
        go.Heatmap(
            z=(mask[i] == 0).astype(int),
            zmin=0,
            zmax=1,
            showscale=False,
            xgap=2,
            ygap=2,
            colorscale=[
                [0, viz.GRID],
                [0.5, viz.GRID],
                [0.5, viz.BLUE],
                [1, viz.BLUE],
            ],
            x=columns,
            y=rows,
            customdata=text,
            hovertemplate="%{customdata}<extra></extra>",
        ),
        row=1,
        col=i + 1,
    )
fig.update_xaxes(showgrid=False, tickangle=-45)
fig.update_yaxes(showgrid=False, autorange="reversed")
fig.update_layout(
    title="One step's attention mask for a batch of two sequences",
    height=440,
    margin=dict(l=8),
)
viz.show(fig)

Sequence 0’s single new token sits in the last row, after three padding rows that each see only themselves. It sees its 6 cached positions and itself. Sequence 1 has no cached tokens, so its 6 cached positions are masked in every row. They hold zeros, there only because the batch’s longest past is 6. Its 4 new tokens form the triangle of the causal mask.

A first engine: one request at a time

A request carries its prompt, how many tokens it may generate, and timestamps for the latency metrics from Section 1.1, such as time to first token (TTFT). Every engine in this chapter serves the same workload: eight requests that arrive at once, each asking for between 8 and 32 new tokens:

from dataclasses import dataclass, field


@dataclass
class Request:
    prompt: list[int]
    max_new_tokens: int
    output: list[int] = field(default_factory=list)
    arrived: float = 0.0
    first_token: float | None = None
    finished: float | None = None

    @property
    def done(self) -> bool:
        return len(self.output) >= self.max_new_tokens

Saved to minillm/step.py.

import time
from collections.abc import Mapping

Tokenizer = Callable[[list[str]], Mapping[str, list[list[int]]]]

WORKLOAD = [
    ("Tell me about Paris.", 32),
    ("Write a haiku about the sea.", 24),
    ("What is 2 + 2?", 8),
    ("List three colors.", 16),
    ("Explain gravity in one sentence.", 32),
    ("Name a fruit.", 8),
    ("Describe a cat.", 24),
    ("Say hello in French.", 16),
]


def make_workload(tokenizer: Tokenizer) -> list[Request]:
    """Returns the chapter's eight requests, all arriving now."""
    now = time.perf_counter()
    return [
        Request(tokenizer([text])["input_ids"][0], n, arrived=now)
        for text, n in WORKLOAD
    ]

Saved to minillm/step.py.

The simplest engine serves requests in arrival order, each to completion, one at a time:

Show report 9 lines
def report(requests: list[Request]) -> None:
    ttft = [(r.first_token - r.arrived) * 1000 for r in requests]
    total = max(r.finished for r in requests) - min(r.arrived for r in requests)
    tokens = sum(len(r.output) for r in requests)
    print(
        f"TTFT: first request {ttft[0]:.0f} ms, "
        f"median {np.median(ttft):.0f} ms, worst {max(ttft):.0f} ms"
    )
    print(f"{tokens} tokens in {total:.1f} s = {tokens / total:.1f} tokens/s")
def serve_one_at_a_time(requests: list[Request]) -> None:
    for request in requests:
        cache = ContiguousCache(config)
        logits = run_step(step, [cache], [request.prompt], DEVICE)
        while True:
            request.output.append(int(logits[0].argmax()))
            request.first_token = request.first_token or time.perf_counter()
            if request.done:
                break
            logits = run_step(step, [cache], [[request.output[-1]]], DEVICE)
        request.finished = time.perf_counter()


sequential = make_workload(tokenizer)
serve_one_at_a_time(sequential)
report(sequential)
TTFT: first request 296 ms, median 16553 ms, worst 30666 ms
160 tokens in 35.6 s = 4.5 tokens/s

Eight users sent their requests at the same moment. The first one got a token almost at once. The last one waited for every request ahead of it to finish, so its time to first token is most of the total run time. Section 4.3 measured a decoder layer’s decode step for 32 sequences at about 1.2 to 1.3 times the cost of a step for one, on the Mac the book was built on, so this engine leaves most of the hardware’s capacity unused.

The engines in this chapter favor clarity over speed. On every step, run_step packs each layer’s past keys and values into NumPy arrays, copies them to the device, and copies the new ones back, so a step costs more here than the cached decode step in Section 1.8. Compare this chapter’s timings with each other, not with Chapter 1’s. Section 5.2 runs several requests in each step.

The handbook’s LLM inference metrics page explains why serving systems track TTFT percentiles as well as averages.

Exercises

  1. Serve the workload in order of max_new_tokens, shortest first. What happens to the median and worst TTFT? What would a user with a long request think of this policy?
  2. Run run_step with three prompts in one call and check each against greedy_reference.
  3. Print the mask run_step builds for two sequences with 2 and 4 new tokens and past lengths 3 and 1. Check each row by hand.

Next

Section 5.2 replaces the one-at-a-time loop with continuous batching.

5.2 Continuous batching

Open in Colab Open in Kaggle View on GitHub

The engine from Section 5.1 ran one sequence per step, while Section 4.3 showed that a decode step for many sequences costs little more than a step for one. Batching uses that spare capacity, and the way you batch decides who waits. This section adds a scheduler to the engine and compares three policies on the same workload.

Show the setup 31 lines
import logging
import warnings

import numpy as np
import plotly.graph_objects as go
from max.pipelines import AutoTokenizer

from minillm import viz
from minillm.config import MODEL_ID, LlamaConfig
from minillm.device import pick_device
from minillm.model import load_model
from minillm.step import (
    WORKLOAD,
    ContiguousCache,
    Request,
    StepModel,
    compile_step,
    make_workload,
    run_step,
)
from minillm.weights import load_checkpoint

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU

config = LlamaConfig.from_pretrained()
tokenizer = AutoTokenizer(MODEL_ID)
step = compile_step(
    load_model(config, load_checkpoint(), DEVICE, StepModel), config, DEVICE
)

The engine

The engine holds two lists: requests waiting to start and requests running. Each call to step does one model step:

  • If there is room in the batch and requests are waiting, it admits them and runs their prefill: all their prompt tokens, in one batched step.
  • Otherwise it runs a decode step: one new token for every running request, in one batched step.

Finished requests leave the batch after every step. That is continuous batching: the batch is re-formed at every step, so a waiting request starts as soon as a slot frees up. Setting continuous=False gives static batching instead: a batch is admitted together and must finish completely before the next one starts.

Running prefill and decode in separate steps is a simple policy that gives new requests priority, at the cost of pausing the running requests’ decode for a step. Production engines can also split a long prompt into chunks (chunked prefill) and run prefill chunks and decodes in the same step, so one big prefill doesn’t stall everyone. MAX Serve chunks long prefills by default (--enable-chunked-prefill) and batches them with decodes when you pass --enable-in-flight-batching.

Show Engine 86 lines
from __future__ import annotations

import time
from collections import deque
from collections.abc import Callable

from max.driver import Device

from minillm.config import LlamaConfig
from minillm.step import ContiguousCache, Request, Step, run_step


class Engine:
    """Serves requests with a batching scheduler and greedy decoding."""

    def __init__(
        self,
        step: Step,
        config: LlamaConfig,
        device: Device,
        max_batch: int = 4,
        continuous: bool = True,
        new_cache: Callable[[LlamaConfig], object] = ContiguousCache,
    ) -> None:
        self.step_fn = step
        self.config = config
        self.device = device
        self.max_batch = max_batch
        self.continuous = continuous
        self.new_cache = new_cache
        self.waiting: deque[Request] = deque()
        self.running: list[tuple[Request, object]] = []
        self.steps = 0

    def add(self, request: Request) -> None:
        self.waiting.append(request)

    def has_work(self) -> bool:
        return bool(self.waiting or self.running)

    def step(self) -> None:
        room = self.max_batch - len(self.running)
        can_admit = (
            self.waiting and room > 0 and (self.continuous or not self.running)
        )
        if can_admit:
            admitted = [
                self.waiting.popleft()
                for _ in range(min(room, len(self.waiting)))
            ]
            batch = [(r, self.new_cache(self.config)) for r in admitted]
            inputs = [self.prefill_tokens(r, c) for r, c in batch]
        else:
            batch = self.running
            inputs = [[r.output[-1]] for r, _ in batch]
        logits = run_step(
            self.step_fn, [c for _, c in batch], inputs, self.device
        )
        self.steps += 1
        now = time.perf_counter()
        for (request, _), token in zip(batch, logits.argmax(-1), strict=True):
            request.output.append(int(token))
            if request.first_token is None:
                request.first_token = now
            if request.done:
                request.finished = now
        if can_admit:
            for request, cache in batch:
                self.prefilled(request, cache)
            self.running += batch
        self.running = [(r, c) for r, c in self.running if not r.done]
        self.release_finished(batch)

    def prefill_tokens(self, request: Request, cache: object) -> list[int]:
        """Returns the prompt tokens to compute; Section 5.4 skips cached ones."""
        return request.prompt

    def prefilled(self, request: Request, cache: object) -> None:
        """Runs after a request's prefill; Section 5.4 uses it."""

    def release_finished(self, batch: list) -> None:
        """Frees the caches of finished requests; Section 5.3 needs this."""

    def run(self) -> None:
        while self.has_work():
            self.step()

Saved to minillm/engine.py.

A workload

Serve the eight requests from Section 5.1 under three policies. A recording subclass of Engine notes which request holds each batch slot at every step, for the figures below:

Show RecordingEngine, run_policy, summarize 51 lines
class RecordingEngine(Engine):
    """Records, per step, the request in each batch slot and what it did."""

    def step(self) -> None:
        before = {id(r): len(r.output) for r in self.requests}
        super().step()
        ran = [r for r in self.requests if len(r.output) > before[id(r)]]
        for r in ran:
            if r not in self.slots:
                self.slots[self.slots.index(None)] = r
        row = []
        for r in self.slots:
            if r is None:
                row.append(None)
            elif r not in ran:
                row.append((self.requests.index(r), "waits"))
            elif before[id(r)] == 0:
                row.append((self.requests.index(r), "prefill"))
            else:
                row.append((self.requests.index(r), "decode"))
        self.history.append(row)
        self.slots = [None if r is None or r.done else r for r in self.slots]


def run_policy(max_batch: int, continuous: bool) -> tuple[list, Engine]:
    engine = RecordingEngine(
        step, config, DEVICE, max_batch=max_batch, continuous=continuous
    )
    engine.requests = make_workload(tokenizer)
    engine.slots = [None] * max_batch
    engine.history = []
    for r in engine.requests:
        engine.add(r)
    engine.run()
    return engine.requests, engine


def summarize(name: str, requests: list[Request], engine: Engine) -> dict:
    ttft = [(r.first_token - r.arrived) * 1000 for r in requests]
    total = max(r.finished for r in requests) - min(r.arrived for r in requests)
    tokens = sum(len(r.output) for r in requests)
    print(
        f"{name:22} {engine.steps:4} steps  {tokens / total:5.1f} tokens/s  "
        f"TTFT median {np.median(ttft):6.0f} ms  worst {max(ttft):6.0f} ms"
    )
    return {
        "ttft": ttft,
        "outputs": [r.output for r in requests],
        "requests": requests,
        "history": engine.history,
    }
results = {}
for name, max_batch, continuous in [
    ("one at a time", 1, True),
    ("static batches of 4", 4, False),
    ("continuous, batch 4", 4, True),
]:
    results[name] = summarize(name, *run_policy(max_batch, continuous))
one at a time           160 steps    2.9 tokens/s  TTFT median  27801 ms  worst  50743 ms
static batches of 4      64 steps    3.2 tokens/s  TTFT median   6450 ms  worst  12693 ms
continuous, batch 4      50 steps    9.3 tokens/s  TTFT median   2151 ms  worst   8903 ms

Batching changes only the order of the work, so greedy decoding must produce the same tokens for every request under every policy:

reference = results["one at a time"]["outputs"]
print(all(r["outputs"] == reference for r in results.values()))
assert all(r["outputs"] == reference for r in results.values())
True
Show the code 41 lines
steps = []
for name, r in results.items():
    start = min(q.arrived for q in r["requests"])
    rows = [f"request {i} ({n} tokens)" for i, (_, n) in enumerate(WORKLOAD)]
    wait = [(q.first_token - q.arrived) * 1000 for q in r["requests"]]
    gen = [(q.finished - q.first_token) * 1000 for q in r["requests"]]
    first = [(q.first_token - start) * 1000 for q in r["requests"]]
    steps.append(
        [
            go.Bar(
                y=rows,
                x=wait,
                base=[(q.arrived - start) * 1000 for q in r["requests"]],
                orientation="h",
                name="waiting, then prefill",
                marker_color=viz.MUTED,
                legendgroup="wait",
                hovertemplate="%{y}: first token after %{x:.0f} ms<extra></extra>",
            ),
            go.Bar(
                y=rows,
                x=gen,
                base=first,
                orientation="h",
                name="generating",
                marker_color=viz.BLUE,
                legendgroup="gen",
                hovertemplate="%{y}: generating for %{x:.0f} ms<extra></extra>",
            ),
        ]
    )
fig = go.Figure(
    layout=dict(
        title="Each request's life under each policy",
        height=460,
        barmode="overlay",
        xaxis_title="milliseconds since the requests arrived",
        yaxis=dict(autorange="reversed", showgrid=False),
    )
)
viz.show(viz.with_slider(fig, steps, list(results), "policy: ", active=2))

Slide between the policies. One at a time, each request waits for every request before it. Static batches run in two rounds, and the second round can’t start until the slowest request of the first finishes. Continuous batching starts a waiting request as soon as any slot frees up.

The same runs from the engine’s side: each column is one engine step, each row one of the four batch slots, and the color says which request holds the slot. “P” marks a request’s prefill step, and a dot marks a running request that sat out the step because another request was prefilling:

Show the code 54 lines
policies = ["static batches of 4", "continuous, batch 4"]
colors = [viz.GRID] + viz.SERIES  # empty, then one color per request
scale = [
    [edge, c] for i, c in enumerate(colors) for edge in (i / 9, (i + 1) / 9)
]
steps = []
for name in policies:
    history = np.array(
        [
            [-1 if s is None else s[0] for s in row]
            for row in results[name]["history"]
        ]
    ).T
    marks = {"prefill": "P", "waits": "·", "decode": ""}
    text = [
        ["" if s is None else marks[s[1]] for s in row]
        for row in results[name]["history"]
    ]
    hover = [
        [
            f"step {t + 1}, slot {k}: "
            + ("empty" if s is None else f"request {s[0]}, {s[1]}")
            for t, s in enumerate(column)
        ]
        for k, column in enumerate(zip(*results[name]["history"]))
    ]
    steps.append(
        [
            go.Heatmap(
                z=history,
                x=np.arange(1, history.shape[1] + 1),
                y=[f"slot {k}" for k in range(history.shape[0])],
                zmin=-1.5,
                zmax=7.5,
                colorscale=scale,
                showscale=False,
                xgap=1,
                ygap=3,
                text=np.array(text).T,
                texttemplate="%{text}",
                customdata=hover,
                hovertemplate="%{customdata}<extra></extra>",
            )
        ]
    )
fig = go.Figure(
    layout=dict(
        title="Which request holds each batch slot at every engine step",
        height=320,
        xaxis=dict(title="engine step", showgrid=False),
        yaxis=dict(autorange="reversed", showgrid=False),
    )
)
viz.show(viz.with_slider(fig, steps, policies, "policy: ", active=1))

Both batching policies cut the number of model steps, from 160 to 64 for static batches and 50 for continuous batching, so they finish the workload sooner and raise throughput. They differ in who waits:

  • Static batching admits four requests and runs until the longest of them finishes. The second group waits for that, even after the short requests in the first group are done and their slots sit empty: the grey holes in the slot chart.
  • Continuous batching refills a slot as soon as a request finishes, so waiting requests start sooner, and the batch stays full while any request is waiting.

Production engines, MAX Serve among them, use continuous batching with much larger batches, bounded by the memory the KV cache needs. That bound is the subject of Section 5.3. The handbook’s static, dynamic, and continuous batching page compares the policies in production terms.

Exercises

  1. Run continuous batching with max_batch 2, 4, and 8. How do throughput and worst-case TTFT change?
  2. Make requests arrive over time: add each one after a few engine steps instead of all at the start. Which policy handles bursts better?
  3. Implement a mixed step: admit new requests’ prefills in the same step as running requests’ decodes. What does it cost the running requests?

Next

Section 5.3 replaces the contiguous cache with a paged one, so the batch can grow without wasting memory.

5.3 A paged KV cache

Open in Colab Open in Kaggle View on GitHub

With continuous batching, memory limits how many requests can be in flight: every request needs room for its keys and values, and Section 1.8 measured 45 KiB per token for SmolLM2-135M in float32. The question is how to hand out that memory. A server doesn’t know how long a reply will be when it starts, so the simplest scheme reserves room for the maximum length, and most of that room goes unused. (ContiguousCache avoids the reservation by copying its arrays on every append, which costs time instead.) This section replaces the contiguous cache with a paged one, the idea behind vLLM’s PagedAttention and MAX Serve’s paged KV cache.

Show the setup 24 lines
import logging
import warnings

import numpy as np
import plotly.graph_objects as go
from max.pipelines import AutoTokenizer

from minillm import viz
from minillm.config import MODEL_ID, LlamaConfig
from minillm.device import pick_device
from minillm.engine import Engine
from minillm.model import load_model
from minillm.step import Request, StepModel, compile_step, make_workload
from minillm.weights import load_checkpoint

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU

config = LlamaConfig.from_pretrained()
tokenizer = AutoTokenizer(MODEL_ID)
step = compile_step(
    load_model(config, load_checkpoint(), DEVICE, StepModel), config, DEVICE
)

Blocks and block tables

Paging works like virtual memory. The cache memory is one pool of fixed-size blocks, each holding the keys and values of block_size tokens. A sequence owns a list of block numbers, its block table, and takes a new block from the pool only when its last block fills. A sequence’s blocks needn’t be next to each other, so free memory never breaks into gaps too small to use; the only waste is the unfilled part of each sequence’s last block. A finished sequence returns its blocks to the pool at once.

Show BlockPool, PagedCache 79 lines
from __future__ import annotations

import numpy as np

from minillm.config import LlamaConfig


class BlockPool:
    """Fixed-size blocks of KV cache memory, shared by all sequences."""

    def __init__(
        self, config: LlamaConfig, num_blocks: int, block_size: int = 16
    ) -> None:
        self.block_size = block_size
        shape = (
            num_blocks,
            block_size,
            config.num_key_value_heads,
            config.head_dim,
        )
        self.keys = [
            np.zeros(shape, np.float32) for _ in range(config.num_hidden_layers)
        ]
        self.values = [
            np.zeros(shape, np.float32) for _ in range(config.num_hidden_layers)
        ]
        self.free = list(range(num_blocks))

    @property
    def num_blocks(self) -> int:
        return len(self.keys[0])

    def allocate(self) -> int:
        if not self.free:
            raise MemoryError("the KV cache pool is full")
        return self.free.pop()

    def available(self) -> int:
        """Returns how many blocks ``allocate`` can still hand out."""
        return len(self.free)

    def release(self, blocks: list[int]) -> None:
        self.free.extend(blocks)


class PagedCache:
    """One sequence's keys and values, stored in pool blocks."""

    def __init__(self, pool: BlockPool) -> None:
        self.pool = pool
        self.blocks: list[int] = []
        self.length = 0

    @property
    def num_layers(self) -> int:
        return len(self.pool.keys)

    def read(self, layer: int) -> tuple[np.ndarray, np.ndarray]:
        keys = self.pool.keys[layer][self.blocks].reshape(
            -1, *self.pool.keys[layer].shape[2:]
        )
        values = self.pool.values[layer][self.blocks].reshape(keys.shape)
        return keys[: self.length], values[: self.length]

    def append(self, layer: int, keys: np.ndarray, values: np.ndarray) -> None:
        size, start = self.pool.block_size, self.length
        if layer == 0:  # every layer stores the same positions
            while len(self.blocks) * size < start + len(keys):
                self.blocks.append(self.pool.allocate())
        for i in range(len(keys)):
            block, slot = self.blocks[(start + i) // size], (start + i) % size
            self.pool.keys[layer][block, slot] = keys[i]
            self.pool.values[layer][block, slot] = values[i]
        if layer == self.num_layers - 1:
            self.length += len(keys)

    def release(self) -> None:
        self.pool.release(self.blocks)
        self.blocks = []

Saved to minillm/paged.py.

read gathers a sequence’s blocks into one array for the step model. That copy keeps the code simple. Production engines skip it: their attention kernel reads the blocks in place through the block table, which takes a custom kernel like the ones Section 3.1 shows how to write. The engine needs three changes. It admits waiting requests in arrival order while the pool has blocks for their prompts. When a decode step needs new blocks and the pool is empty, it preempts the newest running request: it frees that request’s blocks and puts it back at the front of the queue, to recompute its prompt and the tokens it has generated so far when it runs again. And it returns a finished request’s blocks to the pool.

Show PagedEngine 68 lines
from minillm.engine import Engine
from minillm.step import Request


class PagedEngine(Engine):
    """An engine whose requests share one pool of KV cache blocks."""

    def __init__(self, *args, pool: BlockPool, **kwargs) -> None:
        super().__init__(*args, new_cache=lambda _: PagedCache(pool), **kwargs)
        self.pool = pool
        self.preemptions = 0

    def step(self) -> None:
        # Admit waiting requests in arrival order while their tokens so far fit.
        spare, admitted = self.pool.available(), 0
        for request in self.waiting:
            if self.blocks_needed(request) > spare:
                break
            spare -= self.blocks_needed(request)
            admitted += 1
        held = [self.waiting.pop() for _ in range(len(self.waiting) - admitted)]
        held.reverse()  # pop() took them from the back
        prefill = (
            self.waiting
            and len(self.running) < self.max_batch
            and (self.continuous or not self.running)
        )
        if not prefill and not self.running and held:
            raise MemoryError(
                "a waiting request needs more blocks than the pool has"
            )
        preempted = [] if prefill else self.preempt_until_decode_fits()
        super().step()
        # Preempted requests go to the front of the queue, as in MAX.
        self.waiting.extend(held)
        self.waiting.extendleft(reversed(preempted))

    def preempt_until_decode_fits(self) -> list[Request]:
        """Frees the newest requests' blocks until every running request can
        store its next token, and returns them to recompute later."""
        preempted = []
        while self.blocks_to_grow() > self.pool.available():
            if len(self.running) == 1:
                raise MemoryError(
                    "a request needs more blocks than the pool has"
                )
            request, cache = self.running.pop()
            cache.release()
            preempted.insert(0, request)
            self.preemptions += 1
        return preempted

    def blocks_to_grow(self) -> int:
        size = self.pool.block_size
        return sum(c.length + 1 > len(c.blocks) * size for _, c in self.running)

    def blocks_needed(self, request: Request) -> int:
        # A preempted request recomputes the tokens it already generated.
        tokens = len(request.prompt) + len(request.output)
        return -(-tokens // self.pool.block_size)

    def prefill_tokens(self, request: Request, cache: PagedCache) -> list[int]:
        return request.prompt + request.output

    def release_finished(self, batch: list) -> None:
        for request, cache in batch:
            if request.done:
                cache.release()

Saved to minillm/paged.py.

Check it

Paging only changes where the keys and values are stored. Serve the workload from Section 5.2 with a paged cache and compare the tokens:

contiguous_requests = make_workload(tokenizer)
engine = Engine(step, config, DEVICE, max_batch=4)
for r in contiguous_requests:
    engine.add(r)
engine.run()

pool = BlockPool(config, num_blocks=32, block_size=16)
paged_requests = make_workload(tokenizer)
paged = PagedEngine(step, config, DEVICE, max_batch=4, pool=pool)
for r in paged_requests:
    paged.add(r)
paged.run()

same = [r.output for r in paged_requests] == [
    r.output for r in contiguous_requests
]
print("same tokens:", same)
print("blocks free after the run:", len(pool.free), "of", pool.num_blocks)
assert same and len(pool.free) == pool.num_blocks
same tokens: True
blocks free after the run: 32 of 32

The tokens match, and every block is back in the pool after the run. To watch the block tables, run the workload again on a smaller pool of 24 blocks of 8 tokens, and record every running request’s block table after each engine step:

class RecordingEngine(PagedEngine):
    def step(self) -> None:
        super().step()
        self.snapshots.append(
            [
                (requests.index(r), list(c.blocks), c.length)
                for r, c in self.running
            ]
        )


requests = make_workload(tokenizer)
recorder = RecordingEngine(
    step, config, DEVICE, max_batch=4, pool=BlockPool(config, 24, 8)
)
recorder.snapshots = []
for r in requests:
    recorder.add(r)
recorder.run()
print("preemptions:", recorder.preemptions)
preemptions: 0

On the left, each running request’s block table lists its logical blocks in order, with the fill of its last block. On the right is the pool, and each line links a logical block to the physical block that holds it. Slide through the steps:

Show pool_xy 94 lines
size, pool_x = recorder.pool.block_size, 9.0


def pool_xy(block: int) -> tuple[float, float]:
    return pool_x + block % 8, (block // 8) * 1.5


frames = []
for snapshot in recorder.snapshots:
    owner = {b: i for i, blocks, _ in snapshot for b in blocks}
    xs, ys = zip(*(pool_xy(b) for b in range(recorder.pool.num_blocks)))
    traces = [
        go.Scatter(
            x=xs,
            y=ys,
            mode="markers+text",
            marker=dict(
                symbol="square",
                size=26,
                color=[
                    viz.SERIES[owner[b]] if b in owner else viz.GRID
                    for b in range(recorder.pool.num_blocks)
                ],
            ),
            text=[str(b) for b in range(recorder.pool.num_blocks)],
            textfont=dict(color=viz.INK),
            hovertext=[
                f"physical block {b}: "
                + (f"request {owner[b]}" if b in owner else "free")
                for b in range(recorder.pool.num_blocks)
            ],
            hoverinfo="text",
        )
    ]
    for row, (i, blocks, length) in enumerate(snapshot):
        fills = [min(size, length - j * size) for j in range(len(blocks))]
        traces += [
            go.Scatter(
                x=[
                    x
                    for j, b in enumerate(blocks)
                    for x in (j, pool_xy(b)[0], None)
                ],
                y=[y for b in blocks for y in (row, pool_xy(b)[1], None)],
                mode="lines",
                line=dict(color=viz.SERIES[i], width=1),
                opacity=0.6,
                hoverinfo="skip",
            ),
            go.Scatter(
                x=[-1.2, *range(len(blocks))],
                y=[row] * (len(blocks) + 1),
                mode="markers+text",
                marker=dict(
                    symbol="square",
                    size=[0] + [26] * len(blocks),
                    color=viz.SERIES[i],
                ),
                text=[f"request {i}"]
                + ["" if f == size else f"{max(f, 0)}/{size}" for f in fills],
                textfont=dict(color=viz.INK),
                hovertext=[f"request {i}: {length} tokens"]
                + [
                    f"request {i}, logical block {j} -> physical block {b}, "
                    f"{max(f, 0)} of {size} slots filled"
                    for j, (b, f) in enumerate(zip(blocks, fills))
                ],
                hoverinfo="text",
            ),
        ]
    frames.append(traces)
fig = go.Figure(
    go.Scatter(
        x=[2.5, pool_x + 3.5],
        y=[-0.9, -0.9],
        mode="text",
        text=["block tables (logical blocks)", "pool (physical blocks)"],
        textfont=dict(color=viz.INK_SECONDARY),
        hoverinfo="skip",
    ),
    layout=dict(
        title="Block tables and the pool after each engine step",
        height=400,
        showlegend=False,
        xaxis=dict(visible=False, range=[-2.6, pool_x + 8]),
        yaxis=dict(visible=False, autorange="reversed"),
    ),
)
labels = [str(i + 1) for i in range(len(frames))]
viz.show(
    viz.with_slider(
        fig, frames, labels, "after engine step ", active=len(frames) // 3
    )
)

Blocks are handed out from wherever they are free, so a request’s blocks end up scattered across the pool, and a finished request’s blocks go straight to the next request. Only the block table records their order. This pool was big enough that no request was preempted; exercise 1 shrinks it until some are.

Why it matters: memory

Compare the memory each scheme ties up for a batch of four of these requests. A contiguous cache can’t grow in place, so when a request starts it must reserve room for the longest the request could get. That is SmolLM2’s full 8192-token context, or at best the request’s max_tokens limit, which API clients often set well above the length of the replies they get; the table below assumes 512. Paging allocates only what each sequence holds, rounded up to whole blocks of \(B\) tokens:

\[\text{blocks for } n \text{ tokens} = \left\lceil \frac{n}{B} \right\rceil\]

per_token = (
    2
    * config.num_hidden_layers
    * config.num_key_value_heads
    * config.head_dim
    * 4
)
batch = 4
actual = [len(r.prompt) + len(r.output) for r in paged_requests]
reserve_max_context = batch * config.max_position_embeddings
reserve_max_tokens = batch * (max(len(r.prompt) for r in paged_requests) + 512)
paged_tokens = sum(sorted(-(-n // 16) * 16 for n in actual)[-batch:])

rows = [
    (name, f"{tokens:,}", f"{tokens * per_token / 2**20:.1f} MiB")
    for name, tokens in [
        ("reserve full context per request", reserve_max_context),
        ("reserve max_tokens=512 per request", reserve_max_tokens),
        ("paged, 16-token blocks (4 largest)", paged_tokens),
    ]
]
header = ("scheme", "tokens reserved", "memory")
widths = [max(len(row[i]) for row in [header, *rows]) for i in range(3)]
for name, tokens, memory in [header, *rows]:
    print(f"{name:{widths[0]}}  {tokens:>{widths[1]}}  {memory:>{widths[2]}}")
scheme                              tokens reserved      memory
reserve full context per request             32,768  1440.0 MiB
reserve max_tokens=512 per request            2,080    91.4 MiB
paged, 16-token blocks (4 largest)              160     7.0 MiB
Show the code 25 lines
schemes = {
    "reserve full context": reserve_max_context,
    "reserve max_tokens=512": reserve_max_tokens,
    "paged, 16-token blocks": paged_tokens,
}
fig = go.Figure(
    go.Bar(
        y=list(schemes),
        x=[t * per_token / 2**20 for t in schemes.values()],
        orientation="h",
        marker_color=[viz.ORANGE, viz.YELLOW, viz.BLUE],
        text=[f"{t * per_token / 2**20:,.1f} MiB" for t in schemes.values()],
        textposition="outside",
        cliponaxis=False,
        hovertemplate="%{y}: %{x:,.1f} MiB<extra></extra>",
    )
)
fig.update_layout(
    title="KV cache memory for a batch of 4 of these requests",
    height=300,
    margin=dict(r=96),
    xaxis=dict(title="MiB (log scale)", type="log", dtick=1),
    yaxis=dict(autorange="reversed", showgrid=False),
)
viz.show(fig)

Saved memory becomes batch capacity. In the memory that one full-context reservation needs (8192 tokens), a paged cache fits about two hundred of these requests at roughly 40 tokens each, which is why most serving engines page their KV cache. The block size is a trade-off: bigger blocks waste more space in each sequence’s last block, and smaller ones mean longer block tables and more bookkeeping per step.

MAX Serve’s paged cache works the same way, and its attention kernels read the pages through the block table directly. Its scheduler also preempts a running request when the pool runs out of pages, and logs a message that suggests giving the cache more memory. max serve exposes the settings. --kv-cache-page-size sets the block size, 128 tokens by default. --device-memory-utilization sets the fraction of device memory the process may use, 0.95 by default, and the KV cache pool gets what is left after the weights and other allocations. The pool size and --max-batch-size together limit how many requests run at once. The handbook’s PagedAttention page describes the algorithm, and GPU memory for LLMs shows how to size the pool.

Exercises

  1. Shrink the pool to 8 blocks and rerun the paged engine. How does the engine behave when memory runs out, and what happens to time to first token?
  2. Try block sizes 4, 16, and 64. How much memory goes to partly filled last blocks in each case?
  3. This engine preempts the newest request. Change it to preempt the request that holds the most blocks instead. Which choice finishes the workload in fewer steps, and which treats long requests more fairly?

Next

Section 5.4 lets requests that share a prompt prefix share its blocks too.

5.4 Prefix caching

Open in Colab Open in Kaggle View on GitHub

Real traffic repeats itself. Requests to a chat product start with the same system prompt, and every turn of a conversation resends the turns before it. Those shared tokens get the same keys and values every time, because a token’s keys and values depend only on that token and the tokens before it. Prefix caching computes them once and shares their cache blocks between requests. With the paged cache from Section 5.3, it takes little code.

Show the setup 26 lines
import logging
import time
import warnings

import numpy as np
import plotly.graph_objects as go
from max.pipelines import AutoTokenizer

from minillm import viz
from minillm.config import MODEL_ID, LlamaConfig
from minillm.device import pick_device
from minillm.model import load_model
from minillm.paged import BlockPool, PagedEngine
from minillm.prefix import PrefixCachingEngine, PrefixCachingPool
from minillm.step import Request, StepModel, compile_step
from minillm.weights import load_checkpoint

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU

config = LlamaConfig.from_pretrained()
tokenizer = AutoTokenizer(MODEL_ID)
step = compile_step(
    load_model(config, load_checkpoint(), DEVICE, StepModel), config, DEVICE
)

Naming blocks by their content

To share a block, the engine must recognize it. A full block is identified by the tokens it holds and every token before it, since those determine its keys and values. A chained hash captures both: each block’s hash combines the previous block’s hash with the block’s own tokens. Two requests share a block exactly when their prompts agree up to the end of that block.

The pool keeps a table from hash to block, counts how many sequences use each block, and frees a block only when no sequence uses it. A block with no users whose contents are still valid stays in the table, in least-recently-used order, so a later request can still reuse it. Such blocks are evicted only when the pool runs out of free blocks.

Only full blocks are shared, and new tokens always go into a sequence’s own next block, so a shared block is never written to.

Show PrefixCachingPool, PrefixCachingEngine 100 lines
from __future__ import annotations

from collections import OrderedDict

from minillm.config import LlamaConfig
from minillm.paged import BlockPool, PagedCache, PagedEngine
from minillm.step import Request


class PrefixCachingPool(BlockPool):
    """A block pool that remembers full prompt blocks for reuse."""

    def __init__(
        self, config: LlamaConfig, num_blocks: int, block_size: int = 16
    ) -> None:
        super().__init__(config, num_blocks, block_size)
        self.refs = [0] * num_blocks
        self.by_hash: dict[int, int] = {}
        self.hash_of: dict[int, int] = {}
        self.unused: OrderedDict[int, None] = OrderedDict()  # cached, no users

    def allocate(self) -> int:
        if self.free:
            block = self.free.pop()
        elif self.unused:
            block, _ = self.unused.popitem(last=False)  # evict the oldest
            del self.by_hash[self.hash_of.pop(block)]
        else:
            raise MemoryError("the KV cache pool is full")
        self.refs[block] = 1
        return block

    def available(self) -> int:
        # Cached blocks that no request uses can be evicted and reused.
        return len(self.free) + len(self.unused)

    def release(self, blocks: list[int]) -> None:
        # Tail first: eviction then takes a prefix's last block before its
        # first, and lookup can still reuse the rest.
        for block in reversed(blocks):
            self.refs[block] -= 1
            if self.refs[block] == 0:
                if block in self.hash_of:
                    self.unused[block] = None
                else:
                    self.free.append(block)

    def block_hashes(self, tokens: list[int]) -> list[int]:
        hashes, previous = [], 0
        for start in range(
            0, len(tokens) - self.block_size + 1, self.block_size
        ):
            previous = hash(
                (previous, tuple(tokens[start : start + self.block_size]))
            )
            hashes.append(previous)
        return hashes

    def lookup(self, prompt: list[int]) -> list[int]:
        """Returns the cached blocks that start ``prompt``, and claims them.
        The last prompt token is always left to compute, since its logits
        are needed."""
        blocks = []
        for h in self.block_hashes(prompt[:-1]):
            block = self.by_hash.get(h)
            if block is None:
                break
            self.refs[block] += 1
            self.unused.pop(block, None)
            blocks.append(block)
        return blocks

    def register(self, prompt: list[int], blocks: list[int]) -> None:
        """Records a sequence's full prompt blocks for later requests."""
        # The last block may be partial; only full blocks have hashes.
        for h, block in zip(self.block_hashes(prompt), blocks, strict=False):
            if h not in self.by_hash:
                self.by_hash[h] = block
                self.hash_of[block] = h


class PrefixCachingEngine(PagedEngine):
    """A paged engine that skips computing prompt prefixes it has cached."""

    def __init__(self, *args, pool: PrefixCachingPool, **kwargs) -> None:
        super().__init__(*args, pool=pool, **kwargs)
        self.computed_prompt_tokens = 0

    def prefill_tokens(self, request: Request, cache: PagedCache) -> list[int]:
        tokens = (
            request.prompt + request.output
        )  # output is empty unless preempted
        cache.blocks = self.pool.lookup(tokens)
        cache.length = len(cache.blocks) * self.pool.block_size
        new = tokens[cache.length :]
        self.computed_prompt_tokens += len(new)
        return new

    def prefilled(self, request: Request, cache: PagedCache) -> None:
        self.pool.register(request.prompt, cache.blocks)

Saved to minillm/prefix.py.

prefill_tokens and prefilled are the two hooks the engine from Section 5.2 calls around each prefill. The base engine returns the whole prompt from the first and does nothing in the second. The step model already handles the rest: a prefill with a non-empty past is a step with several new tokens.

A workload with a shared prefix

Eight users ask different questions of the same assistant. Its system prompt is about 150 tokens long; real ones are often much longer:

Show workload 26 lines
SYSTEM = (
    "You are a helpful assistant for a travel company. Answer briefly and politely. "
    "Recommend destinations in Europe, mention the best season to visit, give one "
    "practical tip about transport, and never invent prices. "
) * 4
QUESTIONS = [
    "Where should I go in spring?",
    "Is Rome good in winter?",
    "How do I get around Paris?",
    "What about Lisbon?",
    "Best city for museums?",
    "Where can I hike?",
    "Is Berlin expensive?",
    "Suggest a beach town.",
]


def workload() -> list[Request]:
    now = time.perf_counter()
    return [
        Request(tokenizer([SYSTEM + q])["input_ids"][0], 8, arrived=now)
        for q in QUESTIONS
    ]


print("prompt tokens per request:", len(workload()[0].prompt))
prompt tokens per request: 163

Serve the workload twice, without and with prefix caching. The cached engine is a recording subclass that notes, for each request’s prefill, how many prompt tokens it computed and which blocks its block table holds, with their user counts:

Show serve, RecordingPrefixEngine 33 lines
def serve(engine) -> list[Request]:
    requests = workload()
    for r in requests:
        engine.add(r)
    start = time.perf_counter()
    engine.run()
    ttft = [(r.first_token - r.arrived) * 1000 for r in requests]
    print(
        f"{type(engine).__name__:24} {time.perf_counter() - start:5.1f} s   "
        f"TTFT median {np.median(ttft):5.0f} ms, worst {max(ttft):5.0f} ms"
    )
    return requests


class RecordingPrefixEngine(PrefixCachingEngine):
    """Records each request's computed tokens and its blocks' user counts."""

    def __init__(self, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        self.computed, self.tables = {}, {}

    def prefill_tokens(self, request: Request, cache: object) -> list[int]:
        tokens = super().prefill_tokens(request, cache)
        self.computed[id(request)] = len(tokens)
        return tokens

    def step(self) -> None:
        super().step()
        for request, cache in self.running:
            if id(request) not in self.tables:  # just prefilled
                self.tables[id(request)] = [
                    (b, self.pool.refs[b]) for b in cache.blocks
                ]
plain = serve(
    PagedEngine(step, config, DEVICE, max_batch=4, pool=BlockPool(config, 128))
)
prefix_pool = PrefixCachingPool(config, 128)
cached_engine = RecordingPrefixEngine(
    step, config, DEVICE, max_batch=4, pool=prefix_pool
)
cached = serve(cached_engine)
PagedEngine                5.1 s   TTFT median  2192 ms, worst  3499 ms
RecordingPrefixEngine      4.3 s   TTFT median  1617 ms, worst  2961 ms
total_prompt = sum(len(r.prompt) for r in cached)
print(
    f"prompt tokens computed: {cached_engine.computed_prompt_tokens} of {total_prompt} "
    f"({cached_engine.computed_prompt_tokens / total_prompt:.0%})"
)
print("same tokens:", [r.output for r in cached] == [r.output for r in plain])
assert [r.output for r in cached] == [r.output for r in plain]
assert cached_engine.computed_prompt_tokens < total_prompt
prompt tokens computed: 715 of 1291 (55%)
same tokens: True
Show the code 31 lines
computed = [cached_engine.computed[id(r)] for r in cached]
reused = [len(r.prompt) - c for r, c in zip(cached, computed)]
names = [f"request {i}: {q}" for i, q in enumerate(QUESTIONS)]
fig = go.Figure(
    [
        go.Bar(
            y=names,
            x=reused,
            name="reused from the cache",
            orientation="h",
            marker_color=viz.BLUE,
            hovertemplate="%{x} tokens reused<extra>%{y}</extra>",
        ),
        go.Bar(
            y=names,
            x=computed,
            name="computed",
            orientation="h",
            marker_color=viz.ORANGE,
            hovertemplate="%{x} tokens computed<extra>%{y}</extra>",
        ),
    ]
)
fig.update_layout(
    title="Prompt tokens per request with prefix caching",
    barmode="stack",
    height=420,
    xaxis_title="prompt tokens",
    yaxis=dict(autorange="reversed", showgrid=False),
)
viz.show(fig)

The block tables show where the reuse comes from. Each row is one request’s block table right after its prefill, each cell one block, labeled with its physical block number. Blue blocks are shared, and ×4 means four requests use the block:

Show the code 44 lines
tables = [cached_engine.tables[id(r)] for r in cached]
size = prefix_pool.block_size
width = max(len(t) for t in tables)
users = np.full((len(tables), width), np.nan)
text = [[""] * width for _ in tables]
hover = [[""] * width for _ in tables]
for i, (request, table) in enumerate(zip(cached, tables)):
    for j, (block, refs) in enumerate(table):
        users[i, j] = min(refs, 2)
        text[i][j] = f"{block}<br>×{refs}" if refs > 1 else str(block)
        piece = tokenizer.decode(request.prompt[j * size : (j + 1) * size])
        hover[i][j] = (
            f"request {i}, logical block {j}: physical block {block}, "
            f"{refs} user{'s' * (refs != 1)}<br>{piece!r}"
        )
fig = go.Figure(
    go.Heatmap(
        z=users,
        zmin=0.5,
        zmax=2.5,
        colorscale=[
            [0, viz.ORANGE],
            [0.5, viz.ORANGE],
            [0.5, viz.BLUE],
            [1, viz.BLUE],
        ],
        showscale=False,
        xgap=3,
        ygap=3,
        text=text,
        texttemplate="%{text}",
        customdata=hover,
        hovertemplate="%{customdata}<extra></extra>",
        x=[f"block {j}" for j in range(width)],
        y=[f"request {i}" for i in range(len(tables))],
    )
)
fig.update_layout(
    title="Block tables after prefill: blue blocks are shared",
    height=400,
    xaxis=dict(showgrid=False, side="top"),
    yaxis=dict(autorange="reversed", showgrid=False),
)
viz.show(fig)

The first batch computes the system prompt once per request, because nothing is cached yet: requests 0 to 3 are prefilled in the same step, so each holds its own copy of the same blocks, and only request 0’s copy enters the table. From the second batch on, each request finds the system prompt’s full blocks in the table, the blue blocks four requests share, and computes only the rest: the system prompt’s last, partly filled block plus its own question. That brings the prompt tokens computed down to 715 of 1,291, or 55%. The outputs are identical, because the reused keys and values are exactly the ones the request would have computed.

The wall-clock gain is modest here: the whole workload took about 4.3 s with prefix caching against 5.1 s without, on this page’s machine (an Apple-silicon Mac with a Metal GPU), because prefilling about 160 prompt tokens on a 135M model is only part of the workload’s time. Treat these numbers as an example: yours depend on your hardware and vary from run to run. The saving grows with the shared prefix. A multi-turn chat resends the whole conversation so far on every turn, often thousands of tokens, and an agent resends long tool descriptions on every call, so for them prefix caching removes most of the prefill.

MAX Serve enables prefix caching by default; --no-enable-prefix-caching turns it off. The same idea extends across servers: a router can send each request to the replica that already holds its prefix. The handbook covers prefix caching and inference routing.

Exercises

  1. Send the workload twice through the same engine. What fraction of prompt tokens does the second pass compute?
  2. Shrink the pool so cached blocks must be evicted. Which requests lose their prefix, and why does least-recently-used eviction suit chat traffic?
  3. Why does lookup hash prompt[:-1] instead of the whole prompt? What would break otherwise?

Next

Section 5.5 speeds up decoding instead: speculative decoding can produce several tokens per step of the large model.

5.5 Speculative decoding

Open in Colab Open in Kaggle View on GitHub

Decoding produces one token per step of the model, and at small batch sizes each step is memory-bound: Section 4.3 showed that a step for several tokens costs little more than a step for one. Speculative decoding uses that spare capacity. A small, fast draft model guesses the next few tokens, and the large target model checks all the guesses in one step. Each guess the target agrees with is an extra token from that one target step, and with greedy decoding the output is exactly what the target alone would produce.

This section implements it with SmolLM2-135M drafting for SmolLM2-360M, a larger model from the same family, and measures when it pays off.

Show the setup 19 lines
import logging
import time
import warnings

import numpy as np
import plotly.graph_objects as go
from max.pipelines import AutoTokenizer

from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.model import load_model
from minillm.step import ContiguousCache, StepModel, compile_step, run_step
from minillm.weights import load_checkpoint

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU
TARGET_ID = "HuggingFaceTB/SmolLM2-360M-Instruct"

Two changes to the step model

Verification needs the target’s prediction after every guessed token, not only the last, so the target’s step returns logits for all positions. When the target rejects a guess, the keys and values computed for the rejected tokens must be dropped, so the cache gets a truncate method.

Show AllLogitsStepModel, TruncatableCache, build 30 lines
class AllLogitsStepModel(StepModel):
    def forward(self, tokens, positions, mask, past):
        x = self.model.embed_tokens(tokens)
        new_kv = []
        for layer, layer_past in zip(self.model.layers, past, strict=True):
            x, kv = layer(x, positions, mask, layer_past)
            new_kv.append(kv)
        return self.lm_head(self.model.norm(x)), new_kv


class TruncatableCache(ContiguousCache):
    def truncate(self, length: int) -> None:
        for entry in self.layers:
            entry[0], entry[1] = entry[0][:length], entry[1][:length]


def build(model_id: str):
    config = LlamaConfig.from_pretrained(model_id)
    model = load_model(
        config, load_checkpoint(model_id), DEVICE, AllLogitsStepModel
    )
    return config, compile_step(model, config, DEVICE)


draft_config, draft = build(MODEL_ID)
target_config, target = build(TARGET_ID)
print(
    f"draft: {draft_config.num_hidden_layers} layers x {draft_config.hidden_size}, "
    f"target: {target_config.num_hidden_layers} layers x {target_config.hidden_size}"
)
draft: 30 layers x 576, target: 32 layers x 960

Both models share a tokenizer, which this kind of speculative decoding requires: the target must read the draft’s token IDs as the same tokens.

The baseline: the target alone

tokenizer = AutoTokenizer(MODEL_ID)
prompt = tokenizer(["The three primary colors are"])["input_ids"][0]
N = 48


def target_alone() -> tuple[list[int], float]:
    cache, ids = TruncatableCache(target_config), list(prompt)
    start = time.perf_counter()
    logits = run_step(target, [cache], [ids], DEVICE)
    while True:
        ids.append(int(logits[0, -1].argmax()))
        if len(ids) - len(prompt) == N:
            break
        logits = run_step(target, [cache], [[ids[-1]]], DEVICE)
    return ids[len(prompt) :], time.perf_counter() - start


reference, reference_time = target_alone()
print(f"{N / reference_time:.1f} tokens/s")
print(tokenizer.decode(reference))
2.5 tokens/s
 red, blue, and yellow. These colors can be combined in various ways to create other colors. For example, by mixing red and blue, you get purple. Similarly, by mixing red and yellow, you get orange. By mixing blue

Draft, then verify

Speculative decoding: a small draft model proposes k tokens one at a time; the target model checks all k guesses plus one more position in a single step; the matching prefix of the guesses is kept, and the target's own token replaces the first mismatch or follows a fully accepted draft.Speculative decoding: a small draft model proposes k tokens one at a time; the target model checks all k guesses plus one more position in a single step; the matching prefix of the guesses is kept, and the target's own token replaces the first mismatch or follows a fully accepted draft.

Each round:

  1. The draft model generates k tokens greedily, one step each. Its first step also feeds in any accepted tokens it hasn’t seen yet.
  2. The target runs one step over the last accepted token plus the k guesses, and returns its own greedy choice after each of them.
  3. Guesses are accepted from the left for as long as they match the target’s choice. At the first mismatch, the target’s choice replaces the guess. If all k match, the target’s choice after the last one comes free. So every round yields between 1 and k + 1 tokens.
  4. Both caches are truncated to the accepted tokens, which discards the keys and values of rejected guesses.

Because every emitted token is the target’s own greedy choice, the output is identical to the target alone. Sampling versions accept or reject each guess with a probability test that keeps the output distributed exactly as the target’s own samples would be.

Show speculative 42 lines
def speculative(
    k: int, log: list | None = None
) -> tuple[list[int], float, int, int]:
    target_cache, draft_cache = (
        TruncatableCache(target_config),
        TruncatableCache(draft_config),
    )
    ids = list(prompt)
    rounds = accepted = 0
    start = time.perf_counter()
    logits = run_step(target, [target_cache], [ids], DEVICE)
    ids.append(int(logits[0, -1].argmax()))
    if log is not None:  # for the figure below: where each token came from
        log.append(("prefill", ids[-1]))
    while len(ids) - len(prompt) < N:
        guesses = []
        logits = run_step(
            draft, [draft_cache], [ids[draft_cache.length :]], DEVICE
        )
        for i in range(k):
            guesses.append(int(logits[0, -1].argmax()))
            if i < k - 1:
                logits = run_step(draft, [draft_cache], [[guesses[-1]]], DEVICE)
        checks = run_step(
            target,
            [target_cache],
            [ids[target_cache.length :] + guesses],
            DEVICE,
        )[0]
        choices = checks[-(k + 1) :].argmax(-1)
        n = 0
        while n < k and guesses[n] == int(choices[n]):
            n += 1
        ids += guesses[:n] + [int(choices[n])]
        if log is not None:
            log += [("accepted", t) for t in guesses[:n]]
            log.append(("bonus" if n == k else "corrected", int(choices[n])))
        rounds += 1
        accepted += n
        target_cache.truncate(len(ids) - 1)
        draft_cache.truncate(min(draft_cache.length, len(ids) - 1))
    return ids[len(prompt) :][:N], time.perf_counter() - start, rounds, accepted
print(
    f"{'k':>2} {'same output':>11} {'target steps':>12} {'accepted':>9} {'tokens/s':>9}"
)
print(f"{'-':>2} {'-':>11} {N:>12} {'-':>9} {N / reference_time:>9.1f}")
results = {}
for k in [2, 4, 6]:
    out, seconds, rounds, accepted = speculative(k)
    results[k] = (rounds, accepted)
    print(
        f"{k:>2} {str(out == reference):>11} {rounds:>12} {accepted / (rounds * k):>9.0%} {N / seconds:>9.1f}"
    )
    assert out == reference and rounds < N
 k same output target steps  accepted  tokens/s
 -           -           48         -       2.5
 2        True           21       64%       2.2
 4        True           15       58%       1.4
 6        True           15       41%       1.2

Look inside the run with k = 4. Each chip is one output token. The first, in yellow, comes from the target’s prefill. Blue chips are draft guesses the target accepted, orange ones are the target’s corrections where a guess was wrong, and green ones are the extra token a round earns when all four guesses are right. Hover over a chip for its round:

Show the code 23 lines
log = []
speculative(4, log)
log = log[:N]
kinds = {
    "prefill": viz.CHIP_COLORS[3],
    "accepted": viz.CHIP_COLORS[0],
    "corrected": viz.CHIP_COLORS[1],
    "bonus": viz.CHIP_COLORS[2],
}
titles, current = [], 0
for kind, _ in log:
    titles.append(
        "the target's prefill"
        if kind == "prefill"
        else f"round {current}: {kind}"
    )
    current += kind != "accepted"
viz.tokens(
    [tokenizer.decode([t]) for _, t in log],
    [t for _, t in log],
    colors=[kinds[kind] for kind, _ in log],
    titles=titles,
)
·red,·blue,·and·yellow.·These·colors·can·be·combined·in·various·ways·to·create·other·colors.·For·example,·by·mixing·red·and·blue,·you·get·purple.·Similarly,·by·mixing·red·and·yellow,·you·get·orange.·By·mixing·blue

The output matches the target alone, and the target runs far fewer steps: 15 to 21 verification rounds instead of 47 decode steps. Even so, on this page’s machine (an Apple-silicon Mac with a Metal GPU) every k was slower than the target alone, at about 1 to 2 tokens/s against about 2.5. Treat these numbers as an example: yours depend on your hardware and vary from run to run.

A timeline shows where the time goes. Run the target alone and the speculative loop with k = 4 once more, and time every call to run_step. Each bar is one step, blue for the target and orange for the draft; hover over a target step in the second row to see how many tokens its round produced:

Show run_step 19 lines
events, untimed = [], run_step


def run_step(step_fn, caches, new_tokens, device):
    start = time.perf_counter()
    logits = untimed(step_fn, caches, new_tokens, device)
    model = "draft" if step_fn is draft else "target"
    events.append((model, start, time.perf_counter(), len(new_tokens[0])))
    return logits


try:
    target_alone()
    baseline_events, events = events, []
    round_log = []
    speculative(4, round_log)
    spec_events = events
finally:
    run_step = untimed
Show the code 68 lines
# Tokens each round produced: its accepted guesses plus the target's own token.
per_round, accepted = [], 0
for kind, _ in round_log[1:]:
    if kind == "accepted":
        accepted += 1
    else:
        per_round.append(accepted + 1)
        accepted = 0
rows = {"target alone": baseline_events, "speculative, k = 4": spec_events}
traces = {
    "target": ([], [], [], [], viz.BLUE),
    "draft": ([], [], [], [], viz.ORANGE),
}
ends = []
for row, row_events in rows.items():
    t0 = row_events[0][1]
    verify = 0
    for index, (model, start, end, n) in enumerate(row_events):
        y, base, width, hover, _ = traces[model]
        y.append(row)
        base.append(start - t0)
        width.append(end - start)
        if index == 0:
            what = f"target prefill, {n} tokens"
        elif model == "draft":
            what = "draft step, 1 token"
        elif row == "target alone":
            what = "target decode step, 1 token"
        else:
            what = f"target verifies {n} tokens; round {verify + 1} yields {per_round[verify]}"
            verify += 1
        hover.append(f"{what}: {(end - start) * 1000:.0f} ms")
    ends.append((row, row_events[-1][2] - t0))
fig = go.Figure(
    [
        go.Bar(
            y=y,
            x=width,
            base=base,
            orientation="h",
            name=f"{model} step",
            marker=dict(color=color, line=dict(color=viz.SURFACE, width=1)),
            hovertext=hover,
            hoverinfo="text",
        )
        for model, (y, base, width, hover, color) in traces.items()
    ]
    + [
        go.Scatter(
            x=[end for _, end in ends],
            y=[row for row, _ in ends],
            mode="text",
            text=[f"  {N} tokens in {end:.1f} s" for _, end in ends],
            textposition="middle right",
            showlegend=False,
            hoverinfo="skip",
        )
    ]
)
fig.update_layout(
    title=f"Every step of generating {N} tokens, in wall-clock time",
    barmode="overlay",
    height=300,
    xaxis=dict(title="seconds", rangemode="tozero"),
    yaxis=dict(autorange="reversed", showgrid=False),
    margin=dict(r=140),
)
viz.show(fig)

In the second row, an orange draft step is about as wide as a blue target step, and a target step that verifies five tokens costs about the same as one that decodes a single token, as the cost model below assumes. A round spends four draft steps to earn between one and five tokens, so the steps the target saves are paid back, and more, by the draft.

When it pays off

A round costs k draft steps plus one target step. If a draft step costs a fraction \(c\) of a target step, and a round yields \(\bar{n}\) tokens on average, the speedup over the target alone is

\[\text{speedup} \approx \frac{\bar{n}}{1 + k\,c}\]

This assumes the target verifies k + 1 tokens in about the time of a one-token step. Measure \(c\) for these two models:

def ms_per_decode_step(step, config) -> float:
    cache = TruncatableCache(config)
    run_step(step, [cache], [prompt], DEVICE)
    start = time.perf_counter()
    for _ in range(10):
        run_step(step, [cache], [[42]], DEVICE)
    return (time.perf_counter() - start) / 10 * 1000


c = ms_per_decode_step(draft, draft_config) / ms_per_decode_step(
    target, target_config
)
print(f"c = {c:.2f}")
for k, (rounds, accepted) in results.items():
    tokens_per_round = (accepted + rounds) / rounds
    print(
        f"k={k}: {tokens_per_round:.2f} tokens per round -> predicted speedup {tokens_per_round / (1 + k * c):.2f}x"
    )
c = 0.69
k=2: 2.29 tokens per round -> predicted speedup 0.96x
k=4: 3.33 tokens per round -> predicted speedup 0.89x
k=6: 3.47 tokens per round -> predicted speedup 0.68x

Plot the predicted speedup against the draft’s relative cost \(c\), using the acceptance measured for each k. Above the dashed line, speculation is faster than the target alone; the dotted line marks the \(c\) of these two models:

Show the code 32 lines
cs = np.logspace(-2, 0, 60)
fig = go.Figure()
for (k, (rounds, accepted)), color in zip(results.items(), viz.SERIES):
    per_round = (accepted + rounds) / rounds
    fig.add_trace(
        go.Scatter(
            x=cs,
            y=per_round / (1 + k * cs),
            name=f"k = {k}",
            line_color=color,
            hovertemplate=f"k = {k}, c = %{{x:.2f}}: %{{y:.2f}}x<extra></extra>",
        )
    )
fig.add_hline(y=1, line=dict(color=viz.INK_SECONDARY, dash="dash"))
fig.add_vline(
    x=c,
    line=dict(color=viz.INK_SECONDARY, dash="dot"),
    annotation_text=f"these models, c = {c:.2f}",
    annotation_position="top left",
)
fig.update_layout(
    title="Predicted speedup of speculative decoding",
    height=420,
    xaxis=dict(
        title="draft step cost / target step cost, c (log scale)",
        type="log",
        dtick=1,
        range=[-2.05, 0.05],
    ),
    yaxis=dict(title="speedup over the target alone", rangemode="tozero"),
)
viz.show(fig)

Here a draft step costs most of a target step, far more than the draft’s 2.7 times fewer parameters suggest. The two models have almost the same depth, 30 and 32 layers, and at this size much of each layer’s time is fixed per-step work that doesn’t shrink with the layer’s width. Part of that fixed work is this engine’s own: run_step packs and copies every layer’s keys and values on the host at each step, so the measured \(c\) includes that host-side packing too. So the k draft steps in a round cost more than the target steps they save, and the cost model predicts speculation at or below the speed of the target alone. The measured runs were slower still, but the conclusion is the same. A draft’s depth matters as much as its parameter count.

Speculative decoding pays off when the draft is cheap (small \(c\)), its guesses are usually accepted (large \(\bar{n}\)), and verifying k + 1 tokens costs about one target step. A cheap draft can be a 1B model drafting for a 70B target, or a small head trained on the target’s own hidden states, as in Medusa, EAGLE, and multi-token prediction (MTP). Cheap verification holds for memory-bound decoding on GPUs and less so on a busy CPU.

With \(c \approx 0.05\) and the acceptance measured above, the same code would run about two to three times as fast as the target alone, as the chart shows. MAX Serve’s speculative decoding uses drafts that read the target model’s hidden states, such as EAGLE and MTP, selected with --speculative-method. The handbook’s speculative decoding page compares the variants.

Exercises

  1. Plug different \(c\) values into the formula with the measured acceptance rates. For which \(c\) does each k break even?
  2. Change the prompt to something more predictable, such as counting from 1 to 50. How do acceptance and the best k change?
  3. Make k adaptive: increase it after a fully accepted round and decrease it after an early rejection. Does it help?

What you built

Chapter 5 is complete. Your engine schedules requests with continuous batching, stores keys and values in a paged cache, and shares cached prefixes between requests, and a separate loop speculates with a draft model for a single sequence. MAX Serve is built from the same parts, with the kernels and scheduling that make them fast on GPUs. Chapter 6 returns to the model and extends it to newer architectures.

6. Bigger models

Your SmolLM2 is a plain Llama. Newer models change a few parts, each for a reason. This chapter extends your model to them, and everything still runs on a laptop.

SectionYou learn
6.1 Qwen3: normalized queries and keysBringing up Qwen3-0.6B by subclassing your Llama, and what QK-norm does to attention scores.
6.2 Gemma 3: sliding-window attentionBringing up Gemma 3 270M: sliding-window attention, GeGLU, and extra normalization.
6.3 Mixture of expertsA Qwen3-MoE layer checked against Hugging Face, routing, and expert parallelism on a simulated mesh.

Related handbook pages: choosing a model, quantization, and parallelism strategies.

6.1 Qwen3: normalized queries and keys

Open in Colab Open in Kaggle View on GitHub

Most open models are close relatives of the Llama architecture you built in Chapter 1. To bring up a new one, you find what changed and why. This section adds Qwen3, here the 0.6-billion-parameter model Qwen/Qwen3-0.6B. It subclasses your Llama classes, overrides only what differs, and verifies the result with the tools from Chapter 4.

Show the setup 26 lines
import logging
import warnings

import numpy as np
import plotly.graph_objects as go
import torch
from max.driver import CPU
from max.dtype import DType
from max.graph import TensorType
from max.pipelines import AutoTokenizer
from max.tensor import Tensor
from transformers import AutoModelForCausalLM

from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.layers import apply_rope, rope_cos_sin
from minillm.model import load_model
from minillm.sampling import generate
from minillm.testing import compare_outputs
from minillm.weights import load_checkpoint

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU
QWEN3 = "Qwen/Qwen3-0.6B"

What changed

Compare the two configs:

smol, qwen = (
    LlamaConfig.from_pretrained(MODEL_ID),
    LlamaConfig.from_pretrained(QWEN3),
)
for field in [
    "vocab_size",
    "hidden_size",
    "num_hidden_layers",
    "num_attention_heads",
    "num_key_value_heads",
    "head_dim",
    "intermediate_size",
    "rope_theta",
    "rms_norm_eps",
]:
    if field == "vocab_size":
        print(f"{'':22} {'SmolLM2':>10} {'Qwen3':>10}")
    print(f"{field:22} {getattr(smol, field):>10} {getattr(qwen, field):>10}")
                          SmolLM2      Qwen3
vocab_size                  49152     151936
hidden_size                   576       1024
num_hidden_layers              30         28
num_attention_heads             9         16
num_key_value_heads             3          8
head_dim                       64        128
intermediate_size            1536       3072
rope_theta                 100000    1000000
rms_norm_eps                1e-05      1e-06

Four differences matter:

  • A larger vocabulary, 151,936 tokens, which covers many languages and code. The embedding alone holds 156M parameters, about a quarter of the model’s 596M.
  • An explicit head_dim: 16 heads of 128 dimensions make 2048, wider than the 1024-dimension hidden state. q_proj projects up, and o_proj projects back down. The config class from Section 1.3 reads head_dim when a model sets it.
  • A larger RoPE base, 1,000,000, so the rotations turn more slowly and positions stay distinguishable over longer contexts.
  • QK-norm: an RMSNorm on every query and key head, before RoPE.

The first three are config values that your code already handles. QK-norm needs new code.

QK-norm

Attention scores are dot products of queries and keys. During training, nothing stops some heads’ queries and keys from growing until a few scores dominate every softmax, which can destabilize the training of large models. Normalizing the queries and keys, with a learned scale per dimension, bounds the scores. Qwen3 and Gemma 3 normalize each head separately; OLMo 2 normalizes the whole query and key projections.

For each head’s query \(q\) (and likewise each key), with \(d\) = head_dim and a learned weight \(w\) of size \(d\):

\[\hat{q} = \frac{q}{\sqrt{\tfrac{1}{d}\sum_j q_j^2 + \epsilon}} \odot w\]

The division gives every query the same root mean square, so the length of \(\hat q\) depends on \(w\) and on \(q\)’s direction, not on \(q\)’s size. A score \(\hat q \cdot \hat k\) is then bounded however large \(q\) and \(k\) get. In code, QK-norm is two RMSNorms of size head_dim inside attention, applied to each head after the projection and before RoPE. The other classes only swap in the new attention:

Show Qwen3Attention, Qwen3DecoderLayer, Qwen3Model, Qwen3ForCausalLM 58 lines
from __future__ import annotations

from max.graph import Dim
from max.nn import ModuleList, RMSNorm
from max.tensor import Tensor

from minillm.attention import Attention
from minillm.config import LlamaConfig
from minillm.layers import apply_rope, rope_cos_sin
from minillm.model import DecoderLayer, LlamaForCausalLM, LlamaModel


class Qwen3Attention(Attention):
    """Llama attention with an RMSNorm on each query and key head."""

    def __init__(self, config: LlamaConfig) -> None:
        super().__init__(config)
        self.q_norm = RMSNorm(self.head_dim, eps=config.rms_norm_eps)
        self.k_norm = RMSNorm(self.head_dim, eps=config.rms_norm_eps)

    def project(
        self, x: Tensor, start: int | Dim | Tensor
    ) -> tuple[Tensor, Tensor, Tensor]:
        batch, seq, _ = x.shape
        q = self.q_proj(x).reshape([batch, seq, self.num_heads, self.head_dim])
        k = self.k_proj(x).reshape(
            [batch, seq, self.num_kv_heads, self.head_dim]
        )
        v = self.v_proj(x).reshape(
            [batch, seq, self.num_kv_heads, self.head_dim]
        )
        q, k = self.q_norm(q), self.k_norm(k)
        cos, sin = rope_cos_sin(
            self.head_dim, self.rope_theta, start, seq, x.mesh
        )
        return apply_rope(q, cos, sin), apply_rope(k, cos, sin), v


class Qwen3DecoderLayer(DecoderLayer):
    def __init__(self, config: LlamaConfig) -> None:
        super().__init__(config)
        self.self_attn = Qwen3Attention(config)


class Qwen3Model(LlamaModel):
    def __init__(self, config: LlamaConfig) -> None:
        super().__init__(config)
        self.layers = ModuleList(
            [Qwen3DecoderLayer(config) for _ in range(config.num_hidden_layers)]
        )


class Qwen3ForCausalLM(LlamaForCausalLM):
    """Qwen3: a Llama model whose attention normalizes queries and keys."""

    def __init__(self, config: LlamaConfig) -> None:
        super().__init__(config)
        self.model = Qwen3Model(config)

Saved to minillm/qwen3.py.

The new parameters are named q_norm and k_norm, as in Qwen3’s checkpoint, so load_model from Section 1.6 loads it unchanged. load_model builds the model inside F.lazy(), so the Llama layers that the subclasses replace are never allocated.

model = load_model(qwen, load_checkpoint(QWEN3), DEVICE, Qwen3ForCausalLM)
compiled = model.compile(
    TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)
n_params = sum(p.num_elements() for _, p in model.parameters)
print(
    f"{n_params / 1e6:.0f}M values, including the untied copy of the embedding"
)
752M values, including the untied copy of the embedding

To see what QK-norm guarantees on real activations, run layer 0’s attention on a sentence: its queries and keys, with RoPE, the scores \(q \cdot k / \sqrt{d}\), and their softmax. Then scale the layer’s input up, which scales queries and keys alike, as projection weights that grew during training would. The entropy of a softmax row measures how spread out a query’s attention is: 0 when one key takes all the weight, \(\log n\) when \(n\) keys share it equally.

Show softmax, entropy_per_head 43 lines
attn = model.model.layers[0].self_attn
ids = AutoTokenizer(QWEN3)(
    ["Normalizing queries and keys keeps attention scores in a stable range."]
)["input_ids"][0]
seq = len(ids)
h = model.model.layers[0].input_layernorm(
    model.model.embed_tokens(Tensor(np.array([ids]), device=DEVICE))
)
cos, sin = rope_cos_sin(attn.head_dim, attn.rope_theta, 0, seq, DEVICE)
causal = np.triu(np.full((seq, seq), -np.inf), 1)


def softmax(z: np.ndarray) -> np.ndarray:
    e = np.exp(z.astype(np.float64) - z.max(-1, keepdims=True))
    return e / e.sum(-1, keepdims=True)


def entropy_per_head(scale: float, qk_norm: bool) -> np.ndarray:
    """Mean attention entropy of each query head, in nats."""
    x = h * scale
    q = attn.q_proj(x).reshape([1, seq, attn.num_heads, attn.head_dim])
    k = attn.k_proj(x).reshape([1, seq, attn.num_kv_heads, attn.head_dim])
    if qk_norm:
        q, k = attn.q_norm(q), attn.k_norm(k)
    q = apply_rope(q, cos, sin).to_numpy()[0]
    k = apply_rope(k, cos, sin).to_numpy()[0]
    k = np.repeat(k, attn.num_heads // attn.num_kv_heads, axis=1)
    scores = np.einsum("qhd,khd->hqk", q, k) / np.sqrt(attn.head_dim)
    p = softmax(scores + causal)
    # Position 0 sees only itself, so its entropy is always 0.
    return -(p * np.log(p + 1e-30)).sum(-1)[:, 1:].mean(-1)


entropy = {
    (scale, qk_norm): entropy_per_head(scale, qk_norm)
    for scale in [1, 3, 10]
    for qk_norm in [False, True]
}
for (scale, qk_norm), e in entropy.items():
    label = "with" if qk_norm else "without"
    print(
        f"input x{scale:<3} {label:7} QK-norm: mean entropy {e.mean():.2f} nats"
    )
input x1   without QK-norm: mean entropy 1.87 nats
input x1   with    QK-norm: mean entropy 0.77 nats
input x3   without QK-norm: mean entropy 1.39 nats
input x3   with    QK-norm: mean entropy 0.77 nats
input x10  without QK-norm: mean entropy 0.18 nats
input x10  with    QK-norm: mean entropy 0.77 nats
Show the code 26 lines
steps = [
    [
        go.Bar(
            x=np.arange(attn.num_heads),
            y=entropy[scale, qk_norm],
            name=f"{'after' if qk_norm else 'before'} QK-norm",
            marker_color=viz.BLUE if qk_norm else viz.ORANGE,
            hovertemplate="head %{x}: %{y:.2f} nats<extra>%{fullData.name}</extra>",
        )
        for qk_norm in [False, True]
    ]
    for scale in [1, 3, 10]
]
fig = go.Figure(
    layout=dict(
        title="Attention entropy per head, Qwen3 layer 0",
        height=460,
        barmode="group",
        xaxis=dict(title="query head", dtick=1),
        yaxis=dict(
            title="mean entropy (nats)",
            range=[0, 1.1 * max(e.max() for e in entropy.values())],
        ),
    )
)
viz.show(viz.with_slider(fig, steps, ["x1", "x3", "x10"], "input scaled "))

Move the slider. Without the norm (orange), the attention depends on the scale. Scaling the input by 10 makes queries and keys 10 times longer and the scores 100 times larger, and the softmax collapses: the orange bars fall close to 0, so each query attends to almost a single key. At x1 the orange bars sit higher than the blue ones, because Qwen3’s raw queries and keys are small; the trained model always normalizes them. The blue bars don’t move at all. After q_norm and k_norm, a query’s or key’s length depends only on its direction and the learned per-dimension scale, so no input can make a score blow up.

Verify it

Compare with Hugging Face’s PyTorch Qwen3 on a few prompts, using the metrics from Section 4.2:

tokenizer = AutoTokenizer(QWEN3)
reference = AutoModelForCausalLM.from_pretrained(
    QWEN3, torch_dtype=torch.float32
).eval()


prompts = {
    "english": "The capital of France is",
    "code": "def add(a, b):\n    return",
    "french": "Le chat est assis sur",
}
for name, text in prompts.items():
    ids = tokenizer([text])["input_ids"][0]
    ours = compiled(Tensor(np.array([ids]), device=DEVICE)).to_numpy()[0]
    with torch.no_grad():
        theirs = reference(torch.tensor([ids])).logits[0].numpy()
    r = compare_outputs(ours, theirs, atol=1e-3, rtol=1e-3)
    p, q = softmax(theirs), softmax(ours)
    kl = float((p * (np.log(p + 1e-30) - np.log(q + 1e-30))).sum(-1).mean())
    top1 = (ours.argmax(-1) == theirs.argmax(-1)).mean()
    print(
        f"{name:8} max err {r.max_abs_err:.1e}  KL {kl:.1e}  top-1 {top1:.0%}"
    )
    assert top1 >= 0.99 and kl < 1e-6
english  max err 1.4e-04  KL 1.0e-10  top-1 100%
code     max err 1.1e-04  KL 1.9e-11  top-1 100%
french   max err 8.5e-05  KL 7.8e-11  top-1 100%
def next_logits(ids: list[int]) -> Tensor:
    return compiled(Tensor(np.array([ids]), device=DEVICE))[0, -1]


prompt = tokenizer(["The three primary colors are"])["input_ids"][0]
print(
    tokenizer.decode(prompt + generate(next_logits, prompt, 24, temperature=0))
)
The three primary colors are red, blue, and yellow. The three secondary colors are red and blue, blue and yellow, and red and yellow

That is the whole bring-up: about 40 lines of model code for a new architecture, checked against the reference to rounding error. MAX’s own Qwen3 implementation serves the same model with a paged KV cache and continuous batching, the techniques that Chapter 5 builds:

max serve --model Qwen/Qwen3-0.6B --quantization-encoding float32 --no-prefer-module-v3

--no-prefer-module-v3 selects MAX’s text-generation implementation of Qwen3. By default MAX prefers an architecture’s module-based implementation (Section 7.1 explains this preference), and while the only module-based one registered under Qwen3’s architecture name is the embedding model, max serve picks it for this checkpoint and fails to start.

Qwen3 is also a reasoning model: by default it thinks step by step inside <think> tags before it answers, which costs tokens and improves answers to hard questions. Its chat template has an enable_thinking switch that turns this off. The handbook’s choosing the right model page discusses that trade-off.

Exercises

  1. Add a KV cache to Qwen3: subclass CachedAttention from Section 1.8 with the same q_norm and k_norm, and check it against the full model.
  2. Remove QK-norm: copy load_model from Section 1.6, build LlamaForCausalLM instead of Qwen3ForCausalLM, and pass strict=False to load_state_dict so that it skips the q_norm and k_norm weights. Rerun the comparison. What happens to the output?
  3. Count the parameters in the embedding, the attention projections, and the MLPs. How do the proportions compare with SmolLM2’s?

Next

Section 6.2 adds sliding-window attention with Gemma 3, and Section 6.3 builds a mixture of experts.

6.2 Gemma 3: sliding-window attention

Open in Colab Open in Kaggle View on GitHub

Attention’s cost grows with the square of the context, and the KV cache grows linearly with it. Gemma 3 reduces both with sliding-window attention: most of its layers look only at a recent window of tokens (512 in the 270M model), and every sixth layer looks at all of them. This section brings up Gemma 3 270M, which differs from Llama in more places than Qwen3 did in Section 6.1, and verifies it on a prompt long enough for the window to matter.

Gemma’s weights are gated: accept the license on the model page and make a Hugging Face token available (on Colab, as the HF_TOKEN secret).

Show the setup 27 lines
import logging
import math
import warnings
from dataclasses import dataclass

import numpy as np
import plotly.graph_objects as go
import torch
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.graph import TensorType
from max.pipelines import AutoTokenizer
from max.tensor import Tensor
from plotly.subplots import make_subplots
from transformers import AutoModelForCausalLM

from minillm import viz
from minillm.device import pick_device
from minillm.model import load_model
from minillm.testing import compare_outputs
from minillm.weights import load_checkpoint

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU
GEMMA = "google/gemma-3-270m-it"

What changed

Gemma 3’s config.json has fields that Llama’s doesn’t. A config subclass adds them, and from_pretrained reads every dataclass field it finds:

Show Gemma3Config 29 lines
from __future__ import annotations

import math
from dataclasses import dataclass, replace

from max import functional as F
from max.driver import Device
from max.dtype import DType
from max.graph import Dim
from max.nn import Embedding, GemmaRMSNorm, Linear, Module, ModuleList
from max.nn.kernels import flash_attention
from max.tensor import Tensor
from max.tensor.sharding import DeviceMesh

from minillm.attention import Attention, causal_mask
from minillm.config import LlamaConfig
from minillm.layers import MLP, apply_rope, rope_cos_sin


@dataclass(frozen=True)
class Gemma3Config(LlamaConfig):
    sliding_window: int = 512
    layer_types: tuple[str, ...] = ()
    rope_local_base_freq: float = 10000.0
    query_pre_attn_scalar: float = 256.0

    def __post_init__(self) -> None:
        super().__post_init__()
        object.__setattr__(self, "layer_types", tuple(self.layer_types))

Saved to minillm/gemma3.py.

config = Gemma3Config.from_pretrained(GEMMA)
for field in [
    "vocab_size",
    "hidden_size",
    "num_hidden_layers",
    "num_attention_heads",
    "num_key_value_heads",
    "head_dim",
    "intermediate_size",
    "rope_theta",
    "rope_local_base_freq",
    "sliding_window",
    "query_pre_attn_scalar",
    "rms_norm_eps",
]:
    print(f"{field:22} {getattr(config, field):>10}")
pattern = "".join(
    "G" if t == "full_attention" else "s" for t in config.layer_types
)
print(f"{'layer types':22} {pattern:>10}  (s = sliding, G = global)")
vocab_size                 262144
hidden_size                   640
num_hidden_layers              18
num_attention_heads             4
num_key_value_heads             1
head_dim                      256
intermediate_size            2048
rope_theta              1000000.0
rope_local_base_freq      10000.0
sliding_window                512
query_pre_attn_scalar         256
rms_norm_eps                1e-06
layer types            sssssGsssssGsssssG  (s = sliding, G = global)

Compared with your Llama, Gemma 3 changes six things:

PartLlamaGemma 3
Attention spanevery layer sees all past tokens5 of 6 layers see only the last 512
RoPE baseone, 100,000 for SmolLM210,000 for windowed layers, 1,000,000 for global ones
Norms per layer2, before attention and before the MLP4, before and after each
RMSNormx / rms(x) * wx / rms(x) * (1 + w) (max.nn.GemmaRMSNorm)
MLP activationSiLU (SwiGLU)tanh-approximated GELU (GeGLU)
Embeddingas storedmultiplied by \(\sqrt{\text{hidden size}}\)

It also has QK-norm, as in Qwen3, and scales attention scores by one over the square root of query_pre_attn_scalar instead of the head dimension (both are 256 in this model). Its vocabulary has 262,144 tokens, so the embedding holds 168M of the model’s 270M parameters.

Sliding-window attention

A windowed layer adds one more condition to the causal mask: query i may see key j only if \(i - j < 512\). F.band_part builds that band. Each layer reads its own RoPE base and window from the config.

The mask uses a large finite number instead of \(-\infty\), for the reason Section 5.1 gave: a query more than 512 tokens in has its whole first block of keys hidden, and a blockwise attention kernel computes \(e^{-\infty + \infty}\) = NaN for such a row.

Show sliding_window_mask, Gemma3Attention 56 lines
def sliding_window_mask(
    q_len: int | Dim,
    kv_len: int | Dim,
    window: int,
    device: Device | DeviceMesh,
) -> Tensor:
    """Like ``causal_mask``, but each query also can't see keys ``window`` or
    more positions behind it."""
    # A finite value, not -inf: a window hides a query's whole first block of
    # keys, and a blockwise attention kernel would turn that into NaN.
    blocked = F.broadcast_to(
        Tensor(-1e30, dtype=DType.float32, device=device), [q_len, kv_len]
    )
    return F.band_part(
        blocked, num_lower=window - 1, num_upper=kv_len - q_len, exclude=True
    )


class Gemma3Attention(Attention):
    def __init__(self, config: Gemma3Config, layer: int) -> None:
        super().__init__(config)
        self.sliding = config.layer_types[layer] == "sliding_attention"
        self.window = config.sliding_window
        self.rope_theta = (
            config.rope_local_base_freq if self.sliding else config.rope_theta
        )
        self.scale = config.query_pre_attn_scalar**-0.5
        self.q_norm = GemmaRMSNorm(self.head_dim, eps=config.rms_norm_eps)
        self.k_norm = GemmaRMSNorm(self.head_dim, eps=config.rms_norm_eps)

    def forward(self, x: Tensor) -> Tensor:
        batch, seq, _ = x.shape
        q = self.q_norm(
            self.q_proj(x).reshape([batch, seq, self.num_heads, self.head_dim])
        )
        k = self.k_norm(
            self.k_proj(x).reshape(
                [batch, seq, self.num_kv_heads, self.head_dim]
            )
        )
        v = self.v_proj(x).reshape(
            [batch, seq, self.num_kv_heads, self.head_dim]
        )
        cos, sin = rope_cos_sin(self.head_dim, self.rope_theta, 0, seq, x.mesh)
        q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
        if self.sliding:
            mask = sliding_window_mask(seq, seq, self.window, x.mesh)
        else:
            mask = causal_mask(seq, seq, x.mesh)
        mask = F.broadcast_to(mask, [batch, seq, seq])
        out = flash_attention(q, k, v, self.scale, mask).rebind_mapping(
            q.mapping
        )
        return self.o_proj(
            out.reshape([batch, seq, self.num_heads * self.head_dim])
        )

Saved to minillm/gemma3.py.

Compare the two masks on 16 tokens, with a window small enough to see. The slider changes the window; Gemma 3’s is 512. At a window of 16 the two panels match: once the window covers the whole sequence, a sliding layer is a global one.

seq = 16
full = (causal_mask(seq, seq, DEVICE).to_numpy() == 0).astype(int)
windowed_masks = {
    window: (
        sliding_window_mask(seq, seq, window, DEVICE).to_numpy() == 0
    ).astype(int)
    for window in [2, 4, 8, 16]
}
print("visible keys per query, window 4:", windowed_masks[4].sum(axis=1))
visible keys per query, window 4: [1 2 3 4 4 4 4 4 4 4 4 4 4 4 4 4]
Show the code 48 lines
steps = []
for window, windowed in windowed_masks.items():
    steps.append(
        [
            go.Heatmap(
                z=m,
                zmin=0,
                zmax=1,
                showscale=False,
                xgap=2,
                ygap=2,
                xaxis=f"x{a}",
                yaxis=f"y{a}",
                colorscale=[
                    [0, viz.GRID],
                    [0.5, viz.GRID],
                    [0.5, viz.BLUE],
                    [1, viz.BLUE],
                ],
                hovertemplate="query %{y}, key %{x}: "
                + "%{z}"
                + "<extra>"
                + title
                + "</extra>",
            )
            for m, a, title in [
                (full, "", "global layer"),
                (windowed, "2", f"window {window}"),
            ]
        ]
    )
fig = make_subplots(
    rows=1,
    cols=2,
    horizontal_spacing=0.08,
    subplot_titles=[
        "global layer: causal",
        "sliding layer: causal and windowed",
    ],
)
fig.update_xaxes(title_text="key position", showgrid=False)
fig.update_yaxes(
    title_text="query position", showgrid=False, autorange="reversed"
)
fig.update_layout(title="Which keys each query sees (1 = visible)", height=440)
viz.show(
    viz.with_slider(fig, steps, ["2", "4", "8", "16"], "window ", active=1)
)

Once the sequence is longer than the window, a sliding layer’s cost per query stops growing, and so does the KV cache it needs: only the last window keys. That saving needs an attention kernel that skips fully masked blocks and a cache that evicts keys older than the window. The model in this section has neither: its dense mask still visits every key, and it keeps no cache, so it computes the same function without the savings. In a serving engine with both, five sliding layers for every global one mean most of Gemma 3’s KV cache stops growing at 512 tokens.

The rest of the model

GeGLU is the SwiGLU MLP with a different activation. The decoder layer has the four norms, and the model scales its embedding:

Show GeGLU, Gemma3DecoderLayer, Gemma3Model, Gemma3ForCausalLM, ... 64 lines
class GeGLU(MLP):
    def forward(self, x: Tensor) -> Tensor:
        return self.down_proj(
            F.gelu(self.gate_proj(x), approximate="tanh") * self.up_proj(x)
        )


class Gemma3DecoderLayer(Module):
    def __init__(self, config: Gemma3Config, layer: int) -> None:
        eps = config.rms_norm_eps
        self.input_layernorm = GemmaRMSNorm(config.hidden_size, eps=eps)
        self.self_attn = Gemma3Attention(config, layer)
        self.post_attention_layernorm = GemmaRMSNorm(
            config.hidden_size, eps=eps
        )
        self.pre_feedforward_layernorm = GemmaRMSNorm(
            config.hidden_size, eps=eps
        )
        self.mlp = GeGLU(config)
        self.post_feedforward_layernorm = GemmaRMSNorm(
            config.hidden_size, eps=eps
        )

    def forward(self, x: Tensor) -> Tensor:
        x = x + self.post_attention_layernorm(
            self.self_attn(self.input_layernorm(x))
        )
        return x + self.post_feedforward_layernorm(
            self.mlp(self.pre_feedforward_layernorm(x))
        )


class Gemma3Model(Module):
    def __init__(self, config: Gemma3Config) -> None:
        self.embed_scale = math.sqrt(config.hidden_size)
        self.embed_tokens = Embedding(config.vocab_size, dim=config.hidden_size)
        self.layers = ModuleList(
            [
                Gemma3DecoderLayer(config, i)
                for i in range(config.num_hidden_layers)
            ]
        )
        self.norm = GemmaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)

    def forward(self, input_ids: Tensor) -> Tensor:
        x = self.embed_tokens(input_ids) * self.embed_scale
        for layer in self.layers:
            x = layer(x)
        return self.norm(x)


class Gemma3ForCausalLM(Module):
    def __init__(self, config: Gemma3Config) -> None:
        self.model = Gemma3Model(config)
        self.lm_head = Linear(config.hidden_size, config.vocab_size, bias=False)

    def forward(self, input_ids: Tensor) -> Tensor:
        return self.lm_head(self.model(input_ids))


def without_sliding_window(config: Gemma3Config) -> Gemma3Config:
    """Returns ``config`` with windows too wide to hide any token; each layer
    keeps its own RoPE base."""
    return replace(config, sliding_window=config.max_position_embeddings)

Saved to minillm/gemma3.py.

weights = load_checkpoint(GEMMA)
model = load_model(config, weights, DEVICE, Gemma3ForCausalLM)
compiled = model.compile(
    TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)

Verify it

A short prompt never reaches the window, so verify on two prompts: a short one, and one of about 700 tokens, where windowed layers must ignore the oldest tokens.

Show softmax, verify 38 lines
tokenizer = AutoTokenizer(GEMMA)
reference = AutoModelForCausalLM.from_pretrained(
    GEMMA, torch_dtype=torch.float32, attn_implementation="eager"
).eval()
prompts = {
    "short": "The capital of France is",
    "long (700 tokens)": "The quick brown fox jumps over the lazy dog. " * 70
    + "The capital of France is",
}


def softmax(z: np.ndarray) -> np.ndarray:
    e = np.exp(z.astype(np.float64) - z.max(-1, keepdims=True))
    return e / e.sum(-1, keepdims=True)


def verify(run) -> dict[str, tuple[np.ndarray, float]]:  # noqa: ANN001
    # Prints and returns each prompt's KL divergence per position and its
    # top-1 agreement.
    results = {}
    for name, text in prompts.items():
        ids = tokenizer([text])["input_ids"][0]
        ours = run(Tensor(np.array([ids]), device=DEVICE)).to_numpy()[0]
        with torch.no_grad():
            theirs = reference(torch.tensor([ids])).logits[0].numpy()
        r = compare_outputs(ours, theirs)
        p, q = softmax(theirs), softmax(ours)
        kl = (p * (np.log(p + 1e-30) - np.log(q + 1e-30))).sum(-1)
        top1 = (ours.argmax(-1) == theirs.argmax(-1)).mean()
        print(
            f"{name:18} {len(ids):4} tokens  max err {r.max_abs_err:.1e}  KL {kl.mean():.1e}  top-1 {top1:.0%}"
        )
        results[name] = (kl, top1)
    return results


matched = verify(compiled)
assert all(kl.mean() < 1e-6 and top1 >= 0.99 for kl, top1 in matched.values())
short                 6 tokens  max err 1.6e-04  KL 4.8e-11  top-1 100%
long (700 tokens)   706 tokens  max err 4.1e-04  KL 6.5e-12  top-1 100%

Both match. To see that the window does real work, build the same model with windows too wide to hide anything, keep each layer’s RoPE base, and verify again:

no_window = load_model(
    without_sliding_window(config), weights, DEVICE, Gemma3ForCausalLM
)
unwindowed = verify(
    no_window.compile(
        TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
    )
)
long = "long (700 tokens)"
assert unwindowed["short"][0].mean() < 1e-6  # no query is far enough back
assert unwindowed[long][0].mean() > 100 * matched[long][0].mean()
short                 6 tokens  max err 1.6e-04  KL 4.8e-11  top-1 100%
long (700 tokens)   706 tokens  max err 4.9e+01  KL 2.8e-01  top-1 93%
Show the code 30 lines
fig = go.Figure(
    [
        go.Scatter(
            x=np.arange(len(results[long][0])),
            y=np.maximum(results[long][0], 1e-16),
            name=name,
            mode="lines",
            line_color=color,
            hovertemplate="position %{x}: KL %{y:.1e}<extra>"
            + name
            + "</extra>",
        )
        for name, results, color in [
            ("with the window", matched, viz.BLUE),
            ("without the window", unwindowed, viz.ORANGE),
        ]
    ]
)
fig.add_vline(
    x=config.sliding_window,
    line=dict(color=viz.INK_SECONDARY, dash="dot", width=1),
    annotation_text=f"position {config.sliding_window}",
)
fig.update_layout(
    title="KL divergence from the reference at each position of the long prompt",
    height=420,
    xaxis_title="position",
    yaxis=dict(title="KL (nats, log scale)", type="log", dtick=2),
)
viz.show(fig)

On the short prompt the two models are identical, because no query is more than 512 tokens from any key. On the long prompt the model without the window diverges from the reference, and the figure shows where: the two curves agree up to position 511 and split at 512, the first position with a key outside its window. The windowed layers do ignore the oldest tokens, and the bring-up must get the window right to match.

With a kernel and cache that take advantage of the window, its attention cost and KV cache stop growing at 512 tokens. With 15 of Gemma 3’s 18 layers windowed, a 32,000-token context needs about a fifth of the cache that a fully global model would need. The handbook’s GPU memory calculation page shows how cache size limits context and batch size.

Exercises

  1. Compute the KV cache size of this model for 8K and 32K tokens of context, with and without sliding windows.
  2. Plot attention weights, as in Section 1.5, for a windowed and a global layer on the long prompt. What does the window look like?
  3. Check the embedding scale: remove embed_scale and verify again. Which metric shows it?

Next

Section 6.3 builds a mixture-of-experts layer and spreads its experts across devices.

6.3 Mixture of experts

Open in Colab Open in Kaggle View on GitHub

The largest open models are mixtures of experts (MoE). An MoE layer has many MLPs, the experts, instead of one, and a small router sends each token to only a few of them. The model holds the parameters of all its experts, while each token pays for only a fraction of them. This section builds an MoE layer, checks it against Hugging Face’s Qwen3-MoE implementation, and then spreads the experts across devices.

No real MoE checkpoint fits on a laptop, so this section uses Qwen3-MoE’s architecture at a small size with random weights, a common way to test large-model code. As in Section 4.1, the test compares one layer with a reference implementation.

Show the setup 32 lines
import json
import logging
import warnings
from pathlib import Path

import numpy as np
import plotly.graph_objects as go
import torch
from huggingface_hub import hf_hub_download
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.tensor import Tensor, default_device, default_dtype
from max.tensor.sharding import (
    DeviceMapping,
    DeviceMesh,
    Replicated,
    Sharded,
    auto_reshard,
)
from plotly.subplots import make_subplots
from transformers import AutoConfig
from transformers.models.qwen3_moe.modeling_qwen3_moe import (
    Qwen3MoeSparseMoeBlock,
)

from minillm import viz
from minillm.device import pick_device

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU

How big, and how sparse

Qwen3-30B-A3B is a typical MoE model. Its name gives its total parameters, 30 billion, and its active ones, about 3 billion:

real = json.loads(
    Path(hf_hub_download("Qwen/Qwen3-30B-A3B", "config.json")).read_text()
)
E, K, H, I = (
    real["num_experts"],
    real["num_experts_per_tok"],
    real["hidden_size"],
    real["moe_intermediate_size"],
)
per_expert = 3 * H * I
print(f"{E} experts per layer, {K} chosen per token")
print(
    f"expert weights per layer: {E * per_expert / 1e6:.0f}M, used per token: {K * per_expert / 1e6:.0f}M"
)
128 experts per layer, 8 chosen per token
expert weights per layer: 604M, used per token: 38M

Every token uses 8 of the 128 experts, so it touches only a sixteenth of each layer’s expert weights. Adding experts adds parameters without adding work per token. The cost moves to memory, since every expert must be loaded somewhere, which is why MoE models are usually served across several GPUs. The handbook’s parallelism strategies page covers expert parallelism, which the end of this section implements.

The layer

An MoE layer replaces the MLP. For each token:

  1. The router, a Linear from the hidden size to the number of experts, scores every expert. Softmax turns the scores into probabilities.
  2. The K most probable experts are chosen, and their probabilities are renormalized to sum to 1. Qwen3-MoE’s config asks for this with norm_topk_prob: true; a model that sets it to false weights the experts by their raw probabilities, so reusing MoE for one means dropping that step.
  3. Each chosen expert, a SwiGLU MLP like the one in Section 1.4, processes the token, and the outputs are summed, weighted by those probabilities.

Mixture-of-experts layer: a token's hidden state goes to a router, which scores all 128 experts; the 8 highest-scoring experts run on the token, and their outputs are combined, weighted by the renormalized router probabilities.Mixture-of-experts layer: a token's hidden state goes to a router, which scores all 128 experts; the 8 highest-scoring experts run on the token, and their outputs are combined, weighted by the renormalized router probabilities.

In symbols, with router weights \(W_r\), the set \(\mathcal{T}\) of the \(K\) most probable experts, and expert \(e\)’s MLP \(E_e\):

\[p = \mathrm{softmax}(W_r x), \qquad y = \sum_{e \in \mathcal{T}} \frac{p_e}{\sum_{j \in \mathcal{T}} p_j} \, E_e(x)\]

The experts’ weights are stored stacked, with the expert as the first axis: one tensor for the fused gate and up projections, and one for the down projection. That is the layout of Hugging Face’s Qwen3-MoE module; the checkpoint files store each expert’s projections separately. This version computes every expert on every token and multiplies the unselected ones by zero. That wastes work, but it is simple and exact, and the last part of this section shows that it is easy to distribute.

Show Experts, MoE 66 lines
from __future__ import annotations

from max import functional as F
from max.dtype import DType
from max.nn import Linear, Module
from max.tensor import Tensor


class Experts(Module):
    """The stacked weights of every expert: gate and up projections fused."""

    def __init__(
        self, num_experts: int, hidden: int, intermediate: int
    ) -> None:
        self.gate_up_proj = Tensor.zeros(
            [num_experts, 2 * intermediate, hidden]
        )
        self.down_proj = Tensor.zeros([num_experts, hidden, intermediate])


class MoE(Module):
    """A mixture-of-experts layer with top-k routing, as in Qwen3-MoE."""

    def __init__(
        self, num_experts: int, top_k: int, hidden: int, intermediate: int
    ) -> None:
        self.num_experts, self.top_k, self.intermediate = (
            num_experts,
            top_k,
            intermediate,
        )
        self.gate = Linear(hidden, num_experts, bias=False)  # the router
        self.experts = Experts(num_experts, hidden, intermediate)

    def routing(self, x: Tensor) -> Tensor:
        """Returns each token's weight for each expert, ``[tokens, experts]``,
        zero for experts the token wasn't routed to."""
        tokens = x.shape[0]
        weights, chosen = F.top_k(F.softmax(self.gate(x)), self.top_k)
        weights = weights / weights.sum(axis=-1)
        ids = Tensor.arange(self.num_experts, dtype=DType.int64, device=x.mesh)
        one_hot = (
            chosen.reshape([tokens, self.top_k, 1])
            == ids.reshape([1, 1, self.num_experts])
        ).cast(x.dtype)
        return (
            (one_hot * weights.reshape([tokens, self.top_k, 1]))
            .sum(axis=1)
            .reshape([tokens, self.num_experts])
        )

    def forward(self, x: Tensor) -> Tensor:
        """``x`` is ``[tokens, hidden]``."""
        tokens, hidden = x.shape
        gate_up = x.reshape(
            [1, tokens, hidden]
        ) @ self.experts.gate_up_proj.transpose(1, 2)
        gate, up = (
            gate_up[:, :, : self.intermediate],
            gate_up[:, :, self.intermediate :],
        )
        outputs = (F.silu(gate) * up) @ self.experts.down_proj.transpose(
            1, 2
        )  # [E, T, H]
        weights = self.routing(x).T.reshape([self.num_experts, tokens, 1])
        return (weights * outputs).sum(axis=0).reshape([tokens, hidden])

Saved to minillm/moe.py.

Check it against Qwen3-MoE

Build Hugging Face’s Qwen3-MoE block at a small size, 8 experts with 2 per token, fill it with random weights, and load the same weights into yours. The parameter names match, so load_state_dict takes the PyTorch state directly:

torch.manual_seed(0)
small = AutoConfig.from_pretrained("Qwen/Qwen3-30B-A3B")
(
    small.hidden_size,
    small.moe_intermediate_size,
    small.num_experts,
    small.num_experts_per_tok,
) = 64, 32, 8, 2
reference = Qwen3MoeSparseMoeBlock(small).float().eval()
for p in reference.parameters():
    torch.nn.init.normal_(p, std=0.2)
state = {name: p.detach().numpy() for name, p in reference.state_dict().items()}
print({name: w.shape for name, w in state.items()})

with default_device(DEVICE), default_dtype(DType.float32):
    moe = MoE(num_experts=8, top_k=2, hidden=64, intermediate=32)
moe.load_state_dict(state)

x = np.random.default_rng(0).standard_normal((10, 64)).astype(np.float32)
with torch.no_grad():
    expected = reference(torch.from_numpy(x)[None])[0].numpy()
ours = moe(Tensor(x, device=DEVICE)).to_numpy()
print(f"max difference {np.abs(ours - expected).max():.1e}")
assert np.abs(ours - expected).max() < 1e-4
{'experts.gate_up_proj': (8, 64, 64), 'experts.down_proj': (8, 64, 32), 'gate.weight': (8, 64)}
max difference 3.8e-06

Where the tokens go

The routing matrix shows which experts each token uses. Count the tokens per expert for a larger batch, then make the router favor expert 0 by scaling its row of router weights, and count again:

batch = Tensor(
    np.random.default_rng(1).standard_normal((512, 64)).astype(np.float32),
    device=DEVICE,
)
weights_np = moe.routing(batch).to_numpy()
load = (weights_np > 0).sum(axis=0)
print("tokens per expert:", load, " ideal:", 512 * 2 // 8)
assert load.sum() == 512 * 2


def load_with_bias(factor: float) -> np.ndarray:
    router = state["gate.weight"].copy()
    router[0] *= factor
    with default_device(DEVICE), default_dtype(DType.float32):
        biased = MoE(num_experts=8, top_k=2, hidden=64, intermediate=32)
    biased.load_state_dict({**state, "gate.weight": router})
    return (biased.routing(batch).to_numpy() > 0).sum(axis=0)


loads = {factor: load_with_bias(factor) for factor in [1, 2, 5]}
for factor, counts in loads.items():
    print(f"expert 0's row x{factor}: tokens per expert {counts}")
assert (loads[1] == load).all()
tokens per expert: [122 116 157 121 117 134 122 135]  ideal: 128
expert 0's row x1: tokens per expert [122 116 157 121 117 134 122 135]
expert 0's row x2: tokens per expert [178 109 144 116 108 130 113 126]
expert 0's row x5: tokens per expert [227 104 136 109 103 121 104 120]
Show the code 60 lines
fig = make_subplots(
    rows=1,
    cols=2,
    column_widths=[0.6, 0.4],
    horizontal_spacing=0.2,
    subplot_titles=[
        "Routing weights of the first 32 tokens",
        "Tokens per expert, all 512",
    ],
)
fig.add_trace(
    go.Heatmap(
        z=np.where(weights_np[:32] > 0, weights_np[:32], np.nan),
        colorscale=viz.SEQUENTIAL,
        zmin=0,
        zmax=1,
        xgap=2,
        ygap=2,
        colorbar=dict(title="weight", thickness=12, x=0.5),
        hovertemplate="token %{y} -> expert %{x}: weight %{z:.2f}<extra></extra>",
    ),
    row=1,
    col=1,
)
steps = [
    [
        go.Bar(
            x=np.arange(len(counts)),
            y=counts,
            marker_color=[viz.ORANGE] + [viz.BLUE] * (len(counts) - 1),
            showlegend=False,
            xaxis="x2",
            yaxis="y2",
            hovertemplate="expert %{x}: %{y} tokens<extra></extra>",
        )
    ]
    for counts in loads.values()
]
fig.update_xaxes(title_text="expert", dtick=1, showgrid=False)
fig.update_yaxes(
    title_text="token", autorange="reversed", showgrid=False, row=1, col=1
)
fig.update_yaxes(
    title_text="tokens routed",
    range=[0, 1.1 * max(c.max() for c in loads.values())],
    row=1,
    col=2,
)
fig.update_layout(title="Top-2 routing over 8 experts", height=520)
labels = [f"x{factor}" for factor in loads]
viz.with_slider(fig, steps, labels, "expert 0's router row ")
fig.add_hline(
    y=512 * 2 / len(load),
    line=dict(color=viz.INK_SECONDARY, dash="dash"),
    row=1,
    col=2,
    annotation_text="even split",
    annotation_position="top right",
)
viz.show(fig)

Every token went to exactly two experts. Move the slider to scale expert 0’s row of router weights: its scores grow, so more tokens pick it, and its bar rises to nearly twice the even split at x5 while the others shrink. How evenly a batch spreads across experts matters in practice: an expert that gets far more tokens than the others becomes the slowest part of the layer. MoE models are usually trained with a load-balancing term in the loss that pushes the router to spread tokens evenly, and serving systems monitor the load on each expert.

This dense version computes all 8 experts for every token, 4 times the work that the 2 chosen experts need. Production kernels instead group the tokens by expert and run one matrix multiplication per expert over only its tokens (a grouped GEMM), which removes the waste:

per_expert_small = 3 * 64 * 32  # gate, up, and down projections
dense = 2 * 512 * 8 * per_expert_small
sparse = 2 * 512 * 2 * per_expert_small
print(
    f"dense: {dense / 1e6:.1f} MFLOP, routed: {sparse / 1e6:.1f} MFLOP, {dense / sparse:.0f}x less"
)
dense: 50.3 MFLOP, routed: 12.6 MFLOP, 4x less

Expert parallelism

The experts are independent, so they can live on different devices. With 8 experts on 4 devices, each device holds 2. Every device receives every token, runs its own experts, and weights their outputs; the final sum over experts spans devices, which takes one all-reduce. On the simulated mesh from Section 2.1, the only change is to shard the stacked expert weights along the expert axis:

mesh = DeviceMesh((CPU(),) * 4, (4,), ("ep",))
with default_device(mesh), default_dtype(DType.float32):
    ep_moe = MoE(num_experts=8, top_k=2, hidden=64, intermediate=32)
by_expert = DeviceMapping(mesh, (Sharded(0),))
ep_moe.experts.gate_up_proj = ep_moe.experts.gate_up_proj.to(by_expert)
ep_moe.experts.down_proj = ep_moe.experts.down_proj.to(by_expert)
ep_moe.load_state_dict(state)
print(ep_moe.experts.down_proj.mapping)

with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    with auto_reshard(mode="warn"):
        ep_out = ep_moe(Tensor(x, device=DeviceMapping(mesh, (Replicated(),))))
for w in caught:
    if "reshards" in str(w.message):
        print(w.message)
print(
    f"max difference from the reference: {np.abs(ep_out.to_numpy() - expected).max():.1e}"
)
assert np.abs(ep_out.to_numpy() - expected).max() < 1e-4
DeviceMapping(DeviceMesh(ep=4), [Sharded(axis=0)])

mul(lhs, rhs) reshards lhs
  lhs    : float32[8, 10, 1]@mesh(ep=4){"ep":R}
  rhs    : float32[8, 10, 64]@mesh(ep=4){"ep":S0}
  region : auto_reshard(DEFAULT_TRANSITIONS, mode="warn")

  decided per mesh axis
  axis    lhs           rhs           -> out        collective                          bytes    status
  "ep"    lhs{R->S0}    rhs{S0}       -> S0         local_slice(lhs)                      1.25   chosen, reshards lhs
          lhs{R}        rhs{S0->R}    -> R          allgather(rhs)                       15360
          lhs{R}        rhs{S0->S2}   -> S2         all_to_all(rhs)                      15360
          lhs{R->S1}    rhs{S0->S1}   -> S1         local_slice(lhs), all_to_all(rhs)   15361.2
          lhs{R->P}     rhs{S0->R}    -> P          keep_one_copy(lhs), allgather(rhs)  15361.2  not allowed
  result  lhs{"ep":S0}  rhs{"ep":S0}  -> {"ep":S0}

  lhs = transfer_to(lhs, DeviceMapping(mesh, (Sharded(0),)))
  with auto_reshard(mode="silent"):
  at <cell>:67 in forward()
max difference from the reference: 2.9e-06

The output matches. MAX resharded the routing weights to match the experts’ layout, so each device multiplies only its own experts’ slice, and the weighted outputs come out as partial sums that to_numpy() all-reduces.

This dense form sends every token to every device. Expert-parallel serving sends each token only to the devices that hold its chosen experts, an all-to-all exchange, and returns the results with a second one. With many experts this moves far less data, but the communication is uneven and depends on the routing. MAX serves MoE models such as GPT-OSS and DeepSeek, and supports expert parallelism across GPUs with all-to-all dispatch and combine steps.

Exercises

  1. Increase top_k to 8 (every expert). Does the output equal a weighted sum of 8 dense MLPs? What would the model lose?
  2. Implement the routed version for one device: for each expert, gather its tokens, run the MLP on them only, and scatter the results back. Check it against MoE.
  3. With the experts spread over 4 devices as above, 2 per device, how many token copies does each device process at each slider setting of the routing figure? What does the busiest device do to the time of an expert-parallel step?

What you built

Chapter 6 extended your model with three features of current open models: normalized queries and keys, sliding-window attention, and mixtures of experts. Chapter 7 serves your own architecture with max serve, benchmarks it, and constrains its output.

7. Production serving

The last chapter packages your own architecture the way MAX packages its own, serves and measures it, and ends on Modular’s cloud.

SectionYou learn
7.1 Serve your architecturePackaging your model as a MAX architecture and serving it with max serve --custom-architectures.
7.2 Benchmark a serverTTFT, TPOT, and throughput with max benchmark, for MAX’s SmolLM2 and yours, and a concurrency sweep against a latency target.
7.3 Structured output and toolsJSON that follows a schema, and function calling built on top of it.
7.4 Modular’s cloudPointing the same client at Modular’s cloud, an inference service that runs state-of-the-art open models on MAX.

Related handbook pages: OpenAI-compatible API, structured outputs, function calling, and performance benchmarks.

7.1 Serve your architecture

Open in Colab Open in Kaggle View on GitHub

In Section 1.1 you served SmolLM2 with MAX’s own implementation. This section serves yours: the LlamaForCausalLM you wrote in Section 1.6, behind the same AutoPipeline API and the same OpenAI-compatible server. MAX loads model code from outside its own package through custom architectures: a Python package that describes the model, which max serve imports with --custom-architectures.

You register the model twice. The first registration runs your model unchanged and recomputes the whole sequence at every step. The second wraps your layers in a decoder module that reads keys and values from MAX’s paged KV cache, which is how MAX’s own architectures serve.

Show the setup 13 lines
import logging
import time
import warnings

from max.pipelines import AutoPipeline
from openai import OpenAI

from minillm.config import MODEL_ID
from minillm.server import start_max_serve

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
PROMPT = ["The capital of France is"]

First, the reference: SmolLM2 as MAX implements it, loaded with AutoPipeline.

reference = AutoPipeline(MODEL_ID, quantization_encoding="float32")
expected = reference(PROMPT, max_new_tokens=16, top_k=1)
reference.close()
expected
['The capital of France is Paris. Paris is the largest city in France and the capital of the French department']

A custom architecture package

A MAX architecture has three parts:

  • A config class tells MAX the sizes it needs to plan memory before it loads anything: the number of layers, key/value heads, and head size. It also declares the supported encodings; this model computes in float32.
  • A pipeline model loads the weights and runs the model. MAX gives it the checkpoint’s weights and, at each step, the request’s tokens, and it returns the next-token logits.
  • A SupportedArchitecture ties them together under the name in the checkpoint’s config.json, here LlamaForCausalLM, along with the tokenizer and the weight format. ARCHITECTURES lists it for --custom-architectures to find.

To keep this first version short, the pipeline model reruns the whole sequence at each step and serves one request at a time (max_batch_size is 1), like the generation loop in Section 1.7. A decoder on the paged KV cache below removes both limits.

The package is one module, minillm/serve/__init__.py, which the cells below write part by part. The first imports what the package uses from MAX:

Show the imports 43 lines
from __future__ import annotations

from collections.abc import Callable, Sequence
from dataclasses import dataclass
from typing import ClassVar

import numpy as np
from max import functional as F
from max.driver import CPU, Buffer, Device
from max.dtype import DType
from max.graph import TensorType
from max.graph.weights import Weights, WeightsAdapter, WeightsFormat
from max.nn import Embedding, Linear, Module, ModuleList, RMSNorm
from max.nn.attention_mask import MHAMaskVariant
from max.nn.common_layers.kv_cache import per_device_caches
from max.nn.kernels import flash_attention_ragged, rope_split_store_ragged
from max.nn.kv_cache import KVCacheInputs, KVCacheParams, PagedCacheValues
from max.pipelines import upper_bounded_default
from max.pipelines.context import TextContext
from max.pipelines.lib import (
    GraphRole,
    MAXModelConfig,
    ModelInputs,
    ModelOutputs,
    PipelineConfig,
    PipelineModelWithKVCache,
    SupportedArchitecture,
    SupportedEncoding,
    TextTokenizer,
)
from max.pipelines.lib.interfaces import (
    ArchConfigWithAttentionKVCache,
    ArchConfigWithStoredKVParams,
)
from max.pipelines.lib.role_contracts import DecoderInputs, DecoderOutputs
from max.pipelines.lib.utils import parse_state_dict_from_weights
from max.pipelines.modeling.types import PipelineTask
from max.tensor import Tensor
from transformers import AutoConfig

from minillm.config import LlamaConfig
from minillm.layers import MLP, rope_cos_sin
from minillm.model import load_model

Saved to minillm/serve/__init__.py.

The config class reads the sizes from the checkpoint’s Hugging Face config. ArchConfigWithStoredKVParams adds construct_kv_params, which MAX calls to size the KV cache from those sizes. llama_config converts the same config to the LlamaConfig from Section 1.3:

Show MinillmArchConfig, llama_config 38 lines
@dataclass
class MinillmArchConfig(
    ArchConfigWithAttentionKVCache, ArchConfigWithStoredKVParams
):
    DEFAULT_ENCODING: ClassVar[SupportedEncoding] = "float32"
    SUPPORTED_ENCODINGS: ClassVar[set[SupportedEncoding]] = {"float32"}

    @property
    def num_key_value_heads(self) -> int:
        return self.huggingface_config.num_key_value_heads

    @property
    def head_dim(self) -> int:
        return self.get_head_dim(self.huggingface_config)

    @property
    def num_layers(self) -> int:
        return self.huggingface_config.num_hidden_layers

    @classmethod
    def calculate_max_seq_len(
        cls, huggingface_config: AutoConfig, model_config: MAXModelConfig
    ) -> int:
        return upper_bounded_default(
            upper_bound=huggingface_config.max_position_embeddings,
            default=model_config.max_length,
        )


def llama_config(hf: AutoConfig) -> LlamaConfig:
    """Returns the checkpoint's sizes as a ``LlamaConfig``."""
    return LlamaConfig(
        **{
            k: getattr(hf, k)
            for k in LlamaConfig.__dataclass_fields__
            if hasattr(hf, k)
        }
    )

Saved to minillm/serve/__init__.py.

The pipeline model’s main job is to build your model. build_model casts the checkpoint’s bfloat16 weights to float32, builds your LlamaForCausalLM with load_model from Section 1.6, and compiles it for any sequence length:

def build_model(
    config: LlamaConfig,
    pipeline_config: PipelineConfig,
    weights: Weights,
    adapter: WeightsAdapter | None,
    device: Device,
) -> Callable[[Tensor], Tensor]:
    """Builds and compiles minillm's ``LlamaForCausalLM`` from a checkpoint."""
    state = parse_state_dict_from_weights(pipeline_config, weights, adapter)
    state = {name: w.astype(DType.float32) for name, w in state.items()}
    model = load_model(config, state, device)
    return model.compile(
        TensorType(DType.int64, ("batch", "seq_len"), device=device)
    )

Saved to minillm/serve/__init__.py.

MAX constructs the pipeline model with a long list of arguments, which the constructor passes through before it calls build_model. model_config_cls names the config class, which MAX asks to size the KV cache. prepare_initial_token_inputs turns the request’s context into model inputs: here all of its tokens, since the model keeps no cache. execute runs the compiled model and returns the last position’s logits:

Show TokenInputs, MinillmPipelineModel 41 lines
@dataclass
class TokenInputs(ModelInputs):
    tokens: Buffer

    @property
    def buffers(self) -> tuple[Buffer, ...]:
        return (self.tokens,)


class MinillmPipelineModel(PipelineModelWithKVCache[TextContext]):
    model_config_cls: ClassVar[type[MinillmArchConfig]] = MinillmArchConfig

    def __init__(self, *args: object, **kwargs: object) -> None:
        super().__init__(*args, **kwargs)
        self.model = build_model(
            llama_config(self.huggingface_config),
            self.pipeline_config,
            self.weights,
            self.adapter,
            self.devices[0],
        )

    def prepare_initial_token_inputs(
        self,
        replica_batches: Sequence[Sequence[TextContext]],
        kv_cache_inputs: KVCacheInputs[Buffer, Buffer] | None = None,
        return_n_logits: int = 1,
    ) -> TokenInputs:
        ids = replica_batches[0][0].tokens.all
        tokens = np.array([ids], dtype=np.int64)
        inputs = TokenInputs(Buffer.from_numpy(tokens).to(self.devices[0]))
        inputs.kv_cache_inputs = kv_cache_inputs
        return inputs

    def execute(self, model_inputs: ModelInputs) -> ModelOutputs:
        assert isinstance(model_inputs, TokenInputs)
        logits = self.model(Tensor.from_dlpack(model_inputs.tokens))
        last = Buffer.from_numpy(
            np.ascontiguousarray(logits.to_numpy()[0, -1:, :])
        ).to(self.devices[0])
        return ModelOutputs(logits=last, next_token_logits=last)

Saved to minillm/serve/__init__.py.

Finally, SupportedArchitecture registers the parts under the checkpoint’s architecture name. required_arguments makes MAX serve one request at a time and turns off prefix caching, which this model can’t use:

def tie_embeddings(state_dict: dict[str, Weights], **_: object) -> dict:
    """Loads the checkpoint as is, with the LM head sharing the embedding."""
    weights = {name: w.data() for name, w in state_dict.items()}
    weights.setdefault("lm_head.weight", weights["model.embed_tokens.weight"])
    return weights


minillm_arch = SupportedArchitecture(
    name="LlamaForCausalLM",
    task=PipelineTask.TEXT_GENERATION,
    example_repo_ids=["HuggingFaceTB/SmolLM2-135M-Instruct"],
    default_weights_format=WeightsFormat.safetensors,
    default_encoding=MinillmArchConfig.DEFAULT_ENCODING,
    supported_encodings=MinillmArchConfig.SUPPORTED_ENCODINGS,
    pipeline_model=MinillmPipelineModel,
    tokenizer=TextTokenizer,
    context_type=TextContext,
    multi_gpu_supported=False,
    weight_adapters={WeightsFormat.safetensors: tie_embeddings},
    config=MinillmArchConfig,
    required_arguments={"enable_prefix_caching": False, "max_batch_size": 1},
    supports_overlap_scheduler=False,
    supports_device_graph_capture=False,
)

Saved to minillm/serve/__init__.py.

Use it in Python

custom_architectures imports the package and registers its architectures, as --custom-architectures does for the CLI. A registered architecture with the same name as one of MAX’s replaces it in this process. MAX registers module-based implementations under the name with a _ModuleV3 suffix and prefers them, so prefer_module_v3=False selects the LlamaForCausalLM registered above:

ours = AutoPipeline(
    MODEL_ID,
    quantization_encoding="float32",
    custom_architectures=["minillm.serve"],
    prefer_module_v3=False,
)
result = ours(PROMPT, max_new_tokens=16, top_k=1)
ours.close()
print(result)
print("same as MAX's implementation:", result == expected)
assert result == expected
['The capital of France is Paris. Paris is the largest city in France and the capital of the French department']
same as MAX's implementation: True

The pipeline model is yours, and its greedy output matches MAX’s implementation token for token.

A decoder on the paged KV cache

The pipeline model above recomputes every token at every step and runs one request at a time. MAX’s own architectures don’t write a pipeline model. They register a decoder module for the DECODER role, and MAX builds the pipeline model around it. That pipeline model brings what Chapter 5 built by hand: continuous batching, a paged KV cache, and prefix caching.

At each step, the decoder’s forward receives a DecoderInputs:

  • tokens: the new tokens of every request in the batch, concatenated into one ragged sequence. A request’s prompt arrives whole at its first step, and then one token per step.
  • input_row_offsets: where each request starts in tokens, plus the total length at the end.
  • kv: the paged KV cache. per_device_caches unpacks it into one PagedCacheValues per device.

Your embedding, norms, and MLP work on ragged tokens unchanged, since they treat each token on its own. Only attention needs the cache, and two MAX kernels supply it. rope_split_store_ragged splits the fused Q/K/V projection, rotates the queries and keys, and writes the keys and values into the cache’s pages. flash_attention_ragged then attends each new query to its own request’s cached keys, with a causal mask. The kernels take the RoPE table packed as (cos, sin) pairs, which rope_table builds from your rope_cos_sin from Section 1.4:

Show rope_table, PagedAttention 59 lines
def rope_table(config: LlamaConfig, length: int, device: Device) -> Tensor:
    """Returns the RoPE table as ``[length, head_dim]`` (cos, sin) pairs."""
    cos, sin = rope_cos_sin(
        config.head_dim, config.rope_theta, 0, length, device
    )
    half = config.head_dim // 2
    pairs = F.stack([cos[:, :half], sin[:, :half]], axis=-1)
    return pairs.reshape([length, config.head_dim])


class PagedAttention(Module):
    """Grouped-query attention that reads keys and values from the cache."""

    def __init__(
        self, config: LlamaConfig, kv_params: KVCacheParams, layer_idx: int
    ) -> None:
        q_dim = config.num_attention_heads * config.head_dim
        kv_dim = config.num_key_value_heads * config.head_dim
        self.q_proj = Linear(config.hidden_size, q_dim, bias=False)
        self.k_proj = Linear(config.hidden_size, kv_dim, bias=False)
        self.v_proj = Linear(config.hidden_size, kv_dim, bias=False)
        self.o_proj = Linear(q_dim, config.hidden_size, bias=False)
        self.n_heads = config.num_attention_heads
        self.head_dim = config.head_dim
        self.kv_params = kv_params
        self.layer_idx = layer_idx

    def forward(
        self,
        x: Tensor,
        kv: PagedCacheValues,
        rope: Tensor,
        input_row_offsets: Tensor,
    ) -> Tensor:
        qkv = F.concat(
            [self.q_proj(x), self.k_proj(x), self.v_proj(x)], axis=-1
        )
        # The kernels read the layer index on the host.
        layer = F.constant(self.layer_idx, DType.uint32, device=CPU())
        q = rope_split_store_ragged(
            kv_params=self.kv_params,
            qkv=qkv,
            input_row_offsets=input_row_offsets,
            freqs_cis=rope,
            kv_collection=kv,
            layer_idx=layer,
            n_heads=self.n_heads,
            interleaved=False,
        )
        out = flash_attention_ragged(
            self.kv_params,
            input=q.reshape([-1, self.n_heads, self.head_dim]),
            input_row_offsets=input_row_offsets,
            kv_collection=kv,
            layer_idx=layer,
            mask_variant=MHAMaskVariant.CAUSAL_MASK,
            scale=self.head_dim**-0.5,
        )
        return self.o_proj(out.reshape([-1, self.n_heads * self.head_dim]))

Saved to minillm/serve/__init__.py.

The layer, the model, and the decoder mirror DecoderLayer, LlamaModel, and LlamaForCausalLM from Section 1.6, with the same attribute names, so the checkpoint loads into them unchanged. MAX constructs the decoder as MinillmDecoder(config) from the config class. The decoder returns the logits of each request’s last token, the one its next token follows:

Show PagedDecoderLayer, PagedLlamaModel, MinillmDecoder 55 lines
class PagedDecoderLayer(Module):
    def __init__(
        self, config: LlamaConfig, kv_params: KVCacheParams, layer_idx: int
    ) -> None:
        eps = config.rms_norm_eps
        self.input_layernorm = RMSNorm(config.hidden_size, eps=eps)
        self.self_attn = PagedAttention(config, kv_params, layer_idx)
        self.post_attention_layernorm = RMSNorm(config.hidden_size, eps=eps)
        self.mlp = MLP(config)

    def forward(
        self,
        x: Tensor,
        kv: PagedCacheValues,
        rope: Tensor,
        input_row_offsets: Tensor,
    ) -> Tensor:
        h = self.input_layernorm(x)
        x = x + self.self_attn(h, kv, rope, input_row_offsets)
        return x + self.mlp(self.post_attention_layernorm(x))


class PagedLlamaModel(Module):
    def __init__(self, config: LlamaConfig, kv_params: KVCacheParams) -> None:
        self.embed_tokens = Embedding(config.vocab_size, dim=config.hidden_size)
        self.layers = ModuleList(
            [
                PagedDecoderLayer(config, kv_params, i)
                for i in range(config.num_hidden_layers)
            ]
        )
        self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)


class MinillmDecoder(Module[[DecoderInputs], DecoderOutputs]):
    def __init__(self, arch_config: MinillmArchConfig) -> None:
        self.config = llama_config(arch_config.huggingface_config)
        self.max_seq_len = arch_config.max_seq_len
        self.kv_params = arch_config.get_kv_params()
        self.model = PagedLlamaModel(self.config, self.kv_params)
        self.lm_head = Linear(
            self.config.hidden_size, self.config.vocab_size, bias=False
        )

    def forward(self, inputs: DecoderInputs) -> DecoderOutputs:
        (kv,) = per_device_caches(self.kv_params, inputs.kv)
        offsets = inputs.input_row_offsets
        rope = rope_table(self.config, self.max_seq_len, inputs.tokens.device)
        x = self.model.embed_tokens(inputs.tokens)
        for layer in self.model.layers:
            x = layer(x, kv, rope, offsets)
        last = F.gather(x, offsets[1:] - 1, axis=0)
        return DecoderOutputs(
            next_token_logits=self.lm_head(self.model.norm(last))
        )

Saved to minillm/serve/__init__.py.

The registration names the decoder under modules instead of a pipeline model. Its name, LlamaForCausalLM_ModuleV3, replaces MAX’s own module-based Llama, so MAX selects it by default:

minillm_decoder_arch = SupportedArchitecture(
    name="LlamaForCausalLM_ModuleV3",
    task=PipelineTask.TEXT_GENERATION,
    example_repo_ids=["HuggingFaceTB/SmolLM2-135M-Instruct"],
    default_weights_format=WeightsFormat.safetensors,
    default_encoding=MinillmArchConfig.DEFAULT_ENCODING,
    supported_encodings=MinillmArchConfig.SUPPORTED_ENCODINGS,
    modules={GraphRole.DECODER: MinillmDecoder},
    tokenizer=TextTokenizer,
    context_type=TextContext,
    weight_adapters={WeightsFormat.safetensors: tie_embeddings},
    config=MinillmArchConfig,
    supports_overlap_scheduler=False,
    supports_device_graph_capture=False,
)
ARCHITECTURES = [minillm_arch, minillm_decoder_arch]

Saved to minillm/serve/__init__.py.

This time MAX batches requests. Two prompts run in one batch, and the first still matches MAX’s implementation:

paged = AutoPipeline(
    MODEL_ID,
    quantization_encoding="float32",
    custom_architectures=["minillm.serve"],
    max_batch_size=2,
)
batch = paged(PROMPT + ["Once upon a time"], max_new_tokens=16, top_k=1)
paged.close()
for text in batch:
    print(text)
print("same as MAX's implementation:", batch[0] == expected[0])
assert batch[0] == expected[0]
The capital of France is Paris. Paris is the largest city in France and the capital of the French department
Once upon a time, I was a young man, a young woman, and a young man again
same as MAX's implementation: True

Serve it

The same package serves over HTTP. --custom-architectures minillm.serve imports it, and MAX serves the decoder:

start = time.perf_counter()
server = start_max_serve(
    MODEL_ID,
    "--quantization-encoding",
    "float32",
    "--custom-architectures",
    "minillm.serve",
)
print(f"server ready in {time.perf_counter() - start:.0f} s")

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
completion = client.completions.create(
    model=MODEL_ID, prompt=PROMPT[0], max_tokens=16, temperature=0
)
print(PROMPT[0] + completion.choices[0].text)
assert PROMPT[0] + completion.choices[0].text == expected[0]
server.terminate()
_ = server.wait()
server ready in 20 s
The capital of France is Paris. Paris is the largest city in France and the capital of the French department

Your model now answers an OpenAI client exactly as MAX’s implementation does, on MAX’s paged KV cache and scheduler. The handbook’s OpenAI-compatible API page covers the request format. MAX’s guide to serving custom model architectures covers the rest of a production architecture, such as weight adapters for other checkpoint formats and multi-GPU serving.

Exercises

  1. Register Qwen3ForCausalLM from Section 6.1 as a decoder, and serve Qwen/Qwen3-0.6B through it. rope_split_store_ragged takes q_norm_weight, k_norm_weight, and rms_norm_eps for Qwen3’s per-head norms. Which parts of the package depend on the model class, and which work unchanged?
  2. Serve each registration in turn, --no-prefer-module-v3 selecting the first, and measure both with max benchmark from Section 7.2. How does throughput change as you raise --max-concurrency?
  3. rope_table rebuilds the RoPE table at every step. Build it once in MinillmDecoder.__init__ instead. Why does MAX then report the table as a weight missing from the checkpoint, and how could the weight adapter supply it?

Next

Section 7.2 measures the server under load with max benchmark.

7.2 Benchmark a server

Open in Colab Open in Kaggle View on GitHub

A server that answers one request correctly can still fall over under real traffic. max benchmark measures a running server the way users experience it: it sends many requests, some at the same time, and records how long each one waits for its first token and for each token after that. This section benchmarks MAX’s SmolLM2 and the architecture you served in Section 7.1, and explains the numbers.

Show the setup 15 lines
import json
import subprocess
import tempfile
from collections.abc import Iterator
from contextlib import contextmanager
from pathlib import Path

import plotly.graph_objects as go
from plotly.subplots import make_subplots

from minillm import viz
from minillm.config import MODEL_ID
from minillm.server import start_max_serve

WORKDIR = Path(tempfile.mkdtemp())

What to measure

These numbers describe a serving run. The handbook’s LLM inference metrics page defines them in detail:

  • Time to first token (TTFT): the time from sending a request to receiving its first token. It includes the wait in the queue and the prefill. Users notice it as the delay before the answer starts.
  • Time per output token (TPOT): the average gap between later tokens, one decode step. Users notice it as the reading speed.
  • Requests per second and output tokens per second: the throughput of the whole server, which decides how many users one GPU can serve.

Concurrency, the number of requests in flight at once, connects them. More concurrency lets the server batch more requests per step (Chapter 5), which raises throughput, but each request can then wait longer.

Run a benchmark

From a terminal, a benchmark is one command against a running server. This one sends 32 requests of 128 random input tokens each, asks for 32 output tokens per request, and keeps at most 8 in flight:

max benchmark --model HuggingFaceTB/SmolLM2-135M-Instruct --dataset-name random \
  --random-input-len 128 --random-output-len 32 \
  --num-prompts 32 --max-concurrency 8 --result-filename result.json

The random dataset gives every run the same shape, which suits comparisons. The default dataset, sharegpt, replays real chat conversations with varied lengths. The helpers below start a server, run a benchmark from Python with the same flags, and read the result file, a JSON object with the summary metrics (request_throughput, mean_ttft_ms, p95_ttft_ms, and so on) and one record per request (request_records):

Show server, benchmark, summary, show 67 lines
@contextmanager
def server(*flags: str) -> Iterator[None]:
    process = start_max_serve(
        MODEL_ID, "--quantization-encoding", "float32", *flags
    )
    try:
        yield
    finally:
        process.terminate()
        process.wait()


def benchmark(concurrency: int, *count: str) -> dict:
    """Runs ``max benchmark`` and returns its result file. ``count`` is
    ``--num-prompts N`` or ``--num-prompts-multiplier M``."""
    result = WORKDIR / f"result_{len(list(WORKDIR.glob('*.json')))}.json"
    subprocess.run(
        [
            "max",
            "benchmark",
            "--model",
            MODEL_ID,
            "--port",
            "8000",
            "--dataset-name",
            "random",
            "--random-input-len",
            "128",
            "--random-output-len",
            "32",
            "--max-concurrency",
            str(concurrency),
            *count,
            "--result-filename",
            str(result),
            "--log-dir",
            str(WORKDIR / "logs"),
        ],
        check=True,
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
    )
    return json.loads(result.read_text())


def summary(run: dict) -> dict[str, float]:
    return {
        "concurrency": run["max_concurrency"],
        "requests/s": run["request_throughput"],
        "output tokens/s": run["total_output_tokens"] / run["duration"],
        "TTFT (ms)": run["mean_ttft_ms"],
        "TPOT (ms)": run["mean_tpot_ms"],
    }


def show(runs: list[dict]) -> None:
    rows = [summary(run) for run in runs]
    print("".join(f"{name:>17}" for name in rows[0]))
    for row in rows:
        print(
            "".join(
                f"{value:>17.1f}"
                if isinstance(value, float)
                else f"{value:>17}"
                for value in row.values()
            )
        )

MAX’s SmolLM2

with server():
    max_runs = [
        benchmark(1, "--num-prompts", "16"),
        benchmark(8, "--num-prompts", "32"),
    ]
show(max_runs)
      concurrency       requests/s  output tokens/s        TTFT (ms)        TPOT (ms)
                1              1.5             46.6             85.4             19.4
                8              8.5            271.8            236.0             22.7

With 8 requests in flight, the server batches them into each decode step. Throughput rises several times, while TPOT barely changes: one decode step costs about the same for 8 sequences as for 1. At this model size, most of a step’s cost is fixed work, such as reading the weights and launching kernels, and little of it is arithmetic per sequence. The handbook’s batching page explains why. TTFT rises because requests that arrive together are prefilled together, and a new request waits for the current step to finish.

The concurrency-8 run sends more requests because max benchmark doesn’t count all of them. A run this short reports over a trimmed window: at concurrency above 1 it leaves out the first max_concurrency requests sent and the last max_concurrency completed, because the server is still filling up or draining while they run, and divides by the time from the first counted request’s submission to the last one’s completion. With at least 100 requests at concurrency above 1, it first looks for a steady-state window automatically and trims only when it finds none. Either way, the result file says which requests counted (measured in each request record).

Your architecture

Now run the same benchmark against the package from Section 7.1:

with server("--custom-architectures", "minillm.serve", "--no-prefer-module-v3"):
    ours = [
        benchmark(1, "--num-prompts", "8"),
        benchmark(8, "--num-prompts", "24"),
    ]  # fewer: this server is slow
show(ours)
      concurrency       requests/s  output tokens/s        TTFT (ms)        TPOT (ms)
                1              0.5             14.7             82.3             67.7
                8              0.2              6.5          17743.1             94.3
def gain(runs: list[dict]) -> float:
    return runs[1]["request_throughput"] / runs[0]["request_throughput"]


def gain_all_requests(runs: list[dict]) -> float:
    rates = []
    for run in runs:
        records = run["request_records"]
        start = min(r["submit_time"] for r in records)
        end = max(r["complete_time"] for r in records)
        rates.append(len(records) / (end - start))
    return rates[1] / rates[0]


print(
    f"throughput gain from concurrency 8: MAX's {gain(max_runs):.1f}x, "
    f"yours {gain(ours):.1f}x"
)
print(
    f"over every request, not just the counted ones: "
    f"MAX's {gain_all_requests(max_runs):.1f}x, "
    f"yours {gain_all_requests(ours):.1f}x"
)
assert gain(max_runs) > 1.5 and gain_all_requests(ours) < 1.5
throughput gain from concurrency 8: MAX's 5.8x, yours 0.4x
over every request, not just the counted ones: MAX's 5.2x, yours 0.7x
Show the code 32 lines
metrics = ["requests/s", "output tokens/s", "TTFT (ms)", "TPOT (ms)"]
fig = make_subplots(
    rows=1, cols=4, horizontal_spacing=0.07, subplot_titles=metrics
)
for name, runs, color in [
    ("MAX's SmolLM2", max_runs, viz.BLUE),
    ("yours (Section 7.1)", ours, viz.ORANGE),
]:
    rows = [summary(run) for run in runs]
    for col, metric in enumerate(metrics, start=1):
        fig.add_trace(
            go.Bar(
                x=[f"c = {r['concurrency']}" for r in rows],
                y=[r[metric] for r in rows],
                name=name,
                marker_color=color,
                legendgroup=name,
                showlegend=col == 1,
                hovertemplate=f"{name}, %{{x}}: %{{y:,.1f}} {metric}<extra></extra>",
            ),
            row=1,
            col=col,
        )
fig.update_yaxes(type="log", title_text="ms (log scale)", row=1, col=3)
fig.update_layout(
    title="Serving SmolLM2 at concurrency 1 and 8",
    barmode="group",
    height=440,
    legend=dict(y=-0.12, yanchor="top"),
    margin=dict(t=96),
)
viz.show(fig)

The two servers differ in two ways, and earlier sections explain both:

  • TPOT is several times higher. The pipeline model from Section 7.1 reruns the whole sequence for every token, as Section 1.7 did, so a step grows with the sequence. The KV cache from Section 1.8 makes it constant.
  • Concurrency doesn’t raise throughput. The package declares max_batch_size 1, so the server runs one request at a time and queues the rest. The queue shows up in TTFT: a request waits for the ones ahead of it to finish completely.

The reported throughput even drops, to half the concurrency-1 rate or less, while counted over every request of the run it drops much less. The trim explains the difference. Here is every request of your server’s concurrency-8 run, from the result file’s request records. The shaded span is the window that max benchmark divides by:

Show the code 56 lines
records = sorted(ours[1]["request_records"], key=lambda r: r["submit_time"])
start = records[0]["submit_time"]
counted = [r for r in records if r["measured"]]
window = (
    min(r["submit_time"] for r in counted) - start,
    max(r["complete_time"] for r in counted) - start,
)
rows = [
    f"request {i}" + ("" if r["measured"] else " (not counted)")
    for i, r in enumerate(records)
]
fig = go.Figure(
    [
        go.Bar(
            y=rows,
            x=[r["ttft_ms"] / 1000 for r in records],
            base=[r["submit_time"] - start for r in records],
            orientation="h",
            name="waiting, then prefill",
            marker_color=viz.MUTED,
            hovertemplate="%{y}: first token after %{x:.1f} s<extra></extra>",
        ),
        go.Bar(
            y=rows,
            x=[(r["latency_ms"] - r["ttft_ms"]) / 1000 for r in records],
            base=[
                r["submit_time"] - start + r["ttft_ms"] / 1000 for r in records
            ],
            orientation="h",
            name="generating",
            marker_color=[
                viz.ORANGE if r["measured"] else viz.GRID for r in records
            ],
            hovertemplate="%{y}: generating for %{x:.1f} s<extra></extra>",
        ),
    ]
)
fig.add_vrect(
    x0=window[0],
    x1=window[1],
    fillcolor=viz.ORANGE,
    opacity=0.08,
    line_width=0,
    annotation_text=(
        f"counted: {len(counted)} requests in {window[1] - window[0]:.0f} s"
    ),
    annotation_position="top left",
)
fig.update_layout(
    title="Your server at concurrency 8: every request of the run",
    barmode="overlay",
    height=560,
    xaxis_title="seconds since the first request was sent",
    yaxis=dict(autorange="reversed", showgrid=False),
)
viz.show(fig)

The first 8 requests are sent at once and finish one after another. Each finished request frees a client slot, so the counted requests, 8 to 15, are sent while 7 earlier requests still wait in the server’s queue. Their window also holds the service time of those 7 uncounted requests: about 15 requests’ work, credited to 8. Over every request, from the first submission to the last completion, throughput stays close to the concurrency-1 rate, as it should for a server that runs one request at a time. The trim suits a server that batches, where the ramp-up and the drain distort the numbers. For a server that serializes requests, read the untrimmed rate, or pass --skip-first-n-requests 0 --skip-last-n-requests 0.

The engine in Chapter 5 fixed both with a paged KV cache and continuous batching. MAX’s own architectures get them from MAX’s serving stack, which is why a production architecture registers a decoder on MAX’s paged KV cache, as the second registration in Section 7.1 does.

Reading benchmarks well

Measure a server at several concurrency levels, not one, and plot latency against throughput. --max-concurrency 1,2,4,8,16,32 runs such a sweep in one command. Pair it with --num-prompts-multiplier 8 instead of a fixed --num-prompts: each level then sends 8 requests per slot, so the trim never uses up a level’s requests, as a fixed 32 would at concurrency 32. The cell below runs the levels one at a time with the same flag, against MAX’s SmolLM2:

with server():
    benchmark(1, "--num-prompts", "2")  # warm up the new server
    sweep = [
        benchmark(c, "--num-prompts-multiplier", "8")
        for c in [1, 2, 4, 8, 16, 32]
    ]
show(sweep)
      concurrency       requests/s  output tokens/s        TTFT (ms)        TPOT (ms)
                1              1.4             45.3            109.9             19.2
                2              2.8             88.0             95.6             20.4
                4              4.8            153.1            148.1             22.2
                8              8.3            265.0            280.8             22.1
               16             10.8            346.3            609.3             24.2
               32             14.2            455.9            999.4             33.3
Show the code 55 lines
TARGET_MS = 500
rows = [summary(run) for run in sweep]
throughput = [r["output tokens/s"] for r in rows]
p95_ttft = [run["p95_ttft_ms"] for run in sweep]
meets = [t <= TARGET_MS for t in p95_ttft]
best = max(
    (i for i, ok in enumerate(meets) if ok),
    key=lambda i: throughput[i],
    default=None,
)
fig = go.Figure(
    go.Scatter(
        x=throughput,
        y=p95_ttft,
        mode="lines+markers+text",
        text=[f"c = {r['concurrency']}" for r in rows],
        textposition="top left",
        marker=dict(
            size=10, color=[viz.BLUE if ok else viz.RED for ok in meets]
        ),
        line=dict(color=viz.MUTED),
        customdata=[
            [r["concurrency"], run["mean_tpot_ms"]]
            for r, run in zip(rows, sweep)
        ],
        hovertemplate="concurrency %{customdata[0]}: %{x:.0f} tokens/s, "
        "p95 TTFT %{y:.0f} ms, mean TPOT %{customdata[1]:.1f} ms<extra></extra>",
    )
)
fig.add_hline(
    y=TARGET_MS,
    line=dict(color=viz.INK_SECONDARY, dash="dash"),
    annotation_text=f"target: p95 TTFT {TARGET_MS} ms",
    annotation_position="top left",
)
if best is not None:
    fig.add_annotation(
        x=throughput[best],
        y=p95_ttft[best],
        text="highest throughput<br>within the target",
        showarrow=True,
        arrowcolor=viz.INK_SECONDARY,
        ax=40,
        ay=40,
    )
fig.update_layout(
    title="MAX's SmolLM2: latency against throughput as concurrency rises",
    height=440,
    showlegend=False,
    xaxis=dict(title="output tokens/s", rangemode="tozero"),
    yaxis=dict(
        title="p95 time to first token (ms, log scale)", type="log", dtick=1
    ),
)
viz.show(fig)

Each point is one concurrency level, and red points miss a target of a 95th-percentile TTFT under 500 ms. Throughput keeps rising with concurrency, but TTFT rises faster once the batch fills. In the run on this page, the highest throughput within the target came at concurrency 8, and concurrency 16 added about a third more throughput at more than twice the p95 TTFT. The levels at 16 and 32 send 128 and 256 requests, enough for max benchmark’s steady-state window; the others are trimmed. Your numbers depend on your hardware.

  • Benchmark the workload you will serve. Input and output lengths change the results a lot: long prompts stress prefill and TTFT, and long answers stress decode and TPOT.
  • Sweep concurrency, as above, with --num-prompts-multiplier rather than a fixed --num-prompts. Throughput grows ever more slowly once the hardware is busy, while latency keeps rising.
  • Set a latency target first, such as a TTFT under 500 ms at the 95th percentile, then find the highest throughput that meets it. The result file stores percentiles (p95_ttft_ms, p99_tpot_ms, and so on) next to the means, and --goodput ttft:500 tpot:50 adds the share of requests that met every target (goodput_attainment).

The handbook’s LLM performance benchmarks page covers benchmark design, and max benchmark --help lists the other datasets and options.

Exercises

  1. Extend the sweep of MAX’s SmolLM2 to concurrency 64 and 128 with --num-prompts-multiplier 4 and --goodput ttft:500. Where does throughput stop growing, and at which level does goodput_attainment fall below 90%?
  2. Repeat the MAX run with --random-input-len 1024. Which metric changes most, and why?
  3. Do the second exercise of Section 7.1, raise max_batch_size in its package to 4, and benchmark it again at concurrency 8.

Next

Section 7.3 constrains what the server generates: JSON that follows a schema, and calls to your own functions.

7.3 Structured output and tools

Open in Colab Open in Kaggle View on GitHub

A chat answer is for people. A program needs output it can parse: a JSON object with the right fields, or the name of a function to call with its arguments. Instructions in the prompt don’t get that reliably, least of all from a small model. This section makes the server guarantee the format with structured output, then builds function calling on top of it.

Show the imports 7 lines
import json

from openai import OpenAI
from pydantic import BaseModel, ConfigDict, Field

from minillm.config import MODEL_ID
from minillm.server import start_max_serve

Structured output is off by default, and --enable-structured-output turns it on:

server = start_max_serve(
    MODEL_ID,
    "--quantization-encoding",
    "float32",
    "--enable-structured-output",
)
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")


def chat(messages: list[dict[str, str]], **options: object) -> str:
    completion = client.chat.completions.create(
        model=MODEL_ID,
        messages=messages,
        max_tokens=100,
        temperature=0,
        **options,
    )
    return completion.choices[0].message.content

JSON that follows a schema

Without a constraint, the model answers in prose:

question = [
    {
        "role": "user",
        "content": "Tell me about Rome: its country and three landmarks.",
    }
]
print(chat(question))
Rome, the Eternal City, is a city that has witnessed so much history and transformation. As a helpful AI assistant, I'm here to share with you the fascinating facts about Rome, its country, and three iconic landmarks.

Rome, the Eternal City, is a city that has witnessed so much history and transformation. It's a place where ancient civilizations like the Romans, Greeks, and Etruscans have left their mark, and it's a place where the Romans have built a lasting

Describe the answer you want as a Pydantic model instead. Pydantic turns it into a JSON schema, which the request passes as response_format:

class City(BaseModel):
    model_config = ConfigDict(extra="forbid")  # no fields beyond these
    name: str
    country: str
    landmarks: list[str] = Field(min_length=3, max_length=3)


schema = {
    "type": "json_schema",
    "json_schema": {"name": "City", "schema": City.model_json_schema()},
}
answer = chat(question, response_format=schema)
print(answer)
city = City.model_validate_json(answer)
city
{"name":"Rome","country":"Italy","landmarks":["The Colosseum","The Roman Forum","The Pantheon"]}
City(name='Rome', country='Italy', landmarks=['The Colosseum', 'The Roman Forum', 'The Pantheon'])

model_validate_json raises if the text doesn’t match the schema. With structured output, the text always matches.

How the server guarantees it

The guarantee comes from the sampler. MAX compiles the schema into a grammar, and at every decode step the grammar says which tokens may come next: after {"name": "Rome", , only tokens that continue toward "country". MAX masks every other token out of the logits before sampling, as the top-p filter in Section 1.7 did with \(-\infty\), so the model can only pick a token that keeps the output valid. Generation ends when the grammar is complete.

The schema fixes the form of the answer and leaves the facts to the model. The model can skip any requirement that the schema leaves out: without min_length, a small model may return an empty landmarks list. Put every requirement you can into the schema, and check the values that matter. The handbook’s structured outputs page covers schema design and the cost of constrained decoding.

Function calling

A tool is a function that the model may ask your program to run: look up the weather, query a database, call an API. The model never runs anything itself. It writes the function’s name and arguments, your code runs the function, and the result goes back into the conversation. A tool call is therefore a structured output whose schema is “one of these functions, with valid arguments for it”.

Here are two tools, each a Python function plus a schema for its arguments:

Show get_weather, convert_currency, arguments 45 lines
RATES = {"USD": 1.0, "EUR": 0.92, "JPY": 151.0}


def get_weather(city: str) -> dict[str, object]:
    return {
        "city": city,
        "temperature_c": 18,
        "sky": "light rain",
    }  # a stand-in for a weather API


def convert_currency(
    amount: float, source: str, target: str
) -> dict[str, object]:
    return {
        "amount": round(amount / RATES[source] * RATES[target], 2),
        "currency": target,
    }


def arguments(**properties: dict[str, object]) -> dict[str, object]:
    return {
        "type": "object",
        "properties": properties,
        "required": list(properties),
        "additionalProperties": False,
    }


TOOLS = {
    "get_weather": (
        get_weather,
        "Gets the current weather in a city.",
        arguments(city={"type": "string"}),
    ),
    "convert_currency": (
        convert_currency,
        "Converts an amount between currencies.",
        arguments(
            amount={"type": "number"},
            source={"enum": list(RATES)},
            target={"enum": list(RATES)},
        ),
    ),
}

The schema for a call allows any one of the tools: anyOf lists one object per tool, each with its fixed name and its own arguments. enum limits the currencies to the ones RATES knows, so the model can’t ask for one the function would fail on. The system prompt tells the model what the tools do:

Show call_tool 44 lines
call_schema = {
    "anyOf": [
        {
            "type": "object",
            "properties": {"name": {"const": name}, "arguments": args},
            "required": ["name", "arguments"],
            "additionalProperties": False,
        }
        for name, (_, _, args) in TOOLS.items()
    ]
}
system = (
    "You can call these functions. Reply with one call as JSON.\n"
    + "\n".join(
        f"- {name}: {description} Arguments: {json.dumps(args['properties'])}"
        for name, (_, description, args) in TOOLS.items()
    )
)


def call_tool(question: str) -> tuple[dict[str, object], dict[str, object]]:
    messages = [
        {"role": "system", "content": system},
        {"role": "user", "content": question},
    ]
    call = json.loads(
        chat(
            messages,
            response_format={
                "type": "json_schema",
                "json_schema": {"name": "call", "schema": call_schema},
            },
        )
    )
    function = TOOLS[call["name"]][0]
    return call, function(**call["arguments"])


for question in [
    "What's the weather in Tokyo?",
    "How much is 20 US dollars in euros?",
]:
    call, result = call_tool(question)
    print(f"{question}\n  call:   {call}\n  result: {result}")
What's the weather in Tokyo?
  call:   {'name': 'get_weather', 'arguments': {'city': 'Tokyo'}}
  result: {'city': 'Tokyo', 'temperature_c': 18, 'sky': 'light rain'}
How much is 20 US dollars in euros?
  call:   {'name': 'convert_currency', 'arguments': {'amount': 20, 'source': 'USD', 'target': 'EUR'}}
  result: {'amount': 18.4, 'currency': 'EUR'}

Every call parses and runs, since the grammar allows nothing else. Whether the model picks the right tool is up to the model; this one does on these questions.

The last step gives the result back so the model can answer. Here the result goes into the system prompt:

question = "Should I take an umbrella in Paris today?"
call, result = call_tool(question)
print("call:", call)
print(
    chat(
        [
            {
                "role": "system",
                "content": f"Current weather: {json.dumps(result)}",
            },
            {"role": "user", "content": question},
        ]
    )
)
call: {'name': 'get_weather', 'arguments': {'city': 'Paris'}}
As a traveler, it's essential to consider the weather forecast and potential weather-related issues before making a decision. Paris is a city known for its rain, and the city's weather can be unpredictable.

If you're planning to visit Paris in the morning, it's best to take an umbrella. The city's rain can be heavy and unpredictable, especially in the mornings. A rain umbrella can provide some protection from the rain, but it's not a guarantee.

If you're
server.terminate()
_ = server.wait()

A 135M-parameter model uses the result loosely. Larger models follow it much more closely, especially ones trained for function calling.

Those models also take tools through the tools parameter of the OpenAI API. Their chat templates describe the tools to the model in the format it was trained on, and the model writes calls in its own format. MAX parses those calls with a tool parser for each model family, such as Llama, Qwen, and DeepSeek, and returns them as message.tool_calls; tool_choice="required" forces a call. max serve --tool-parser <name> picks a different parser than the architecture’s default, and --tool-parser none turns parsing off. SmolLM2’s chat template has no place for tools, which is why this section built the protocol by hand. The mechanism is the same in both cases: the model writes a structured call, and your code runs it. The handbook’s function calling page covers the API and the agent loops built on it.

Exercises

  1. Remove Field(min_length=3, max_length=3) from City. What does the model return now?
  2. Add a third tool, get_time(city), and a question that needs it. Does the model choose it?
  3. max benchmark takes --response-format with a JSON schema. Start the server with --enable-structured-output, then benchmark it with and without one, as in Section 7.2. How much does constrained decoding cost per token? Compare TPOT: constrained requests stop at the end of the JSON, so their output lengths differ.

Next

Section 7.4 points the same client at state-of-the-art open models on Modular’s cloud.

7.4 Modular’s cloud

Open in Colab Open in Kaggle View on GitHub

Everything in this book ran on your own machine: models of up to 0.6B parameters, with multi-device meshes simulated on a CPU. Serving state-of-the-art open models to real traffic takes more: the largest models span many GPUs, capacity has to follow demand, and someone has to respond when a node fails. Modular’s cloud is an inference service that runs those models on MAX, the framework you used in this book. This last section points your client code at it by changing the base URL, the API key, and the model name.

import os

from openai import OpenAI

from minillm.config import MODEL_ID
from minillm.server import start_max_serve

Two ways to call a hosted model

Modular’s cloud serves popular open models behind the same OpenAI-compatible API that the earlier sections of this chapter used:

  • Shared endpoints are billed per token, and their capacity scales automatically, down to zero when idle.
  • Dedicated endpoints reserve GPUs for you and are billed per minute, for consistent latency and guaranteed throughput.

The pricing page compares them.

The same client, a different URL

Every server in Chapter 7 spoke the OpenAI API, and so do Modular’s endpoints. Here is the client from Section 7.1 against a local server again:

server = start_max_serve(MODEL_ID, "--quantization-encoding", "float32")


def ask(client: OpenAI, model: str, question: str) -> str:
    completion = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": question}],
        max_tokens=60,
        temperature=0,
    )
    return completion.choices[0].message.content


local = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
print(ask(local, MODEL_ID, "What is the capital of France?"))
server.terminate()
_ = server.wait()
The capital of France is Paris.

To call a hosted model instead:

  1. Sign up at console.modular.com.
  2. Create an API key with model access on the API keys page, and set it as the MODULAR_API_KEY environment variable. In Colab, add it as a secret and copy it into os.environ.
  3. Pick a model on the endpoints page, and set its model ID as MODULAR_MODEL.

Only the base URL, the key, and the model name change. The cell skips the request when no key is set:

api_key = os.environ.get("MODULAR_API_KEY")
if api_key is None:
    print("Set MODULAR_API_KEY and MODULAR_MODEL to call a hosted model.")
else:
    hosted = OpenAI(base_url="https://api.modular.com/v1", api_key=api_key)
    print(
        ask(
            hosted,
            os.environ["MODULAR_MODEL"],
            "What is the capital of France?",
        )
    )
Set MODULAR_API_KEY and MODULAR_MODEL to call a hosted model.

The API is the same, so code written against it, such as the structured output in Section 7.3, works with hosted models that support the feature. max benchmark measures a hosted endpoint as it measured your local server. The mcloud backend sends the key from OPENAI_API_KEY. A hosted endpoint manages its own cache and doesn’t expose server metrics, so the command skips both. If the model ID isn’t a Hugging Face repo, add --tokenizer with the repo of its tokenizer:

OPENAI_API_KEY=$MODULAR_API_KEY max benchmark --backend mcloud \
  --base-url https://api.modular.com --endpoint /v1/chat/completions \
  --no-flush-prefix-cache --no-collect-server-stats \
  --model $MODULAR_MODEL \
  --dataset-name random --random-input-len 128 --random-output-len 32 \
  --num-prompts 32 --max-concurrency 8

Before you choose between shared and dedicated endpoints, benchmark with your real prompt and answer lengths; the latency targets from Section 7.2 apply unchanged. The handbook’s choosing the right model and calculating GPU memory pages help you choose a model and estimate the GPUs it needs.

What you built

You started this book with three lines of AutoPipeline. Since then you:

  • wrote SmolLM2 from tensors up, checked it against MAX’s implementation, and added sampling and a KV cache (Chapter 1);
  • sharded it across a device mesh (Chapter 2);
  • wrote a Mojo kernel and built a graph by hand (Chapter 3);
  • tested, benchmarked, and profiled it (Chapter 4);
  • built a serving engine with continuous batching, a paged KV cache, prefix caching, and speculative decoding (Chapter 5);
  • brought up Qwen3, Gemma 3, and a mixture of experts (Chapter 6);
  • served your own architecture, measured it, and constrained its output (Chapter 7).

A model takes the same path into MAX, at a larger size. To keep going:

  • Read MAX’s guide to model bring-up and port a model you care about.
  • Work through the Mojo GPU puzzles to write the kernels behind it.
  • Read the handbook for the ideas behind every chapter.
  • Sign up to run state-of-the-art open models on Modular’s cloud, or request a demo to talk about your workload.