Build and serve an LLM with MAX
This book takes you from zero to serving your own language model with MAX, Modular’s framework for AI inference. You start by running a real model in three lines of Python. Then you rebuild that model from scratch, check every part against MAX’s own implementation, and go on to the techniques that production inference engines use: distributing a model across devices, writing your own Mojo kernels, testing and benchmarking, building a serving engine, bringing up newer models, and serving your own architecture.
Why this book
Most explanations of LLM inference stop at diagrams, and most inference engines are too large to read. This book sits between the two. You write every piece yourself, small enough to read in one sitting, and you run it on a real model so that each claim is a number you measured: the logits match the reference, the KV cache stops each new token from recomputing the whole text, batching raises throughput.
You do it with MAX’s own APIs, the same ones its production models use, so what you learn carries over directly:
max.pipelines:AutoPipeline,AutoModel, andAutoTokenizerfor running models from Python.max.tensorandmax.functional: tensors and operations that run eagerly, like NumPy, and compile into fast graphs.max.nn: modules and layers for building models.- Mojo custom ops: kernels you write yourself and call from Python.
max.graph: the graph API thatcompilebuilds on, for full control.max serve,max generate, andmax benchmark: the command-line tools for serving, testing, and measuring models.
Every section is a notebook with interactive figures you can hover, zoom, and slide through, and every notebook checks its own results.
The model
The book uses SmolLM2-135M-Instruct. It is a real chat model, small enough to run on a laptop CPU, and it uses the Llama architecture that most open models share: RMSNorm, rotary position embeddings, grouped-query attention, and a SwiGLU MLP. What you build for it applies to Llama, Qwen, Mistral, and their relatives, and Chapter 6 extends it to Qwen3, Gemma 3, and mixture-of-experts models.
Get started
Check your hardware first
Before you install anything, check that MAX supports your machine and GPU in the MAX system requirements. MAX runs on Linux and on macOS with Apple silicon. On NVIDIA, the notebooks need an Ampere-generation GPU or newer; on an older one, such as a T4 or P100, they run on the CPU instead.
You can run the notebooks in Colab or Kaggle with nothing installed, or on your own machine.
In Colab or Kaggle
Every section page starts with Open in Colab and Open in Kaggle buttons. The first cell of every notebook installs what the notebook needs, so run the cells in order from the top:
# The setup cell, shortened: it installs MAX and this book's package
# with uv, but only if they're missing.
if importlib.util.find_spec("max") is None:
uv_install("max[all]") # from https://whl.modular.com/nightly/simple/
if importlib.util.find_spec("minillm") is None:
uv_install("git+https://github.com/modular/max-llm-book@main")
max[all] is the MAX package with everything the book uses: the Python
APIs, the max and mojo commands, serving, and the Jupyter %%mojo
magic. minillm is the package you build in the book, installed so that
later sections can import what earlier ones wrote.
Choose the runtime before you run the first cell:
- In Colab, the CPU runtime works for every notebook. For a GPU, choose an L4 or A100 runtime. MAX’s matrix multiplication needs an NVIDIA GPU from the Ampere generation or later, so on a T4 runtime the setup cell hides the GPU and the notebook runs on the CPU.
- In Kaggle, turn on Internet in the notebook settings so that the setup cell can download packages. Kaggle’s GPUs, the T4 and the P100, are older than Ampere, so the notebooks run on the CPU there.
On your own machine
Install with pixi or uv, on a machine that meets the system requirements.
With pixi, which installs Python, MAX, and every other dependency into a project environment:
-
Install pixi:
curl -fsSL https://pixi.sh/install.sh | sh -
Clone the book:
git clone https://github.com/modular/max-llm-book cd max-llm-book -
Open the notebooks in JupyterLab:
pixi run notebook
With uv and Python 3.10 or later:
-
Create and activate an environment:
uv venv --python 3.12 source .venv/bin/activate -
Install MAX from the nightly package index:
uv pip install --prerelease allow --index-strategy unsafe-best-match \ --extra-index-url https://whl.modular.com/nightly/simple/ "max[all]" -
Clone the book and install its package, with JupyterLab:
git clone https://github.com/modular/max-llm-book cd max-llm-book uv pip install -e . jupyterlab -
Open the notebooks:
jupyter lab notebooks
Check the installation, and whether MAX found a GPU:
max --version
python -c "from max.driver import accelerator_count; print('GPUs:', accelerator_count())"
The book uses the MAX nightly build, because some of the APIs it teaches are newer than the last stable release. Every notebook runs on a CPU and uses a GPU automatically when MAX finds a supported one.
About the numbers
Timings, speedups, and other measurements on these pages come from one run on an Apple-silicon Mac with a Metal GPU. Treat them as examples. Your numbers depend on your hardware and vary from run to run; the notebooks check results, such as matching logits, that hold on every machine.
Chapters
| Chapter | What you do |
|---|---|
| 1. Foundations | Run and serve SmolLM2, then rebuild it from scratch: tensors, weights, layers, attention, the full model checked against MAX’s, sampling, and a KV cache. |
| 2. Distributed tensors | Shard tensors and the model across a device mesh, simulated on one machine, and check that the result doesn’t change. |
| 3. Custom ops in Mojo | Write a kernel in Mojo, call it from your model, and see the graph API that compile builds on. |
| 4. Testing and benchmarking | Test layers against references, verify logits, and benchmark and profile models and kernels. |
| 5. A serving engine | Build the core of an inference engine: continuous batching, a paged KV cache, prefix caching, and speculative decoding. |
| 6. Bigger models | Extend your model to Qwen3, Gemma 3, and mixture-of-experts. |
| 7. Production serving | Serve your own architecture with max serve, benchmark it, use structured output and tools, and call state-of-the-art open models on Modular’s cloud. |
Read the chapters in order the first time: each section builds on code from
earlier ones, which the minillm package collects for you.
The handbook
The Modular handbook explains how LLM inference works in production: metrics, batching, KV cache management, quantization, and GPU kernels. This book is its hands-on companion. Each section links the handbook pages for the ideas it implements, so you can read the concept and then build it.
Prerequisites
You should be comfortable with Python and NumPy-style array code. Knowing
what a neural network layer is helps; the book explains everything specific
to transformers as it goes. Section 6.2 uses Gemma 3,
whose weights require you to accept Google’s license on Hugging Face and log
in with hf auth login.
1. Foundations
This chapter goes from running a model to rebuilding it. You run and serve SmolLM2-135M-Instruct with MAX, then rewrite it from scratch, one part per section, until your version produces the same logits as MAX’s and generates text with a KV cache.
| Section | You learn |
|---|---|
| 1.1 Run and serve an LLM | AutoPipeline, AutoModel, max generate, max serve, and the two numbers that describe speed. |
| 1.2 Tensors and eager execution | max.tensor, max.functional, and max.nn: eager execution and compilation. |
| 1.3 Tokens, embeddings, and weights | What a config and a checkpoint hold, and how to build the token embedding from them. |
| 1.4 Building blocks | RMSNorm, the SwiGLU MLP, and rotary position embeddings. |
| 1.5 Attention | Causal grouped-query attention, first from basic operations and then with flash attention. |
| 1.6 The full model | Assembling 30 layers and matching MAX’s logits. |
| 1.7 Generation and sampling | The generation loop, temperature, top-k, and top-p. |
| 1.8 The KV cache | Caching keys and values so that decoding stops redoing work. |
Every section is a notebook that runs in a few minutes, on a GPU if you have
one and on a CPU otherwise. Sections 1.2 to 1.8 add code to the minillm
package as they go, so each section can start from the previous ones without
rerunning them.
By the end, you have a working LLM inference stack that you wrote yourself, and you know the parts of MAX it is built from. Chapter 2 splits that model across several devices.
1.1 Run and serve an LLM
In this section you run a real language model with MAX four ways, from three lines of Python to an OpenAI-compatible server:
AutoPipeline: text in, text out.AutoTokenizerandAutoModel: the tokens and scores underneath.max generate: the same thing from the command line, with timings.max serve: an HTTP server that any OpenAI client can call.
The model is SmolLM2-135M-Instruct, a 135M-parameter chat model from Hugging Face. It is small enough to run anywhere, and its architecture (the Llama family) is the one most open models use. The rest of the book rebuilds it from scratch.
If you are new to LLM inference, read What is LLM inference? in the Modular handbook first.
Setup
MAX runs on a GPU if it finds one, otherwise on the CPU. Every call below
asks for float32 numbers. MAX requires float32 on a CPU, since it runs
bfloat16 weights only on GPUs; on a GPU, float32 keeps the results
comparable with the rest of the book. The first run downloads the
model (about 270 MB) and compiles it, which takes a minute; later runs reuse
both caches.
import logging
import time
import numpy as np
import plotly.graph_objects as go
from max.pipelines import AutoPipeline
from openai import OpenAI
from transformers import AutoTokenizer as HFTokenizer
from minillm import viz
# MAX logs performance hints that don't matter at this model size.
logging.getLogger("max").setLevel(logging.ERROR)
MODEL = "HuggingFaceTB/SmolLM2-135M-Instruct"
FLOAT32 = dict(quantization_encoding="float32")
Generate text in three lines
AutoPipeline loads a model and its tokenizer and generates text, much like
transformers.pipeline. top_k=1 makes generation greedy: it always picks
the most likely next token, so the output is the same on every run.
pipe = AutoPipeline(MODEL, **FLOAT32)
completion = pipe(["The capital of France is"], max_new_tokens=16, top_k=1)
assert "Paris" in completion[0]
completion
['The capital of France is Paris. Paris is the largest city in France and the capital of the French department']
A plain string goes to the model as is, and the output starts with the prompt. An instruct model is trained on conversations, so a list of chat messages usually works better. The pipeline formats them with the model’s chat template and returns only the reply.
messages = [{"role": "user", "content": "What is the capital of France?"}]
pipe([messages], max_new_tokens=32, top_k=1)
['The capital of France is Paris.']
Look underneath: tokens and scores
A language model doesn’t read text. A tokenizer splits the text into tokens (whole words, pieces of words, or punctuation) and maps each to an integer ID. SmolLM2’s vocabulary has 49,152 tokens.
tokenizer = pipe.tokenizer
ids = tokenizer(["The capital of France is"])["input_ids"][0]
print(ids)
viz.tokens([tokenizer.decode([i]) for i in ids], ids)
[504, 3575, 282, 4649, 314]
Each chip is one token; hover over it for its ID, and the dots mark spaces. Common words are single tokens, usually with their leading space. Rare words, numbers, and code split into pieces:
text = (
"Tokenizers split unbelievably rare words, 3.14159, and snake_case_names."
)
pieces = tokenizer([text])["input_ids"][0]
viz.tokens([tokenizer.decode([i]) for i in pieces], pieces)
A chat message is text and tokens too. The chat template wraps each message
in special tokens and adds a default system prompt. MAX loads the model’s
Hugging Face tokenizer, so the transformers library can show what the
question from earlier becomes before the model sees it:
template = HFTokenizer.from_pretrained(MODEL).apply_chat_template(
messages, add_generation_prompt=True, tokenize=False
)
chat_ids = tokenizer([template])["input_ids"][0]
print(len(chat_ids), "tokens")
viz.tokens([tokenizer.decode([i]) for i in chat_ids], chat_ids)
37 tokens
Given the IDs, the model produces one score (a logit) for every token in
the vocabulary: its prediction of what comes next. AutoModel.generate can
return those scores. Softmax turns them into probabilities: it
exponentiates each logit \(z_i\) and divides by the total, so every
probability is positive and they sum to 1:
\[p_i = \frac{e^{z_i}}{\sum_j e^{z_j}}\]
model = pipe.model
out = model.generate(input_ids=[ids], max_new_tokens=1, output_logits=True)
logits = out.logits[0][0] # first prompt, first new token: [vocab_size]
probs = np.exp(logits - logits.max())
probs /= probs.sum()
assert tokenizer.decode([int(probs.argmax())]) == " Paris"
Show the code 20 lines
top = np.argsort(probs)[::-1][:10]
fig = go.Figure(
go.Bar(
x=probs[top],
y=[repr(tokenizer.decode([int(i)])) for i in top],
orientation="h",
marker_color=viz.BLUE,
text=[f"{p:.1%}" for p in probs[top]],
textposition="outside",
customdata=top,
hovertemplate="%{y} (ID %{customdata}): %{x:.2%}<extra></extra>",
)
)
fig.update_layout(
title="The ten likeliest tokens after 'The capital of France is'",
height=420,
xaxis=dict(title="probability", tickformat=".0%", range=[0, 1.1]),
yaxis=dict(autorange="reversed", showgrid=False),
)
viz.show(fig)
Generation repeats this step: pick a token from the distribution, append it, and run the model again. Picking the top token is greedy decoding; sampling from the distribution gives more varied text. Section 1.7 implements both.
Measure it
Two numbers describe how fast a model feels:
- Time to first token (TTFT) is how long the user waits before anything appears. It covers prefill, which processes the whole prompt at once.
- Time per output token (TPOT) is how long each later token takes. Each one is a decode step that processes a single new token.
Generating one token measures prefill; generating 65 adds 64 decode steps. Each timing below is the median of three runs:
def seconds(prompt: list[int], n_tokens: int) -> float:
start = time.perf_counter()
model.generate(
input_ids=[prompt], max_new_tokens=n_tokens, top_k=1, ignore_eos=True
)
return time.perf_counter() - start
def ttft_tpot(prompt: list[int], repeats: int = 3) -> tuple[float, float]:
"""Returns the median TTFT and TPOT of ``prompt``, in seconds."""
ttft = float(np.median([seconds(prompt, 1) for _ in range(repeats)]))
total = float(np.median([seconds(prompt, 65) for _ in range(repeats)]))
return ttft, (total - ttft) / 64
ttft, tpot = ttft_tpot(ids)
print(
f"TTFT {ttft * 1000:.0f} ms, TPOT {tpot * 1000:.1f} ms ({1 / tpot:.0f} tokens/s)"
)
TTFT 11 ms, TPOT 9.2 ms (108 tokens/s)
Prefill processes every prompt token, so TTFT should grow with the prompt. A decode step processes one token, so TPOT should barely change. Check both with prompts of random token IDs from 8 to 2,048 tokens long:
rng = np.random.default_rng(0)
prompt_lengths = [8, 64, 256, 1024, 2048]
ttfts, tpots = [], []
for length in prompt_lengths:
prompt = rng.integers(0, len(probs), length).tolist()
seconds(prompt, 2) # warm up this length
ttft_s, tpot_s = ttft_tpot(prompt)
ttfts.append(ttft_s * 1000)
tpots.append(tpot_s * 1000)
print(
f"{length:5} prompt tokens: TTFT {ttfts[-1]:6.1f} ms, "
f"TPOT {tpots[-1]:5.1f} ms"
)
8 prompt tokens: TTFT 11.0 ms, TPOT 9.3 ms
64 prompt tokens: TTFT 14.8 ms, TPOT 9.2 ms
256 prompt tokens: TTFT 73.6 ms, TPOT 9.4 ms
1024 prompt tokens: TTFT 670.0 ms, TPOT 9.4 ms
2048 prompt tokens: TTFT 2470.4 ms, TPOT 9.1 ms
Show the code 23 lines
fig = go.Figure(
[
go.Scatter(
x=prompt_lengths,
y=ms,
name=name,
mode="lines+markers",
line_color=color,
hovertemplate=name + ", %{x} tokens: %{y:.1f} ms<extra></extra>",
)
for name, ms, color in [
("time to first token", ttfts, viz.ORANGE),
("time per output token", tpots, viz.BLUE),
]
]
)
fig.update_layout(
title="Prefill grows with the prompt; a decode step barely does",
height=420,
xaxis=dict(title="prompt length (tokens, log scale)", type="log", dtick=1),
yaxis=dict(title="ms (log scale)", type="log", dtick=1),
)
viz.show(fig)
TTFT grows with the prompt, steeply past a few hundred tokens, while TPOT stays nearly flat. A decode step still attends over the whole sequence, but that work is small next to reading every weight once per step.
ignore_eos=True keeps generating past the end-of-sequence token, so every run
does the same amount of work.
LLM inference metrics
in the handbook defines these and the other metrics engines report.
Close the pipeline to free its memory. max generate and max serve
below load the model again.
pipe.close()
The command line: max generate
The max command, installed with MAX, runs the same pipeline from a shell.
In a notebook, a line that starts with ! runs as a shell command.
max generate applies the chat template to --prompt and prints timing
statistics after the text.
!max generate --model HuggingFaceTB/SmolLM2-135M-Instruct --quantization-encoding float32 \
--prompt "What is the capital of France?" --max-new-tokens 32 --top-k 1 2>/dev/null
The capital of France is Paris.
Prompt size: 37
Output size: 8
Startup time: 0.32830238342285156 ms
Time to first token: 29.28781509399414 ms
Prompt eval throughput (context-encoding): 1263.3240097035216 tokens per second
Time per Output Token: 9.826013020106725 ms
Eval throughput (token-generation): 101.77067727813153 tokens per second
Total Latency: 98.06990623474121 ms
Total Throughput: 10.196807954586895 req/s
The lines to read are time to first token and time per output token: TTFT
and TPOT again. Expect different values from the ones above, since this is
one timed run of a different prompt. The prompt size is 37 tokens, not the 7
words typed, because the chat template adds the special tokens and system
prompt shown earlier. 2>/dev/null hides the startup log. Run max generate --help
to see every option.
Serve the model
max serve starts an HTTP server with an OpenAI-compatible API, so any
OpenAI client library or tool can use your model. In a terminal you would run:
max serve --model HuggingFaceTB/SmolLM2-135M-Instruct \
--quantization-encoding float32
From a notebook, start it in the background and wait until its /health
endpoint answers:
Show start_max_serve 29 lines
from __future__ import annotations
import subprocess
import time
import urllib.request
def start_max_serve(
model: str, *flags: str, port: int = 8000
) -> subprocess.Popen:
"""Starts ``max serve`` and returns once its health check answers.
Call ``terminate()`` on the returned process to stop the server.
"""
process = subprocess.Popen(
["max", "serve", "--model", model, "--port", str(port), *flags],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
while True:
try:
urllib.request.urlopen(f"http://localhost:{port}/health")
return process
except OSError:
if process.poll() is not None:
raise RuntimeError(
"max serve exited; run it in a terminal to see why"
) from None
time.sleep(1)
Saved to minillm/server.py.
start = time.perf_counter()
server = start_max_serve(MODEL, "--quantization-encoding", "float32")
print(f"server ready in {time.perf_counter() - start:.0f} s")
server ready in 17 s
Now call it with the official openai package. The API key is unused but
required by the client.
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
reply = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": "Name three primary colors."}],
max_tokens=128,
temperature=0,
)
print(reply.choices[0].message.content)
print(
reply.usage.prompt_tokens,
"prompt tokens,",
reply.usage.completion_tokens,
"completion tokens",
)
assert reply.choices[0].message.content.strip()
The primary colors are:
1. Red: A warm, reddish-brown color that is often associated with fire, blood, and fire.
2. Blue: A cool, deep, and deep blue color that is often associated with water, sky, and sky.
3. Yellow: A bright, golden yellow color that is often associated with earth, light, and light.
These colors are often used to represent the primary elements of a painting or artwork, and they can be used to create a sense of depth and dimensionality.
35 prompt tokens, 112 completion tokens
Chat apps show the reply as it is generated. With stream=True the server
sends each token as soon as the model produces it:
stream = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": "Count from one to ten in words."}],
max_tokens=48,
temperature=0,
stream=True,
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="", flush=True)
1. One
2. Two
3. Three
4. Four
5. Five
6. Six
7. Seven
8. Eight
9. Nine
10. Ten
server.terminate()
_ = server.wait()
OpenAI-compatible API
in the handbook covers the request format, and
Inference parameters
explains temperature, top_p, and the other sampling options.
What MAX did
Each call above ran the same pipeline:
Inside, MAX also compiled the model into optimized kernels, kept a KV cache so each decode step processes one token instead of the whole sequence, and scheduled requests so the server can batch them. In the rest of the book you build each of these parts yourself:
| Where | You build | With |
|---|---|---|
| 1.2 | Tensors, eager execution, compilation | max.tensor, max.functional |
| 1.3 | Tokens, embeddings, weight loading | max.nn.Embedding, safetensors |
| 1.4 | RMSNorm, rotary embeddings, the MLP | max.nn layers |
| 1.5 | Causal grouped-query attention | max.functional, flash attention |
| 1.6 | The full model, checked against AutoModel | max.nn.Module, compile |
| 1.7 | Generation and sampling | your model |
| 1.8 | A KV cache | your model |
| 2 | The model split across several devices | max.tensor.sharding |
| 3 | Your own kernels in Mojo | %%mojo, F.custom, max.graph |
| 4 to 7 | Testing, a serving engine, bigger models, production serving |
Exercises
- Ask the chat model a few questions of your own. Where does a 135M model go wrong?
- Pass
temperature=0.8instead oftop_k=1and run the same prompt several times. What changes? - Load the model with
AutoModel(MODEL, max_batch_size=4, **FLOAT32)and time four prompts at once (input_ids=[ids] * 4). How do TTFT and TPOT change compared with one prompt?
Next
Section 1.2 introduces
the tensor library you build the model with: max.tensor, max.functional, and
eager execution.
1.2 Tensors and eager execution
Every model in this book is written with three MAX modules:
max.tensor: theTensortype.max.functional(imported asF): operations on tensors, such asF.softmaxandF.silu.max.nn:Moduleand layers such asLinear, built from the two above.
This section covers what you need from each: creating tensors, running operations, and moving data to and from NumPy. It also shows how MAX differs from NumPy. Operations run eagerly, and the same code also compiles into a fast graph.
import logging
import time
import numpy as np
import plotly.graph_objects as go
from max import functional as F
from max.driver import CPU, accelerator_count
from max.dtype import DType
from max.graph import TensorType
from max.nn import Linear, Module
from max.tensor import Tensor, default_device, default_dtype
from minillm import viz
logging.getLogger("max").setLevel(logging.ERROR)
Pick a device
MAX runs the same code on CPUs and on NVIDIA, AMD, and Apple silicon GPUs.
max.tensor.defaults() returns the dtype and the device mesh MAX would
choose on its own, and pick_device takes the mesh’s first device: a GPU if
MAX finds one, otherwise the CPU. Every notebook in this book uses that
device, so on Colab or Kaggle with a GPU runtime, or on a Mac, your code
runs on the GPU without changes.
Code cells that start with # export: become part of this book’s minillm
package, so later sections import them instead of repeating them.
pick_device is the first:
from __future__ import annotations
from max.driver import Device
from max.tensor import defaults
def pick_device() -> Device:
"""Returns the device MAX uses by default: a GPU if there is one,
otherwise the CPU."""
_, mesh = defaults()
return mesh.devices[0]
Saved to minillm/device.py.
DEVICE = pick_device()
print("GPUs found:", accelerator_count(), " using:", DEVICE)
GPUs found: 1 using: Device(type=gpu,id=0)
Create tensors
A Tensor has a shape, a dtype, and a device. You can build one from Python
lists, from a NumPy array, or with a constructor such as Tensor.ones. The
examples name the dtype, float32, for a reason the next cells show:
x = Tensor(
[[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]], dtype=DType.float32, device=DEVICE
)
print(x)
print(x.shape, x.dtype, x.device)
Tensor([1 2 3
4 5 6], dtype=DType.float32, device=Device(type=gpu,id=0))
[Dim(2), Dim(3)] DType.float32 Device(type=gpu,id=0)
print(Tensor(np.arange(6, dtype=np.float32).reshape(2, 3), device=DEVICE))
print(Tensor.zeros([2, 2], dtype=DType.float32, device=DEVICE))
print(Tensor.arange(5, dtype=DType.float32, device=DEVICE))
Tensor([0 1 2
3 4 5], dtype=DType.float32, device=Device(type=gpu,id=0))
Tensor([0 0
0 0], dtype=DType.float32, device=Device(type=gpu,id=0))
Tensor([0 1 2 3 4], dtype=DType.float32, device=Device(type=gpu,id=0))
Without device= or dtype=, MAX also picks the dtype: bfloat16 on a GPU, a
16-bit format most models ship in, and float32 on a CPU. This book computes in
float32 everywhere, so its results can be compared exactly with reference
implementations; the exercises in
Section 4.2 measure
what bfloat16 changes. NumPy also has no bfloat16 type, so a GPU tensor left
at the default dtype can’t be read with to_numpy().
print("default placement:", Tensor.ones([2]).device, Tensor.ones([2]).dtype)
default placement: Device(type=gpu,id=0) DType.bfloat16
To change the defaults for a block of code, use the default_device and
default_dtype context managers. Later sections build models inside them.
with default_device(DEVICE), default_dtype(DType.float32):
y = Tensor.ones([2, 2])
print(y.device, y.dtype)
Device(type=gpu,id=0) DType.float32
Run operations
Python operators work as in NumPy, including @ for matrix multiplication
and .T to transpose the last two axes. max.functional holds everything
else.
print(x @ x.T)
print(x.sum(axis=-1))
print(x[:, 1:])
print(x.reshape([3, 2]))
print(F.softmax(x, axis=-1))
Tensor([14 32
32 77], dtype=DType.float32, device=Device(type=gpu,id=0))
Tensor([ 6
15], dtype=DType.float32, device=Device(type=gpu,id=0))
Tensor([2 3
5 6], dtype=DType.float32, device=Device(type=gpu,id=0))
Tensor([1 2
3 4
5 6], dtype=DType.float32, device=Device(type=gpu,id=0))
Tensor([0.09003 0.2447 0.6652
0.09003 0.2447 0.6652], dtype=DType.float32, device=Device(type=gpu,id=0))
Results are tensors on the same device as their inputs. To read one as a
NumPy array, call to_numpy(), which copies it from the GPU first if
needed. For a tensor already in CPU memory, np.from_dlpack() shares the
memory instead of copying:
print(x.to_numpy())
on_cpu = x.to(CPU())
print(np.from_dlpack(on_cpu).dtype)
[[1. 2. 3.]
[4. 5. 6.]]
float32
Eager execution
Each operation above ran immediately, so you could print its result. MAX builds a small graph for each operation, compiles it into a kernel the first time it sees that operation and shape, and caches the kernel. Compare the first call with the later ones:
z = Tensor(np.random.rand(256, 256).astype(np.float32), device=DEVICE)
call_ms = []
for i in range(4):
start = time.perf_counter()
F.gelu(z @ z).to_numpy()
call_ms.append((time.perf_counter() - start) * 1000)
print(f"call {i}: {call_ms[-1]:.0f} ms")
call 0: 4380 ms
call 1: 6 ms
call 2: 5 ms
call 3: 5 ms
Show the code 14 lines
fig = go.Figure(
go.Bar(
x=[f"call {i}" for i in range(4)],
y=call_ms,
marker_color=[viz.ORANGE] + [viz.BLUE] * 3,
hovertemplate="%{x}: %{y:.1f} ms<extra></extra>",
)
)
fig.update_layout(
title="F.gelu(z @ z): the first call compiles, later calls reuse the kernels",
yaxis=dict(title="ms (log scale)", type="log", dtick=1),
height=360,
)
viz.show(fig)
The first call pays for compilation. On this page’s machine, an Apple-silicon
Mac with a Metal GPU, it took several seconds, and the later calls took a few
milliseconds. Treat these numbers as an example: yours depend on your hardware
and vary from run to run. The cache persists across sessions, and
max warm-interpreter-cache compiles the common kernels ahead of time.
Defer work with F.lazy
Inside F.lazy(), operations record a graph without running it. The
tensors stay unrealized until you read a value:
with F.lazy():
u = F.silu(x) * 2
print(u)
print(u.to_numpy())
Tensor(
'<unrealized>',
shape=[Dim(2), Dim(3)],
dtype=float32,
device=Device(type=gpu,id=0)
)
[[ 1.4621172 3.523188 5.7154446]
[ 7.85611 9.933072 11.970329 ]]
Later sections build models inside F.lazy(), so the random initial weights
are never computed: real weights from the checkpoint replace them.
Fuse operations with @F.functional
A function decorated with @F.functional runs as one graph, so MAX can
fuse its operations into fewer kernels:
@F.functional
def silu_times_two(t: Tensor) -> Tensor:
return F.silu(t) * 2
print(silu_times_two(x))
Tensor([1.462 3.523 5.715
7.856 9.933 11.97], dtype=DType.float32, device=Device(type=gpu,id=0))
Modules and compilation
A Module groups parameters (tensors) and a forward method. Here is the
feed-forward block SmolLM2 uses, at its real size: hidden size 576, intermediate
size 1536. Section 1.4
explains the design; for now it’s a realistic workload.
class MLP(Module):
def __init__(self, hidden: int, intermediate: int) -> None:
self.gate_proj = Linear(hidden, intermediate, bias=False)
self.up_proj = Linear(hidden, intermediate, bias=False)
self.down_proj = Linear(intermediate, hidden, bias=False)
def forward(self, x: Tensor) -> Tensor:
return self.down_proj(F.silu(self.gate_proj(x)) * self.up_proj(x))
with default_device(DEVICE), default_dtype(DType.float32):
mlp = MLP(576, 1536)
print(mlp)
print([name for name, _ in mlp.parameters])
MLP(
gate_proj=Linear(in_dim=Dim(576), out_dim=Dim(1536), bias=False),
up_proj=Linear(in_dim=Dim(576), out_dim=Dim(1536), bias=False),
down_proj=Linear(in_dim=Dim(1536), out_dim=Dim(576), bias=False)
)
['gate_proj.weight', 'up_proj.weight', 'down_proj.weight']
Calling the module runs it eagerly, one operation at a time. A single timing
is noisy, so median_ms calls the function 20 times, after one untimed call
that compiles its kernels, and reports the median:
def median_ms(fn, x: Tensor, repeats: int = 20) -> float:
fn(x).to_numpy() # warm up: the first call compiles its kernels
times = []
for _ in range(repeats):
start = time.perf_counter()
fn(x).to_numpy()
times.append((time.perf_counter() - start) * 1000)
return float(np.median(times))
inputs = Tensor(np.random.rand(1, 64, 576).astype(np.float32), device=DEVICE)
eager_out = mlp(inputs).to_numpy()
print(f"eager: {median_ms(mlp, inputs):.1f} ms")
eager: 49.0 ms
compile traces forward once into a whole-model graph and optimizes it:
it fuses operations, plans memory, and removes the per-operation overhead.
TensorType declares the input. Named dimensions such as "batch" and
"seq_len" may take any size at run time.
start = time.perf_counter()
compiled_mlp = mlp.compile(
TensorType(DType.float32, ("batch", "seq_len", 576), device=DEVICE)
)
print(f"compiled in {time.perf_counter() - start:.1f} s")
compiled_out = compiled_mlp(inputs).to_numpy()
print(f"compiled: {median_ms(compiled_mlp, inputs):.1f} ms")
relative = np.abs(compiled_out - eager_out).max() / np.abs(eager_out).max()
print(f"max difference, relative to the largest output: {relative:.1e}")
assert relative < 1e-5
compiled in 9.1 s
compiled: 0.6 ms
max difference, relative to the largest output: 1.2e-06
How much faster depends on the input size. Each eager operation pays a fixed cost to dispatch, which dominates small inputs; the compiled graph pays it once. Time both over a range of sequence lengths:
lengths = [1, 16, 64, 256, 1024]
timings = {"eager": [], "compiled": []}
for length in lengths:
x_len = Tensor(
np.random.rand(1, length, 576).astype(np.float32), device=DEVICE
)
timings["eager"].append(median_ms(mlp, x_len))
timings["compiled"].append(median_ms(compiled_mlp, x_len))
print(
f"{length:5} tokens: eager {timings['eager'][-1]:6.1f} ms, "
f"compiled {timings['compiled'][-1]:6.1f} ms"
)
1 tokens: eager 50.9 ms, compiled 0.7 ms
16 tokens: eager 31.0 ms, compiled 1.0 ms
64 tokens: eager 44.6 ms, compiled 1.8 ms
256 tokens: eager 54.5 ms, compiled 4.1 ms
1024 tokens: eager 59.0 ms, compiled 9.7 ms
Show the code 20 lines
fig = go.Figure(
[
go.Scatter(
x=lengths,
y=ms,
name=name,
mode="lines+markers",
line_color=color,
hovertemplate=name + ", %{x} tokens: %{y:.2f} ms<extra></extra>",
)
for (name, ms), color in zip(timings.items(), [viz.ORANGE, viz.BLUE])
]
)
fig.update_layout(
title="The MLP block, eager and compiled (median of 20 calls)",
height=420,
xaxis=dict(title="sequence length (tokens)", type="log", dtick=1),
yaxis=dict(title="ms (log scale)", type="log", dtick=1),
)
viz.show(fig)
The compiled MLP gives the same numbers up to float32 rounding (the compiled
graph may order additions differently) and runs many times faster. In the run
shown here, on an Apple-silicon Mac with a Metal GPU, an eager call took about
20 to 50 ms at every length, and a compiled one took about 1 ms up to 64 tokens
and a few milliseconds at the longest lengths. The rest of the book works the same way:
write and debug a model eagerly, printing anything you like, then compile it to
run it for real. MAX’s docs cover
eager execution and
tensor realization in more
detail.
The handbook’s CPU vs GPU vs TPU and GPU architecture fundamentals pages explain why matrix multiplications dominate the cost and why GPUs speed them up.
Exercises
- Create a
[4, 4]tensor, applyF.softmaxalong each axis, and check with NumPy that the rows (or columns) sum to 1. - Time the eager and compiled MLP with
seq_len1, 64, and 512. Where is the speedup largest? Why might small inputs benefit most? - Print a tensor created inside
F.lazy()before and after calling.to_numpy()on it.
Next
Section 1.3 loads SmolLM2’s real weights and turns token IDs into vectors.
1.3 Tokens, embeddings, and weights
From this section on, you rebuild SmolLM2-135M-Instruct piece by piece. A model on Hugging Face is two things: a configuration that gives its sizes, and a checkpoint that holds its trained weights. This section loads both, then builds the model’s first layer, the token embedding.
As in Section 1.2, code
cells that start with # export: go into the minillm package.
import json
import logging
from pathlib import Path
import numpy as np
import plotly.graph_objects as go
from max.driver import CPU
from max.dtype import DType
from max.nn import Embedding
from max.pipelines import AutoTokenizer
from max.tensor import Tensor, default_device, default_dtype
from minillm import viz
from minillm.device import pick_device
logging.getLogger("max").setLevel(logging.ERROR)
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
The configuration
config.json lists the architecture and its sizes. These numbers decide
the shape of every weight:
from huggingface_hub import hf_hub_download
MODEL_ID = "HuggingFaceTB/SmolLM2-135M-Instruct"
raw = json.loads(Path(hf_hub_download(MODEL_ID, "config.json")).read_text())
{
k: raw[k]
for k in [
"architectures",
"vocab_size",
"hidden_size",
"intermediate_size",
"num_hidden_layers",
"num_attention_heads",
"num_key_value_heads",
"rope_theta",
"rms_norm_eps",
"tie_word_embeddings",
]
}
{'architectures': ['LlamaForCausalLM'],
'vocab_size': 49152,
'hidden_size': 576,
'intermediate_size': 1536,
'num_hidden_layers': 30,
'num_attention_heads': 9,
'num_key_value_heads': 3,
'rope_theta': 100000,
'rms_norm_eps': 1e-05,
'tie_word_embeddings': True}
Read it as: 30 identical layers (num_hidden_layers), each working on vectors
of 576 numbers (hidden_size). Attention uses 9 query heads but only 3
key/value heads, so each head has 576 / 9 = 64 dimensions;
Section 1.5 explains why the
counts differ. LlamaForCausalLM is the architecture the code must implement.
A small dataclass keeps these fields typed. The field names match Hugging
Face’s, so you can compare against any Llama-family config.json.
Show LlamaConfig 44 lines
from __future__ import annotations
import json
from dataclasses import dataclass, fields
from pathlib import Path
from huggingface_hub import hf_hub_download
MODEL_ID = "HuggingFaceTB/SmolLM2-135M-Instruct"
@dataclass(frozen=True)
class LlamaConfig:
"""The sizes of a Llama-family model, as in its ``config.json``."""
vocab_size: int = 49152
hidden_size: int = 576
intermediate_size: int = 1536
num_hidden_layers: int = 30
num_attention_heads: int = 9
num_key_value_heads: int = 3
rope_theta: float = 100000.0
rms_norm_eps: float = 1e-5
max_position_embeddings: int = 8192
tie_word_embeddings: bool = True
eos_token_id: int = 2
head_dim: int = 0 # 0 means hidden_size // num_attention_heads
def __post_init__(self) -> None:
# Newer configs, such as Qwen3's, give head_dim explicitly.
if not self.head_dim:
object.__setattr__(
self, "head_dim", self.hidden_size // self.num_attention_heads
)
@classmethod
def from_pretrained(cls, model_id: str = MODEL_ID) -> LlamaConfig:
"""Reads the config of a Hugging Face model."""
raw = json.loads(
Path(hf_hub_download(model_id, "config.json")).read_text()
)
return cls(
**{f.name: raw[f.name] for f in fields(cls) if f.name in raw}
)
Saved to minillm/config.py.
config = LlamaConfig.from_pretrained()
print(config)
print("head_dim:", config.head_dim)
LlamaConfig(vocab_size=49152, hidden_size=576, intermediate_size=1536, num_hidden_layers=30, num_attention_heads=9, num_key_value_heads=3, rope_theta=100000, rms_norm_eps=1e-05, max_position_embeddings=8192, tie_word_embeddings=True, eos_token_id=2, head_dim=64)
head_dim: 64
The checkpoint
The weights live in model.safetensors. The format is simple enough to read
by hand: 8 bytes giving the length of a JSON header, the header itself, then
the raw bytes of every tensor. The header maps each tensor’s name to its
dtype, shape, and byte range:
path = hf_hub_download(MODEL_ID, "model.safetensors")
with open(path, "rb") as f:
header_len = int.from_bytes(f.read(8), "little")
header = json.loads(f.read(header_len))
header.pop("__metadata__", None)
print(len(header), "tensors")
first = list(header)[:8]
width = max(len(name) for name in first)
for name in first:
print(f"{name:{width}} {header[name]}")
272 tensors
model.embed_tokens.weight {'dtype': 'BF16', 'shape': [49152, 576], 'data_offsets': [0, 56623104]}
model.layers.0.input_layernorm.weight {'dtype': 'BF16', 'shape': [576], 'data_offsets': [56623104, 56624256]}
model.layers.0.mlp.down_proj.weight {'dtype': 'BF16', 'shape': [576, 1536], 'data_offsets': [56624256, 58393728]}
model.layers.0.mlp.gate_proj.weight {'dtype': 'BF16', 'shape': [1536, 576], 'data_offsets': [58393728, 60163200]}
model.layers.0.mlp.up_proj.weight {'dtype': 'BF16', 'shape': [1536, 576], 'data_offsets': [60163200, 61932672]}
model.layers.0.post_attention_layernorm.weight {'dtype': 'BF16', 'shape': [576], 'data_offsets': [61932672, 61933824]}
model.layers.0.self_attn.k_proj.weight {'dtype': 'BF16', 'shape': [192, 576], 'data_offsets': [61933824, 62155008]}
model.layers.0.self_attn.o_proj.weight {'dtype': 'BF16', 'shape': [576, 576], 'data_offsets': [62155008, 62818560]}
The names follow the module tree of Hugging Face’s LlamaForCausalLM: an
embedding, then model.layers.0 through model.layers.29, each with
attention projections (q_proj, k_proj, v_proj, o_proj), an MLP
(gate_proj, up_proj, down_proj), and two norms. Count the parameters:
n_params = sum(int(np.prod(t["shape"])) for t in header.values())
print(f"{n_params / 1e6:.1f}M parameters")
print(
f"bfloat16: {n_params * 2 / 2**20:.0f} MiB, float32: {n_params * 4 / 2**20:.0f} MiB"
)
print("lm_head.weight in checkpoint:", "lm_head.weight" in header)
134.5M parameters
bfloat16: 257 MiB, float32: 513 MiB
lm_head.weight in checkpoint: False
Where do the parameters live? Click a box to zoom in, and click the bar at the top to zoom back out:
Show part, layer_number, order 71 lines
def part(name: str) -> str:
if "embed_tokens" in name:
return "embedding"
if "self_attn" in name:
return "attention"
if "mlp" in name:
return "MLP"
return "norms"
color_of = {
"embedding": viz.ORANGE,
"attention": viz.BLUE,
"MLP": viz.AQUA,
"norms": viz.VIOLET,
}
# id -> [parent, label, total, color]
nodes = {"SmolLM2-135M": ["", "SmolLM2-135M", 0, viz.NEUTRAL]}
def layer_number(name: str) -> int:
return int(name.split(".")[2]) if ".layers." in name else -1
def order(name: str) -> int:
"""Sorts the embedding first, then the layers in order, then the norm."""
if "embed_tokens" in name:
return -1
return layer_number(name) if ".layers." in name else 99
for name in sorted(header, key=order):
count = int(np.prod(header[name]["shape"]))
kind = part(name)
layer = layer_number(name)
path = (
["SmolLM2-135M"]
+ ([f"layer {layer}"] if layer >= 0 else [])
+ [kind, name]
)
for depth in range(1, len(path)):
node = "/".join(path[: depth + 1])
label = name.split(".")[-2] if depth == len(path) - 1 else path[depth]
color = color_of[kind] if path[depth] in (kind, name) else viz.NEUTRAL
nodes.setdefault(node, ["/".join(path[:depth]), label, 0, color])
nodes[node][2] += count
nodes["SmolLM2-135M"][2] += count
fig = go.Figure(
go.Treemap(
ids=list(nodes),
parents=[n[0] for n in nodes.values()],
labels=[n[1] for n in nodes.values()],
values=[n[2] for n in nodes.values()],
branchvalues="total",
sort=False,
maxdepth=3,
marker=dict(
colors=[n[3] for n in nodes.values()],
line=dict(color=viz.SURFACE, width=2),
),
hovertemplate="%{label}<br>%{value:,} parameters, %{percentRoot:.1%} of the model<extra></extra>",
texttemplate="%{label}<br>%{value:.3s}",
)
)
fig.update_layout(
title="SmolLM2-135M's parameters",
height=520,
margin=dict(t=56, l=8, r=8, b=8),
)
viz.show(fig)
- Memory is parameters times bytes per parameter. The checkpoint stores
bfloat16(2 bytes); this book converts tofloat32(4 bytes), so its results can be compared exactly with reference implementations. The handbook’s GPU memory calculation applies the same arithmetic to large models, plus the KV cache you build in Section 1.8. - There is no
lm_head.weight. Withtie_word_embeddings, the output layer reuses the embedding matrix; Section 1.6 uses that.
The loader below maps the file into memory and returns NumPy arrays. NumPy
has no bfloat16 type, but the conversion is exact and cheap: a bfloat16
value is the upper 16 bits of a float32, so shifting the bits left by 16
gives the same number as a float32. MAX modules accept NumPy arrays
directly, so this is all the loading code the model needs.
Show read_safetensors, load_checkpoint 44 lines
from __future__ import annotations
import json
import numpy as np
from huggingface_hub import hf_hub_download
from minillm.config import MODEL_ID
_NUMPY_DTYPES = {
"F32": np.float32,
"F16": np.float16,
"I64": np.int64,
"I32": np.int32,
}
def read_safetensors(path: str) -> dict[str, np.ndarray]:
"""Reads a safetensors file, converting floating-point tensors to float32."""
with open(path, "rb") as f:
header_len = int.from_bytes(f.read(8), "little")
header = json.loads(f.read(header_len))
data = np.memmap(path, dtype=np.uint8, mode="r", offset=8 + header_len)
tensors = {}
for name, info in header.items():
if name == "__metadata__":
continue
start, end = info["data_offsets"]
raw = data[start:end]
if info["dtype"] == "BF16":
array = (raw.view(np.uint16).astype(np.uint32) << 16).view(
np.float32
)
else:
array = raw.view(_NUMPY_DTYPES[info["dtype"]])
if array.dtype == np.float16:
array = array.astype(np.float32)
tensors[name] = array.reshape(info["shape"])
return tensors
def load_checkpoint(model_id: str = MODEL_ID) -> dict[str, np.ndarray]:
"""Downloads a model's safetensors checkpoint and returns its tensors."""
return read_safetensors(hf_hub_download(model_id, "model.safetensors"))
Saved to minillm/weights.py.
weights = load_checkpoint()
w = weights["model.embed_tokens.weight"]
print(w.shape, w.dtype)
(49152, 576) float32
Token embeddings
The model’s first layer turns each token ID into a vector of 576 numbers.
It is a lookup table: row i of a [49152, 576] matrix is the vector for
token i. max.nn.Embedding implements it. load_state_dict replaces its
random weight with the checkpoint’s:
with default_device(DEVICE), default_dtype(DType.float32):
embed = Embedding(config.vocab_size, dim=config.hidden_size)
embed.load_state_dict({"weight": weights["model.embed_tokens.weight"]})
tokenizer = AutoTokenizer(MODEL_ID)
ids = tokenizer([" Paris France banana"])["input_ids"][0]
vectors = embed(Tensor(np.array([ids]), device=DEVICE))
print(ids, [tokenizer.decode([i]) for i in ids])
print(vectors.shape)
[7042, 4649, 21285] [' Paris', ' France', ' banana']
[Dim(1), Dim(3), Dim(576)]
Check it: each output vector must equal its row of the weight matrix.
table = embed.weight.to_numpy()
v = vectors.to_numpy()[0]
assert np.array_equal(v, table[ids])
print("embedding rows match the checkpoint")
embedding rows match the checkpoint
Training places related tokens near each other. Cosine similarity, the cosine of the angle between two vectors, shows it even before any layer runs:
\[\cos\theta = \frac{a \cdot b}{\lVert a \rVert \, \lVert b \rVert}\]
It is 1 for vectors pointing the same way, 0 for unrelated (orthogonal) ones, and \(-1\) for opposite ones, whatever their lengths.
def cosine(a: np.ndarray, b: np.ndarray) -> float:
return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
print(f"Paris ~ France: {cosine(v[0], v[1]):.2f}")
print(f"Paris ~ banana: {cosine(v[0], v[2]):.2f}")
print(f"France ~ banana: {cosine(v[1], v[2]):.2f}")
Paris ~ France: 0.59
Paris ~ banana: 0.36
France ~ banana: 0.37
Three numbers don’t show much. Project the embeddings of a few dozen words from three groups onto the two directions along which they vary most (principal component analysis, done here with NumPy’s SVD). Hover over a point for its word:
groups = {
"countries": "France Italy Germany Spain Japan China Brazil Canada Egypt India",
"fruits": "apple banana orange grape mango lemon cherry peach pear melon",
"numbers": "one two three four five six seven eight nine ten",
}
words, group_of, rows = [], [], []
for group, members in groups.items():
for word in members.split():
token = tokenizer([" " + word])["input_ids"][0]
if len(token) == 1: # keep words that are a single token
words.append(word)
group_of.append(group)
rows.append(table[token[0]])
points = np.array(rows) - np.mean(rows, axis=0)
_, _, axes = np.linalg.svd(points, full_matrices=False)
xy = points @ axes[:2].T
Show the code 30 lines
fig = go.Figure()
for (group, color), symbol in zip(
zip(groups, viz.SERIES), ["circle", "square", "diamond"]
):
index = [i for i, g in enumerate(group_of) if g == group]
fig.add_trace(
go.Scatter(
x=xy[index, 0],
y=xy[index, 1],
mode="markers+text",
name=group,
text=[words[i] for i in index],
textposition="top center",
textfont=dict(size=11),
marker=dict(
color=color,
symbol=symbol,
size=10,
line=dict(color=viz.SURFACE, width=2),
),
hovertemplate="%{text}<extra>" + group + "</extra>",
)
)
fig.update_layout(
title="Word embeddings projected to two dimensions",
height=520,
xaxis=dict(title="first principal component", showticklabels=False),
yaxis=dict(title="second principal component", showticklabels=False),
)
viz.show(fig)
The groups separate without the model ever being told what a country or a fruit is: words used in similar contexts during training end up with similar vectors.
The layers refine these vectors using context: what Paris means depends
on the tokens around it.
Exercises
- Find the 5 tokens whose embeddings are most similar to
" Paris". (Hint: normalize the table’s rows, then take one matrix-vector product.) - The treemap counts the embedding once because the output layer reuses
it. How many parameters, and how many MiB in
float32, would SmolLM2 need with a separatelm_head?
Next
Section 1.4 builds the layers that run inside each of the 30 blocks: RMSNorm, rotary position embeddings, and the SwiGLU MLP.
1.4 Building blocks
Each of SmolLM2’s 30 layers has the same structure:
x = x + attention(rms_norm(x)) # mix information across positions
x = x + mlp(rms_norm(x)) # transform each position on its own
This section builds the three pieces besides attention: RMSNorm, the SwiGLU
MLP, and rotary position embeddings (RoPE), which attention uses to know
where each token is. Each piece is a few lines with max.nn and
max.functional, and each is checked against a NumPy version of its
formula, using the real weights of layer 0.
Show the setup 20 lines
import logging
import numpy as np
import plotly.graph_objects as go
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.nn import Linear, Module, RMSNorm
from max.tensor import Tensor, default_device, default_dtype
from minillm import viz
from minillm.device import pick_device
from minillm.config import LlamaConfig
from minillm.weights import load_checkpoint
logging.getLogger("max").setLevel(logging.ERROR)
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
config = LlamaConfig.from_pretrained()
weights = load_checkpoint()
A layer’s weights are the checkpoint entries under its name prefix. This
helper selects them and strips the prefix, so a module can load them with
load_state_dict:
def select(
weights: dict[str, np.ndarray], prefix: str
) -> dict[str, np.ndarray]:
"""Returns the entries under ``prefix``, with the prefix removed."""
return {
name.removeprefix(prefix): value
for name, value in weights.items()
if name.startswith(prefix)
}
Saved to minillm/weights.py.
list(select(weights, "model.layers.0.mlp."))
['down_proj.weight', 'gate_proj.weight', 'up_proj.weight']
A random input stands in for the hidden states that reach layer 0. to_np
reads a tensor back, and compare reports the largest difference from the
NumPy result, absolute and relative to the largest value:
rng = np.random.default_rng(0)
x_np = rng.standard_normal((1, 6, config.hidden_size)).astype(np.float32)
x = Tensor(x_np, device=DEVICE)
def to_np(t: Tensor) -> np.ndarray:
return t.to_numpy()
def compare(actual: np.ndarray, expected: np.ndarray) -> None:
diff = np.abs(actual - expected).max()
relative = diff / np.abs(expected).max()
print(f"max difference {diff:.1e} (relative {relative:.1e})")
assert relative < 1e-4, "doesn't match the NumPy reference"
RMSNorm
Deep networks train better when each layer sees inputs of a stable scale. RMSNorm divides each vector by its root mean square, then multiplies by a learned weight per dimension:
\[\mathrm{RMSNorm}(x) = \frac{x}{\sqrt{\mathrm{mean}(x^2) + \epsilon}} \cdot w\]
Unlike LayerNorm, it doesn’t subtract the mean or add a bias, which
saves work and in practice trains as well. max.nn.RMSNorm implements it:
with default_device(DEVICE), default_dtype(DType.float32):
norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
norm.load_state_dict(select(weights, "model.layers.0.input_layernorm."))
w = to_np(norm.weight)
expected = (
x_np / np.sqrt((x_np**2).mean(-1, keepdims=True) + config.rms_norm_eps) * w
)
compare(to_np(norm(x)), expected)
max difference 3.0e-08 (relative 7.7e-08)
The learned weight is far from the all-ones it starts as. In layer 0, most dimensions are scaled close to zero, a few stand out several times larger, and nearly half are negative, so the norm also flips their sign. Hover over a bar to read its value:
Show the code 16 lines
fig = go.Figure(
go.Bar(
x=np.arange(config.hidden_size),
y=w,
marker_color=viz.BLUE,
hovertemplate="dimension %{x}: %{y:.3f}<extra></extra>",
)
)
fig.update_layout(
title="Layer 0 input RMSNorm weight, per dimension",
height=340,
bargap=0,
xaxis_title="dimension",
yaxis_title="weight",
)
viz.show(fig)
The SwiGLU MLP
The MLP transforms each position independently. SmolLM2 uses a gated variant: one projection, passed through SiLU, decides how much of a second projection to let through, and a third projection maps back to the hidden size:
\[\mathrm{MLP}(x) = W_{\text{down}}\,\big(\mathrm{SiLU}(W_{\text{gate}}\,x) \odot W_{\text{up}}\,x\big)\]
where \(\mathrm{SiLU}(z) = z \cdot \sigma(z)\). The attribute names match the
checkpoint (gate_proj, up_proj, down_proj), so load_state_dict needs
no renaming.
Show the code 23 lines
z = np.linspace(-6, 6, 241)
fig = go.Figure(
[
go.Scatter(
x=z,
y=np.maximum(z, 0),
name="ReLU",
line=dict(color=viz.ORANGE, dash="dot"),
),
go.Scatter(
x=z, y=z / (1 + np.exp(-z)), name="SiLU", line_color=viz.BLUE
),
]
)
fig.update_traces(hovertemplate="%{x:.2f} -> %{y:.3f}")
fig.update_layout(
title="SiLU is a smooth ReLU that lets small negative values through",
height=360,
xaxis_title="z",
yaxis_title="activation",
hovermode="x unified",
)
viz.show(fig)
from __future__ import annotations
from max import functional as F
from max.driver import Device
from max.dtype import DType
from max.graph import Dim
from max.nn import Linear, Module
from max.tensor import Tensor
from max.tensor.sharding import DeviceMesh
from minillm.config import LlamaConfig
class MLP(Module):
"""The SwiGLU feed-forward network of a Llama layer."""
def __init__(self, config: LlamaConfig) -> None:
hidden, inner = config.hidden_size, config.intermediate_size
self.gate_proj = Linear(hidden, inner, bias=False)
self.up_proj = Linear(hidden, inner, bias=False)
self.down_proj = Linear(inner, hidden, bias=False)
def forward(self, x: Tensor) -> Tensor:
return self.down_proj(F.silu(self.gate_proj(x)) * self.up_proj(x))
Saved to minillm/layers.py.
with F.lazy(), default_device(DEVICE), default_dtype(DType.float32):
mlp = MLP(config)
mlp.load_state_dict(select(weights, "model.layers.0.mlp."))
gate, up, down = (
to_np(getattr(mlp, n).weight) for n in ("gate_proj", "up_proj", "down_proj")
)
g = x_np @ gate.T
expected = (g / (1 + np.exp(-g)) * (x_np @ up.T)) @ down.T
compare(to_np(mlp(x)), expected)
max difference 1.0e-03 (relative 8.2e-07)
Linear stores its weight as [out_features, in_features], the same layout
as PyTorch, so the NumPy version multiplies by the transpose. The
differences, about one part in a million, come from summing thousands of
float32 products in a different order than NumPy does.
Rotary position embeddings
Attention compares tokens with dot products, which by themselves ignore order: “dog bites man” and “man bites dog” contain the same tokens. The model needs to know positions.
RoPE encodes position by rotating query and key vectors. It splits each
64-dimension head vector into 32 pairs and rotates pair i at position p
by the angle \(p \cdot \theta_i\), where \(\theta_i = \text{base}^{-2i/d}\).
Early pairs rotate quickly and late pairs slowly, like the hands of a clock.
SmolLM2’s base is rope_theta = 100000.
Following the Hugging Face layout, the two halves of the vector form the pairs: element i pairs with element i + 32. Rotating every pair at once is then two multiplications:
\[\mathrm{rope}(x) = x \cdot \cos + \mathrm{rotate\_half}(x) \cdot \sin, \qquad \mathrm{rotate\_half}([x_1, x_2]) = [-x_2, x_1]\]
start offsets the positions;
Section 1.8 needs it for the KV
cache. The tables name their dtype: a Tensor built without one takes the
device’s default, which is bfloat16 on a GPU, and bfloat16 can’t even
represent every integer position above 256 exactly. Lengths are typed
int | Dim because a compiled model sees symbolic dimensions such as seq_len
instead of numbers;
Section 1.2
shows where they come from.
Show rope_cos_sin, rotate_half, apply_rope 33 lines
def rope_cos_sin(
head_dim: int,
theta: float,
start: int | Dim | Tensor,
length: int | Dim,
device: Device | DeviceMesh,
) -> tuple[Tensor, Tensor]:
"""Returns the RoPE cos and sin tables, ``[length, head_dim]``, for
positions ``start`` to ``start + length - 1``."""
inv_freq = Tensor(
[theta ** (-i / head_dim) for i in range(0, head_dim, 2)],
dtype=DType.float32,
device=device,
)
positions = (
Tensor.arange(length, dtype=DType.float32, device=device) + start
)
angles = F.outer(positions, inv_freq)
angles = F.concat([angles, angles], axis=-1)
return F.cos(angles), F.sin(angles)
def rotate_half(x: Tensor) -> Tensor:
half = x.shape[-1] // 2
return F.concat([-x[..., half:], x[..., :half]], axis=-1)
def apply_rope(x: Tensor, cos: Tensor, sin: Tensor) -> Tensor:
"""Rotates ``x``, shaped ``[batch, seq, heads, head_dim]``."""
seq, dim = cos.shape
cos = cos.reshape([1, seq, 1, dim])
sin = sin.reshape([1, seq, 1, dim])
return x * cos + rotate_half(x) * sin
Saved to minillm/layers.py.
Each line is one pair’s rotation at a position; move the slider to advance the position. Pair 0 turns a full radian per token, while pair 31 has barely moved after 24 tokens:
positions = 25
cos_t, sin_t = rope_cos_sin(
config.head_dim, config.rope_theta, 0, positions, DEVICE
)
# One angle per pair and position: [position, head_dim].
angles = np.arctan2(to_np(sin_t), to_np(cos_t))
Show the code 33 lines
pairs = [0, 2, 8, 31]
steps = []
for pos in range(positions):
group = []
for pair, color in zip(pairs, viz.SERIES):
degrees = float(np.degrees(angles[pos, pair]))
group.append(
go.Scatterpolar(
r=[0, 1],
theta=[0, degrees],
mode="lines+markers",
name=f"pair {pair}",
line=dict(color=color, width=3),
marker=dict(size=[0, 10], color=color),
hovertemplate=f"pair {pair} at position {pos}: {degrees:.0f} degrees<extra></extra>",
)
)
steps.append(group)
fig = go.Figure(
layout=dict(
title="RoPE rotates each pair at its own speed",
height=520,
polar=dict(
radialaxis=dict(visible=False, range=[0, 1.05]),
angularaxis=dict(gridcolor=viz.GRID, linecolor=viz.GRID),
),
)
)
viz.show(
viz.with_slider(
fig, steps, [str(p) for p in range(positions)], "position ", active=3
)
)
The whole cos table shows every pair at once. Each row is a position and
each column a dimension. The left columns, the fast pairs, cycle every few
positions; toward the middle the stripes widen, and the last pairs stay near
1 over all 256 positions. The right half repeats the left half, because
element i and element i + 32 form one pair and share its angle:
table_cos, _ = rope_cos_sin(config.head_dim, config.rope_theta, 0, 256, DEVICE)
table_cos = to_np(table_cos)
half = config.head_dim // 2
assert np.array_equal(table_cos[:, :half], table_cos[:, half:])
Show the code 18 lines
fig = go.Figure(
go.Heatmap(
z=table_cos,
zmin=-1,
zmax=1,
colorscale=viz.DIVERGING,
colorbar=dict(title="cos", thickness=12),
hovertemplate="position %{y}, dimension %{x}: %{z:.2f}<extra></extra>",
)
)
fig.add_vline(x=half - 0.5, line=dict(color=viz.INK_SECONDARY, dash="dot"))
fig.update_layout(
title="The RoPE cos table: positions by dimensions",
height=520,
xaxis=dict(title="dimension", showgrid=False),
yaxis=dict(title="position", autorange="reversed", showgrid=False),
)
viz.show(fig)
RoPE relies on two properties, and both are easy to check. First, a rotation doesn’t change a vector’s length:
q = Tensor(
rng.standard_normal((1, 8, 1, config.head_dim)).astype(np.float32),
device=DEVICE,
)
cos, sin = rope_cos_sin(config.head_dim, config.rope_theta, 0, 8, DEVICE)
rotated = to_np(apply_rope(q, cos, sin))
print(np.linalg.norm(to_np(q), axis=-1).ravel()[:4])
print(np.linalg.norm(rotated, axis=-1).ravel()[:4])
assert np.allclose(
np.linalg.norm(to_np(q), axis=-1),
np.linalg.norm(rotated, axis=-1),
rtol=1e-5,
)
[6.897215 6.9866242 7.8358827 6.884518 ]
[6.897215 6.9866247 7.8358827 6.884518 ]
Second, the dot product of a rotated query and key depends only on how far apart they are. Here a query at position 7 and a key at position 3 score the same as at positions 107 and 103:
def score(q_pos: int, k_pos: int) -> float:
qv = Tensor(np.ones((1, 1, 1, config.head_dim), np.float32), device=DEVICE)
kv = Tensor(
np.linspace(-1, 1, config.head_dim, dtype=np.float32).reshape(
1, 1, 1, -1
),
device=DEVICE,
)
q_rot = apply_rope(
qv, *rope_cos_sin(config.head_dim, config.rope_theta, q_pos, 1, DEVICE)
)
k_rot = apply_rope(
kv, *rope_cos_sin(config.head_dim, config.rope_theta, k_pos, 1, DEVICE)
)
return float((to_np(q_rot) * to_np(k_rot)).sum())
print(f"{score(7, 3):.4f} {score(107, 103):.4f} {score(7, 5):.4f}")
assert (
abs(score(7, 3) - score(107, 103)) < 1e-3
and abs(score(7, 3) - score(7, 5)) > 1e-2
)
10.2645 10.2645 7.8206
The first two scores match. The third, for positions two apart instead of four, differs. The sweep below fixes the key at position 10 or at 110 and moves the query up to 48 positions after it. The two curves lie on top of each other:
q_vec = np.ones(config.head_dim, np.float32)
k_vec = np.linspace(-1, 1, config.head_dim, dtype=np.float32)
cos_t, sin_t = (
to_np(t)
for t in rope_cos_sin(config.head_dim, config.rope_theta, 0, 160, DEVICE)
)
def rope_np(v: np.ndarray, pos: int) -> np.ndarray:
half = len(v) // 2
return v * cos_t[pos] + np.concatenate([-v[half:], v[:half]]) * sin_t[pos]
distances = np.arange(49)
sweeps = {
k_pos: [
rope_np(q_vec, k_pos + d) @ rope_np(k_vec, k_pos) for d in distances
]
for k_pos in (10, 110)
}
Show the code 22 lines
fig = go.Figure(
[
go.Scatter(
x=distances,
y=sweeps[k_pos],
name=f"key at position {k_pos}",
line=dict(color=color, dash=dash, width=width),
hovertemplate="distance %{x}: %{y:.3f}<extra></extra>",
)
for k_pos, color, dash, width in [
(10, viz.BLUE, "solid", 5),
(110, viz.ORANGE, "dash", 2),
]
]
)
fig.update_layout(
title="Query-key score after RoPE depends only on the distance",
height=380,
xaxis_title="query position minus key position",
yaxis_title="q · k",
)
viz.show(fig)
Because scores depend only on relative position, RoPE generalizes well, and most open models use it.
Exercises
- Replace
RMSNormwith a version you write fromF.rsqrt,.mean, and*, and check that it gives the same output. - Redraw the
costable withrope_theta=10000, the original RoPE base. Which pairs change, and why might a long-context model choose a larger base? - Run the RMSNorm check with
bfloat16instead offloat32(cast the input and the module’s weight). How large is the error, and why do real models compute norms infloat32?
Next
Section 1.5 builds attention, the part of the layer that mixes information between positions, and uses RoPE to do it.
1.5 Attention
Attention is how a position gathers information from earlier positions. In “The capital of France is”, predicting the next token requires the last position to look back at “capital” and “France”. This section builds causal grouped-query attention, the variant SmolLM2 and most current models use, first from basic operations and then with MAX’s flash attention kernel.
Show the setup 28 lines
import logging
import math
import time
import warnings
import numpy as np
import plotly.graph_objects as go
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.nn import Embedding, RMSNorm
from max.pipelines import AutoTokenizer
from max.tensor import Tensor, default_device, default_dtype
from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.layers import apply_rope, rope_cos_sin
from minillm.weights import load_checkpoint, select
logging.getLogger("max").setLevel(logging.ERROR)
# MAX runs some kernels, such as flash attention, through a compiled
# fallback when executing eagerly, and warns each time it does.
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
config = LlamaConfig.from_pretrained()
weights = load_checkpoint()
The computation
Each position projects its hidden vector into three vectors: a query for what the position is looking for, a key for what it offers, and a value for what it passes on.
A position’s output is a weighted average of the values of all positions it may see, weighted by how well its query matches their keys:
\[\mathrm{attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{Q K^\top}{\sqrt{d}} + M\right) V\]
Dividing by \(\sqrt{d}\), with \(d\) the head dimension, keeps the scores in a range where softmax still has a useful gradient. \(M\) is the causal mask.
The model runs several heads in parallel, each with its own projections, so different heads can track different relationships. SmolLM2 has 9 heads of 64 dimensions.
The causal mask
A language model predicts each token from the tokens before it, so position i must not see positions after i. The mask adds \(-\infty\) to those scores, and softmax turns them into zero weight.
The mask takes separate query and key lengths, so it also works when new queries attend to cached keys, as with the KV cache in Section 1.8. Here the lengths are equal.
Show causal_mask 30 lines
from __future__ import annotations
import math
from max import functional as F
from max.driver import Device
from max.dtype import DType
from max.graph import Dim
from max.nn import Linear, Module
from max.nn.kernels import flash_attention
from max.tensor import Tensor
from max.tensor.sharding import DeviceMesh
from minillm.config import LlamaConfig
from minillm.layers import apply_rope, rope_cos_sin
def causal_mask(
q_len: int | Dim, kv_len: int | Dim, device: Device | DeviceMesh
) -> Tensor:
"""Returns a ``[q_len, kv_len]`` additive mask for queries at the last
``q_len`` of ``kv_len`` positions: 0 where a query may attend, -inf
where it may not. The lengths may be ints or symbolic dimensions."""
blocked = F.broadcast_to(
Tensor(float("-inf"), dtype=DType.float32, device=device),
[q_len, kv_len],
)
return F.band_part(
blocked, num_lower=None, num_upper=kv_len - q_len, exclude=True
)
Saved to minillm/attention.py.
causal_mask(4, 4, DEVICE)
Tensor([ 0 -inf -inf -inf
0 0 -inf -inf
0 0 0 -inf
0 0 0 0], dtype=DType.float32, device=Device(type=gpu,id=0))
Grouped-query attention
SmolLM2 has 9 query heads but only 3 key/value heads: each key/value head is shared by a group of 3 query heads. This is grouped-query attention (GQA). It costs little quality, and it makes the keys and values the model must store per token 3 times smaller, which matters once you cache them (Section 1.8).
To compute it with plain matrix products, repeat each key/value head 3
times with broadcast_to:
def repeat_kv(x: Tensor, n_rep: int) -> Tensor:
"""Repeats each head of ``[batch, seq, kv_heads, dim]`` ``n_rep`` times."""
batch, seq, kv_heads, dim = x.shape
x = x.reshape([batch, seq, kv_heads, 1, dim])
x = F.broadcast_to(x, [batch, seq, kv_heads, n_rep, dim])
return x.reshape([batch, seq, kv_heads * n_rep, dim])
def naive_attention(q: Tensor, k: Tensor, v: Tensor, mask: Tensor) -> Tensor:
"""Computes causal attention with basic operations.
``q`` is ``[batch, seq, heads, dim]``; ``k`` and ``v`` may have fewer
heads, which are repeated to match.
"""
k = repeat_kv(k, q.shape[2] // k.shape[2])
v = repeat_kv(v, q.shape[2] // v.shape[2])
q, k, v = (t.permute([0, 2, 1, 3]) for t in (q, k, v)) # heads first
scores = (q @ k.transpose(-1, -2)) / math.sqrt(int(q.shape[-1])) + mask
return (F.softmax(scores) @ v).permute([0, 2, 1, 3])
Saved to minillm/attention.py.
Look at real attention
Run layer 0’s projections on a real sentence: embed the tokens, normalize, project to queries and keys, apply RoPE, and look at the attention weights.
tokenizer = AutoTokenizer(MODEL_ID)
ids = tokenizer(
["The capital of France is Paris, and the capital of Italy is"]
)["input_ids"][0]
tokens = [tokenizer.decode([i]) for i in ids]
seq = len(ids)
with default_device(DEVICE), default_dtype(DType.float32):
embed = Embedding(config.vocab_size, dim=config.hidden_size)
norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
embed.load_state_dict({"weight": weights["model.embed_tokens.weight"]})
norm.load_state_dict(select(weights, "model.layers.0.input_layernorm."))
h = norm(embed(Tensor(np.array([ids]), device=DEVICE)))
wq = Tensor(weights["model.layers.0.self_attn.q_proj.weight"], device=DEVICE)
wk = Tensor(weights["model.layers.0.self_attn.k_proj.weight"], device=DEVICE)
q = (h @ wq.T).reshape([1, seq, config.num_attention_heads, config.head_dim])
k = (h @ wk.T).reshape([1, seq, config.num_key_value_heads, config.head_dim])
cos, sin = rope_cos_sin(config.head_dim, config.rope_theta, 0, seq, DEVICE)
q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
mask = causal_mask(seq, seq, DEVICE)
k_full = repeat_kv(k, 3).permute([0, 2, 1, 3])
scores = (q.permute([0, 2, 1, 3]) @ k_full.transpose(-1, -2)) / math.sqrt(
config.head_dim
) + mask
probs = F.softmax(scores).to_numpy()[0] # [heads, seq, seq]
Show the code 47 lines
# Number the labels so that they stay unique when a token repeats.
labels = [f"{i}: {t!r}" for i, t in enumerate(tokens)]
pairs = [[[labels[i], labels[j]] for j in range(seq)] for i in range(seq)]
visible = np.tri(seq, dtype=bool) # masked cells are left blank, not drawn as 0
heads = [
[
go.Heatmap(
z=np.where(visible, probs[head], np.nan),
customdata=pairs,
zmin=0,
zmax=1,
coloraxis="coloraxis",
hovertemplate="query %{customdata[0]}<br>key %{customdata[1]}<br>weight %{z:.2f}<extra></extra>",
)
]
for head in range(config.num_attention_heads)
]
fig = go.Figure(
layout=dict(
title="Layer 0 attention weights: rows are queries, columns are the keys they attend to",
height=640,
coloraxis=dict(
colorscale=viz.SEQUENTIAL,
cmin=0,
cmax=1,
colorbar=dict(title="weight", thickness=12),
),
xaxis=dict(
tickvals=list(range(seq)),
ticktext=tokens,
tickangle=-45,
showgrid=False,
),
yaxis=dict(
tickvals=list(range(seq)),
ticktext=tokens,
autorange="reversed",
scaleanchor="x",
showgrid=False,
),
)
)
viz.show(
viz.with_slider(
fig, heads, [str(h) for h in range(config.num_attention_heads)], "head "
)
)
Each row is a query position; each column is a key it attends to. Hover over a cell to read its weight, and move the slider to change heads. The upper triangle is empty: that is the causal mask. The heads differ. Heads 7 and 8 put the most weight on the first token, which models use as a resting place when nothing else is relevant. Others, such as heads 2 and 5, put more weight on the current token.
The attention module
The module projects to queries, keys, and values, applies RoPE, attends, and
projects the heads back to the hidden size with o_proj. Splitting project
from forward lets Section 1.8
reuse it with a cache.
It creates the RoPE tables and the mask on x.mesh, the set of devices that
hold x. Here that is one device; in
Chapter 2 the same code runs on several
devices at once.
For the attention itself it uses flash_attention from
max.nn.kernels, which takes grouped heads directly. Flash
attention computes the same result as naive_attention but never builds the
per-head [seq, seq] score matrices: it processes keys in blocks and keeps
a running softmax. This module still passes a dense [seq, seq] mask, one for
the whole batch rather than one per head; production kernels take a causal flag
instead, so their memory grows with the sequence length rather than its square.
Show Attention 51 lines
class Attention(Module):
"""Causal grouped-query attention with rotary position embeddings."""
def __init__(self, config: LlamaConfig) -> None:
self.num_heads = config.num_attention_heads
self.num_kv_heads = config.num_key_value_heads
self.head_dim = config.head_dim
self.rope_theta = config.rope_theta
hidden = config.hidden_size
self.q_proj = Linear(hidden, self.num_heads * self.head_dim, bias=False)
self.k_proj = Linear(
hidden, self.num_kv_heads * self.head_dim, bias=False
)
self.v_proj = Linear(
hidden, self.num_kv_heads * self.head_dim, bias=False
)
self.o_proj = Linear(self.num_heads * self.head_dim, hidden, bias=False)
def project(
self, x: Tensor, start: int | Dim | Tensor
) -> tuple[Tensor, Tensor, Tensor]:
"""Returns RoPE-rotated queries and keys, and values, for ``x``,
whose first position is ``start``."""
batch, seq, _ = x.shape
q = self.q_proj(x).reshape([batch, seq, self.num_heads, self.head_dim])
k = self.k_proj(x).reshape(
[batch, seq, self.num_kv_heads, self.head_dim]
)
v = self.v_proj(x).reshape(
[batch, seq, self.num_kv_heads, self.head_dim]
)
cos, sin = rope_cos_sin(
self.head_dim, self.rope_theta, start, seq, x.mesh
)
return apply_rope(q, cos, sin), apply_rope(k, cos, sin), v
def attend(self, q: Tensor, k: Tensor, v: Tensor) -> Tensor:
"""Attends from ``q`` to ``k`` and ``v`` and projects the heads back."""
batch, q_len = q.shape[0], q.shape[1]
mask = causal_mask(q_len, k.shape[1], q.mesh)
mask = F.broadcast_to(mask, [batch, q_len, k.shape[1]])
out = flash_attention(q, k, v, 1 / math.sqrt(self.head_dim), mask)
# Attention never mixes heads, so the output is placed like the
# queries. Chapter 2 relies on this when heads span several devices.
out = out.rebind_mapping(q.mapping)
return self.o_proj(
out.reshape([batch, q_len, self.num_heads * self.head_dim])
)
def forward(self, x: Tensor) -> Tensor:
return self.attend(*self.project(x, 0))
Saved to minillm/attention.py.
Load layer 0’s attention and check the module against naive_attention
on the same normalized hidden states:
with F.lazy(), default_device(DEVICE), default_dtype(DType.float32):
attn = Attention(config)
attn.load_state_dict(select(weights, "model.layers.0.self_attn."))
q, k, v = attn.project(h, 0)
reference = attn.o_proj(
naive_attention(q, k, v, causal_mask(seq, seq, DEVICE)).reshape(
[1, seq, config.hidden_size]
)
)
diff = np.abs(attn(h).to_numpy() - reference.to_numpy()).max()
print(f"max difference {diff:.1e}")
assert diff < 1e-4
max difference 3.0e-07
Now compare their speed on a longer input, 1024 positions:
long = Tensor(
np.random.default_rng(0)
.standard_normal((1, 1024, config.hidden_size))
.astype(np.float32),
device=DEVICE,
)
q, k, v = attn.project(long, 0)
long_mask = causal_mask(1024, 1024, DEVICE)
def timed(fn) -> float:
fn().to_numpy() # compile the kernels first
start = time.perf_counter()
fn().to_numpy()
return (time.perf_counter() - start) * 1000
print(f"naive: {timed(lambda: naive_attention(q, k, v, long_mask)):.0f} ms")
print(f"flash: {timed(lambda: attn.attend(q, k, v)):.0f} ms (includes o_proj)")
naive: 93 ms
flash: 41 ms (includes o_proj)
In the run shown here, on an Apple-silicon Mac with a Metal GPU, flash attention
took about 55 ms including o_proj, against about 80 ms for the naive version.
Treat these numbers as an example: yours depend on your hardware and vary from
run to run. The handbook’s
FlashAttention
page explains the algorithm and why it matters most on GPUs, where moving data
between memory levels costs more than the arithmetic.
Exercises
- Find the head in the plot that puts the most weight on the previous token. Does the pattern hold on a different sentence?
- Change
repeat_kvto use 1 key/value head for all 9 query heads (multi-query attention). What would it save, and what might it cost? - Time both versions at 256, 1024, and 2048 positions. How does each grow with sequence length?
Next
Section 1.6 combines attention with the Section 1.4 building blocks into complete layers, stacks 30 of them, and checks the result against MAX’s own implementation of SmolLM2.
1.6 The full model
You now have every part of SmolLM2. This section assembles them into a
decoder layer, stacks 30 layers into the model, adds the output head, loads
all 272 weights, and compiles the result. Then it checks that your model’s
logits match MAX’s own implementation of SmolLM2, the one AutoModel
loads.
Show the setup 27 lines
import logging
import time
import warnings
import numpy as np
import plotly.graph_objects as go
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.graph import TensorType
from max.nn import Embedding, Linear, Module, ModuleList, RMSNorm
from max.pipelines import AutoModel, AutoTokenizer
from max.tensor import Tensor, default_device, default_dtype
from minillm import viz
from minillm.attention import Attention
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.layers import MLP
from minillm.weights import load_checkpoint, select
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
config = LlamaConfig.from_pretrained()
weights = load_checkpoint()
The decoder layer
A layer normalizes its input, runs attention, and adds the result back to the input; then it does the same with the MLP:
\[\begin{aligned} h &= x + \mathrm{Attention}(\mathrm{RMSNorm}(x)) \\ x’ &= h + \mathrm{MLP}(\mathrm{RMSNorm}(h)) \end{aligned}\]
Adding back (the residual connection) means each layer only refines the hidden state instead of replacing it, which helps a stack of 30 layers train well.
The attribute names match the checkpoint once more, so a layer loads
model.layers.N.* directly.
Show DecoderLayer 27 lines
from __future__ import annotations
from max import functional as F
from max.driver import Device
from max.dtype import DType
from max.nn import Embedding, Linear, Module, ModuleList, RMSNorm
from max.tensor import Tensor, default_device, default_dtype
from numpy.typing import NDArray
from minillm.attention import Attention
from minillm.config import LlamaConfig
from minillm.layers import MLP
class DecoderLayer(Module):
"""One transformer layer: attention and an MLP, each with a residual."""
def __init__(self, config: LlamaConfig) -> None:
eps = config.rms_norm_eps
self.input_layernorm = RMSNorm(config.hidden_size, eps=eps)
self.self_attn = Attention(config)
self.post_attention_layernorm = RMSNorm(config.hidden_size, eps=eps)
self.mlp = MLP(config)
def forward(self, x: Tensor) -> Tensor:
x = x + self.self_attn(self.input_layernorm(x))
return x + self.mlp(self.post_attention_layernorm(x))
Saved to minillm/model.py.
Run layer 0 on real token embeddings and compare the size of each token’s input with the size of the update the layer adds to it:
tokenizer = AutoTokenizer(MODEL_ID)
ids = tokenizer(["The capital of France is"])["input_ids"][0]
with F.lazy(), default_device(DEVICE), default_dtype(DType.float32):
layer = DecoderLayer(config)
layer.load_state_dict(select(weights, "model.layers.0."))
x = Tensor(weights["model.embed_tokens.weight"][ids][None], device=DEVICE)
y = layer(x)
x_np, y_np = x.to_numpy(), y.to_numpy()
print("input norm per token: ", np.linalg.norm(x_np, axis=-1).round(2))
print("update norm per token:", np.linalg.norm(y_np - x_np, axis=-1).round(2))
input norm per token: [[2. 2.54 2.91 2.28 2.6 ]]
update norm per token: [[47.39 51.99 31.03 68.12 35.3 ]]
The update is 10 to 30 times larger than the input, because embeddings start as small vectors. The residual sum still carries the original embedding forward. Later in this section, a plot follows this residual stream through all 30 layers.
The model
The model embeds the tokens, runs the 30 layers in order, and normalizes the
result. The causal LM wrapper adds lm_head, a Linear from 576 dimensions
to one logit per vocabulary entry. The module tree mirrors Hugging Face’s
LlamaForCausalLM, so every checkpoint name is also a parameter name.
load_model builds the model inside F.lazy(), so the random
initialization never runs, and then loads the weights. SmolLM2 ties
lm_head to the embedding table, so the loader uses the same array for
both.
Show LlamaModel, LlamaForCausalLM, load_model 44 lines
class LlamaModel(Module):
"""The embedding, the decoder layers, and the final norm."""
def __init__(self, config: LlamaConfig) -> None:
self.embed_tokens = Embedding(config.vocab_size, dim=config.hidden_size)
self.layers = ModuleList(
[DecoderLayer(config) for _ in range(config.num_hidden_layers)]
)
self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
def forward(self, input_ids: Tensor) -> Tensor:
x = self.embed_tokens(input_ids)
for layer in self.layers:
x = layer(x)
return self.norm(x)
class LlamaForCausalLM(Module):
"""A Llama model with its language-model head: token IDs in, logits out."""
def __init__(self, config: LlamaConfig) -> None:
self.model = LlamaModel(config)
self.lm_head = Linear(config.hidden_size, config.vocab_size, bias=False)
def forward(self, input_ids: Tensor) -> Tensor:
return self.lm_head(self.model(input_ids))
def load_model(
config: LlamaConfig,
weights: dict[str, NDArray],
device: Device,
model_class: type[Module] = LlamaForCausalLM,
) -> Module:
"""Builds ``model_class`` on ``device`` and loads ``weights`` into it."""
with F.lazy(), default_device(device), default_dtype(DType.float32):
model = model_class(config)
if config.tie_word_embeddings:
weights = {
**weights,
"lm_head.weight": weights["model.embed_tokens.weight"],
}
model.load_state_dict(weights)
return model
Saved to minillm/model.py.
Here is the whole model as a diagram, built from the config. The shaded block repeats 30 times, and the curved lines are the residual connections. Hover over a box for the shape of the tensor it produces:
Show the code 121 lines
H, V, I = config.hidden_size, config.vocab_size, config.intermediate_size
stages = [ # (label, kind, output shape)
("token IDs", "io", "[batch, seq]"),
(f"Embedding: {V:,} × {H}", "embedding", f"[batch, seq, {H}]"),
("RMSNorm", "norm", f"[batch, seq, {H}]"),
(
f"Attention: {config.num_attention_heads} query heads, {config.num_key_value_heads} key/value heads, RoPE",
"attention",
f"[batch, seq, {H}]",
),
("+", "add", f"[batch, seq, {H}]"),
("RMSNorm", "norm", f"[batch, seq, {H}]"),
(f"SwiGLU MLP: {H} → {I} → {H}", "mlp", f"[batch, seq, {H}]"),
("+", "add", f"[batch, seq, {H}]"),
("final RMSNorm", "norm", f"[batch, seq, {H}]"),
(
f"LM head: {H} → {V:,} (shares the embedding)",
"embedding",
f"[batch, seq, {V}]",
),
("logits", "io", f"[batch, seq, {V}]"),
]
colors = {
"io": viz.NEUTRAL,
"embedding": viz.ORANGE,
"norm": viz.VIOLET,
"attention": viz.BLUE,
"mlp": viz.AQUA,
"add": viz.SURFACE,
}
ys = [-i for i in range(len(stages))]
fig = go.Figure()
fig.add_shape(
type="rect",
x0=-2.6,
x1=2.6,
y0=ys[7] - 0.45,
y1=ys[2] + 0.75,
line=dict(color=viz.GRID),
fillcolor=viz.NEUTRAL,
layer="below",
)
fig.add_annotation(
x=2.5,
y=ys[2] + 0.55,
text=f"× {config.num_hidden_layers} layers",
showarrow=False,
xanchor="right",
font=dict(color=viz.INK_SECONDARY),
)
for (label, kind, _), y in zip(stages, ys):
width = 0.25 if kind == "add" else 2.1
fig.add_shape(
type="rect",
x0=-width,
x1=width,
y0=y - 0.3,
y1=y + 0.3,
fillcolor=colors[kind],
line=dict(
color=viz.INK_SECONDARY if kind in ("add", "io") else colors[kind]
),
layer="above" if kind == "add" else "below",
)
fig.add_annotation(
x=0,
y=y,
text=label,
showarrow=False,
font=dict(
color=viz.INK if kind in ("io", "add") else viz.SURFACE, size=13
),
)
for upper, lower in zip(ys, ys[1:]): # the main path
fig.add_annotation(
x=0,
y=lower + 0.3,
ax=0,
ay=upper - 0.3,
xref="x",
yref="y",
axref="x",
ayref="y",
showarrow=True,
arrowhead=2,
arrowcolor=viz.INK_SECONDARY,
text="",
)
# The residual connections skip the norm and the sublayer.
for start, add in [(1, 4), (4, 7)]:
y0, y1 = ys[start] - 0.5, ys[add]
fig.add_trace(
go.Scatter(
x=[0, -2.35, -2.35, -0.25],
y=[y0, y0, y1, y1],
mode="lines",
hoverinfo="skip",
line=dict(color=viz.INK_SECONDARY, dash="dot"),
showlegend=False,
)
)
fig.add_trace(
go.Scatter(
x=[0] * len(stages),
y=ys,
mode="markers",
marker=dict(size=40, opacity=0),
customdata=[[label, shape] for label, _, shape in stages],
showlegend=False,
hovertemplate="%{customdata[0]}<br>output: %{customdata[1]}<extra></extra>",
)
)
fig.update_layout(
title="SmolLM2-135M, from token IDs to logits",
height=640,
xaxis=dict(visible=False, range=[-2.8, 2.8]),
yaxis=dict(visible=False, range=[ys[-1] - 0.6, 0.6]),
margin=dict(l=8, r=8),
)
viz.show(fig)
model = load_model(config, weights, DEVICE)
n_params = sum(p.num_elements() for _, p in model.parameters)
print(
f"{len(list(model.parameters))} parameter tensors, {n_params / 1e6:.1f}M values"
)
273 parameter tensors, 162.8M values
Your model has 273 parameter tensors and 162.8M values. That is 28M more than
the checkpoint’s 134.5M because lm_head counts as its own 49152 × 576
parameter, loaded from the same array as the embedding table. load_state_dict
is strict: a missing or misnamed weight raises an error instead of silently
keeping a random value, so a successful load already proves the names line up.
Inside the forward pass
Before compiling, run the layers one at a time and keep each one’s output. The size of each token’s hidden state grows as layers add their updates:
x = model.model.embed_tokens(Tensor(np.array([ids]), device=DEVICE))
states = [x.to_numpy()[0]]
for layer in model.model.layers:
x = layer(x)
states.append(x.to_numpy()[0])
states = np.array(states) # [layer + 1, token, hidden]
Show the code 21 lines
labels = [repr(tokenizer.decode([i])) for i in ids]
fig = go.Figure(
[
go.Scatter(
x=np.arange(31),
y=np.linalg.norm(states[:, t], axis=-1),
name=label,
line_color=viz.SERIES[t],
mode="lines",
hovertemplate=label + " after layer %{x}: %{y:.1f}<extra></extra>",
)
for t, label in enumerate(labels)
]
)
fig.update_layout(
title="Hidden-state norm through the layers (0 is the embedding)",
height=420,
xaxis_title="layer",
yaxis=dict(title="norm (log scale)", type="log", dtick=1),
)
viz.show(fig)
The first token behaves differently. Its state jumps to about 24,000 at layer 12, over a hundred times the other tokens’ (about 120 to 165), and stays there until it drops in the last two layers. Models use the first position as a resting place for attention (Section 1.5), and this large state is part of how they do it.
Every layer writes into the same stream, so you can ask what the model would
predict if it stopped early: apply the final norm and lm_head to each
layer’s output. For the hidden state \(h_\ell\) after layer \(\ell\), the
logit lens reads
\[p_\ell = \mathrm{softmax}\big(E \, \mathrm{RMSNorm}(h_\ell)\big)\]
where \(E\) is the embedding table that lm_head shares. Apply it at every
position and after every layer:
final_norm = weights["model.norm.weight"]
table = weights["model.embed_tokens.weight"] # lm_head shares it
normed = (
states
/ np.sqrt((states**2).mean(-1, keepdims=True) + config.rms_norm_eps)
* final_norm
)
lens = normed @ table.T # [layer + 1, token, vocab]
lens_probs = np.exp(lens - lens.max(-1, keepdims=True))
lens_probs /= lens_probs.sum(-1, keepdims=True)
positions = np.arange(len(ids))
final_top = lens[-1].argmax(-1) # each position's prediction after layer 30
p_final = lens_probs[:, positions, final_top] # [layer + 1, token]
paris = tokenizer([" Paris"])["input_ids"][0][0]
assert final_top[-1] == paris
Show the code 29 lines
top_tokens = [
[repr(tokenizer.decode([int(i)])) for i in row] for row in lens.argmax(-1)
]
finals = [repr(tokenizer.decode([int(i)])) for i in final_top]
fig = go.Figure(
go.Heatmap(
z=p_final,
x=[f"{t}: {label}" for t, label in enumerate(labels)],
y=np.arange(31),
text=top_tokens,
texttemplate="%{text}",
textfont=dict(size=10),
customdata=[finals] * 31,
zmin=0,
zmax=1,
colorscale=viz.SEQUENTIAL,
colorbar=dict(title="P(final)", thickness=12),
xgap=1,
ygap=1,
hovertemplate="after %{x}, layer %{y}<br>top token %{text}<br>P(%{customdata}) = %{z:.3f}<extra></extra>",
)
)
fig.update_layout(
title="Logit lens: the top next token at each position and layer",
height=820,
xaxis=dict(title="position: the token read last", showgrid=False),
yaxis=dict(title="layer", dtick=2, showgrid=False),
)
viz.show(fig)
Each column is a position, labeled with the token read last; each cell shows the layer’s top guess for the next token, shaded by the probability of the position’s final prediction. Read the last column from the bottom up: the top token is filler, a period or a space and then “ the“ from layer 23, and “ Paris“ becomes the top token only after layer 30, the last. Hover to read its probability: near zero through layer 28, about 0.06 after layer 29 and about 0.47 after layer 30. The middle positions settle earlier: “ of“ after “ capital“ from layer 25, and “ the“ after “ of“ from layer 23. Their probability peaks a few layers before the end, and the last layers spread some of it to other tokens.
Compile
Running 30 layers eagerly works but executes hundreds of operations one at a time. Compiling turns the whole forward pass into one optimized graph. The input is a batch of token-ID sequences of any length:
start = time.perf_counter()
compiled = model.compile(
TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)
print(f"compiled in {time.perf_counter() - start:.0f} s")
start = time.perf_counter()
logits = compiled(Tensor(np.array([ids]), device=DEVICE)).to_numpy()
print(
f"forward pass: {(time.perf_counter() - start) * 1000:.0f} ms, logits {logits.shape}"
)
compiled in 13 s
forward pass: 115 ms, logits (1, 5, 49152)
Check against MAX’s SmolLM2
AutoModel loads SmolLM2 with MAX’s own implementation of the architecture
that config.json names LlamaForCausalLM, and can return the logits for
each token it generates.
The first generated token’s logits must match your model’s logits at the
last prompt position:
reference_model = AutoModel(MODEL_ID, quantization_encoding="float32")
reference = reference_model.generate(
input_ids=[ids], max_new_tokens=1, output_logits=True
)
expected = reference.logits[0][0]
reference_model.close()
ours = logits[0, -1]
print(
f"max difference {np.abs(ours - expected).max():.1e} (logits range up to {np.abs(expected).max():.0f})"
)
print("top token, yours:", repr(tokenizer.decode([int(ours.argmax())])))
print("top token, MAX's:", repr(tokenizer.decode([int(expected.argmax())])))
assert (
ours.argmax() == expected.argmax() and np.abs(ours - expected).max() < 1e-2
)
max difference 4.0e-05 (logits range up to 21)
top token, yours: ' Paris'
top token, MAX's: ' Paris'
The check covers every one of the 49,152 logits, not only the top one. Count how far each of your logits is from MAX’s; the dashed line marks the tolerance the check allows, and every logit sits orders of magnitude below it:
Show the code 23 lines
# Exact matches go in the lowest bin, since a log scale has no zero.
log_error = np.log10(np.maximum(np.abs(ours - expected), 1e-9))
fig = go.Figure(
go.Histogram(
x=log_error,
xbins=dict(start=-9, end=0, size=0.25),
marker_color=viz.BLUE,
hovertemplate="%{y:,} logits<extra></extra>",
)
)
fig.add_vline(x=-2, line=dict(color=viz.ORANGE, dash="dash"))
fig.update_layout(
title="Difference between your logits and MAX's, over the whole vocabulary",
height=380,
bargap=0.05,
xaxis=dict(
title="absolute difference",
tickvals=list(range(-9, 1)),
ticktext=[f"1e{p}" for p in range(-9, 1)],
),
yaxis_title="logits",
)
viz.show(fig)
In the run shown here, on an Apple-silicon Mac with a Metal GPU, the largest
difference between your logits and MAX’s was about 4e-5, and both models
predict the same next token. Treat these numbers as an example: yours depend on
your hardware and vary from run to run. An error in any layer, such as a wrong
RoPE layout or a missing scale, would change the logits by whole units, so the
match shows that every layer you wrote is correct.
The same code runs the larger SmolLM2 models by changing MODEL_ID. Other
Llama-family models need small additions: Llama 3.1, for example, rescales
RoPE’s frequencies (rope_scaling in its config), and Qwen2 adds biases to the
query, key, and value projections. The handbook’s
Choosing the right model
discusses how size, architecture, and license trade off when you pick one.
Exercises
- Run the logit lens on a longer prompt of your own. At which layer does each position settle on its final prediction?
- Load
HuggingFaceTB/SmolLM2-360M-Instructwith the same code. What changes inconfig, and how do compile and forward times scale? - Break something on purpose, such as dropping
rotate_halffromapply_rope, and see how far the logits move.
Next
Section 1.7 turns logits into
text: the generation loop, and the sampling strategies behind temperature,
top_k, and top_p.
1.7 Generation and sampling
Your model turns tokens into next-token logits. This section turns logits into
text: the generation loop, and the sampling strategies behind the temperature,
top_k, and top_p settings from
Section 1.1. Everything runs
on your own model from
Section 1.6.
Show the setup 32 lines
import logging
import time
import warnings
import numpy as np
import plotly.graph_objects as go
from IPython.display import Markdown, display
from plotly.subplots import make_subplots
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.functional import random
from max.graph import TensorType
from max.pipelines import AutoModel, AutoTokenizer
from max.tensor import Tensor
from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.model import load_model
from minillm.weights import load_checkpoint
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
config = LlamaConfig.from_pretrained()
tokenizer = AutoTokenizer(MODEL_ID)
model = load_model(config, load_checkpoint(), DEVICE)
compiled = model.compile(
TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)
The generation loop needs one thing from the model: the logits for the next token, given the tokens so far.
def next_logits(ids: list[int]) -> Tensor:
logits = compiled(Tensor(np.array([ids]), device=DEVICE))
return logits[0, -1] # [vocab_size]
Greedy decoding
The simplest strategy picks the most likely token every time:
prompt = tokenizer(["The capital of France is"])["input_ids"][0]
ids = list(prompt)
for _ in range(24):
ids.append(int(F.argmax(next_logits(ids)).item()))
print(tokenizer.decode(ids))
The capital of France is Paris. Paris is the largest city in France and the capital of the French department of the Espace de Paris.
Greedy decoding is deterministic, so it is the one strategy you can compare
token for token with MAX’s own pipeline. top_k=1 makes AutoModel greedy
too:
reference = AutoModel(MODEL_ID, quantization_encoding="float32")
expected = reference.generate(
input_ids=[prompt], max_new_tokens=24, top_k=1
).sequences[0]
reference.close()
print("same tokens as AutoModel:", list(expected[-24:]) == ids[-24:])
assert list(expected[-24:]) == ids[-24:]
same tokens as AutoModel: True
Greedy text is coherent but tends to repeat itself and to pick the blandest continuation. Sampling picks tokens at random, in proportion to their probability, to get more varied text.
Temperature
Softmax turns logits \(z\) into probabilities. Dividing the logits by a temperature \(T\) first reshapes the distribution:
\[p_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}\]
Below 1, the likely tokens get likelier; above 1, the distribution flattens toward uniform. As \(T \to 0\), sampling becomes greedy. Move the slider to see it on the twelve likeliest next tokens of a story:
story = tokenizer(["Once upon a time, there was a"])["input_ids"][0]
logits = next_logits(story)
top = np.argsort(logits.to_numpy())[::-1][:12]
labels = [repr(tokenizer.decode([int(i)])) for i in top]
temperatures = [0.1, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5, 2.0]
Show the code 26 lines
steps = []
for temperature in temperatures:
probs = F.softmax(logits / temperature).to_numpy()
steps.append(
[
go.Bar(
x=labels,
y=probs[top],
marker_color=viz.BLUE,
hovertemplate="%{x}: %{y:.3f}<extra></extra>",
)
]
)
fig = go.Figure(
layout=dict(
title="Next-token probabilities after 'Once upon a time, there was a'",
yaxis=dict(title="probability", range=[0, 1]),
height=460,
bargap=0.25,
)
)
viz.show(
viz.with_slider(
fig, steps, [str(t) for t in temperatures], "temperature ", active=4
)
)
Sampling with MAX operations
To sample from a distribution, you could compute the softmax and its cumulative sum, then search it with a random number. The Gumbel-max trick does it with operations you already have: add Gumbel noise \(-\log(-\log u)\), with \(u\) uniform in \((0, 1)\), to the scaled logits, and take the argmax:
\[\text{token} = \arg\max_i \left( \frac{z_i}{T} - \log(-\log u_i) \right), \qquad u_i \sim \mathrm{Uniform}(0, 1)\]
The sampled tokens follow the softmax distribution exactly, without computing the softmax. A quick check against the probabilities:
def gumbel_argmax(scores: Tensor) -> Tensor:
u = random.uniform(
scores.shape,
range=(1e-6, 1.0),
dtype=scores.dtype,
device=scores.device,
)
return F.argmax(scores - F.log(-F.log(u)))
toy = Tensor(np.array([2.0, 1.0, 0.5, 0.1, -1.0], np.float32), device=DEVICE)
counts = np.bincount(
[int(gumbel_argmax(toy).item()) for _ in range(2000)], minlength=5
)
print("sampled:", (counts / 2000).round(3))
print("softmax:", F.softmax(toy).to_numpy().round(3))
assert np.abs(counts / 2000 - F.softmax(toy).to_numpy()).max() < 0.05
sampled: [0.546 0.209 0.138 0.079 0.028]
softmax: [0.559 0.205 0.125 0.084 0.028]
Top-k and top-p
Even at a sensible temperature, the long tail of unlikely tokens adds up, and one bad draw can derail the text. Two filters cut the tail before sampling:
- Top-k keeps the \(k\) most likely tokens.
- Top-p (nucleus sampling) keeps the smallest set of tokens whose probabilities add up to at least \(p\), so it keeps few tokens when the model is confident and many when it isn’t.
Both filters work on tokens sorted from most to least likely. F.top_k
returns the \(k\) largest scores already sorted, with their token IDs. For
top-p alone, F.argsort sorts the whole vocabulary and F.gather puts the
scores in that order. F.cumsum over the sorted probabilities then gives
the running total, and F.where drops every token that the total had
already passed \(p\) before reaching. The filtered scores go through the
Gumbel-max trick, and the ID map turns the winning position back into a
token. Without a filter, nothing needs sorting.
Show sample, generate 57 lines
from __future__ import annotations
from collections.abc import Callable
from max import functional as F
from max.functional import random
from max.tensor import Tensor
def sample(
logits: Tensor,
temperature: float = 1.0,
top_k: int | None = None,
top_p: float | None = None,
) -> int:
"""Picks a token ID from 1-D ``logits``; ``temperature=0`` is greedy."""
if temperature == 0:
return int(F.argmax(logits).item())
scores, token_ids = logits / temperature, None
if top_k is not None:
scores, token_ids = F.top_k(scores, top_k) # sorted, largest first
if top_p is not None:
if token_ids is None:
token_ids = F.argsort(scores, ascending=False)
scores = F.gather(scores, token_ids, axis=0)
probs = F.softmax(scores)
before = F.cumsum(probs, axis=-1) - probs # mass of likelier tokens
scores = F.where(before < top_p, scores, float("-inf"))
u = random.uniform(
scores.shape,
range=(1e-6, 1.0),
dtype=scores.dtype,
device=scores.device,
)
choice = int(F.argmax(scores - F.log(-F.log(u))).item())
return choice if token_ids is None else int(token_ids.to_numpy()[choice])
def generate(
next_logits: Callable[[list[int]], Tensor],
prompt: list[int],
max_new_tokens: int,
eos_token_id: int | None = None,
**sampling: float | None,
) -> list[int]:
"""Generates up to ``max_new_tokens`` tokens after ``prompt``.
``next_logits`` maps the tokens so far to the next token's logits.
Generation stops early at ``eos_token_id``. Returns only the new tokens.
"""
ids = list(prompt)
for _ in range(max_new_tokens):
token = sample(next_logits(ids), **sampling)
ids.append(token)
if token == eos_token_id:
break
return ids[len(prompt) :]
Saved to minillm/sampling.py.
Top-p adapts to the model’s confidence. After “The capital of France is”, two tokens hold about three quarters of the probability; after “Once upon a time, there was a”, it spreads over many. Move \(p\) and compare how many tokens each prompt keeps (blue) and cuts (gray). The bars show the likeliest 40; the count covers the whole vocabulary:
prompts = ["The capital of France is", "Once upon a time, there was a"]
shown = 40
sorted_probs = []
for text in prompts:
probs = F.softmax(next_logits(tokenizer([text])["input_ids"][0])).to_numpy()
sorted_probs.append(np.sort(probs)[::-1])
Show the code 58 lines
ps = [0.5, 0.75, 0.9, 0.95, 0.99]
rank = np.arange(1, shown + 1)
steps = []
for p in ps:
group = []
for axis, probs in zip(["", "2"], sorted_probs):
before = np.cumsum(probs) - probs # the same test as `sample`
kept = before < p
count = int(kept.sum())
kept, top = kept[:shown], probs[:shown]
for mask, color, name in [
(kept, viz.BLUE, "kept"),
(~kept, viz.MUTED, "cut"),
]:
group.append(
go.Bar(
x=rank[mask],
y=top[mask],
marker_color=color,
name=name,
showlegend=False,
xaxis=f"x{axis}",
yaxis=f"y{axis}",
hovertemplate="rank %{x}: %{y:.3f}<extra>"
+ name
+ "</extra>",
)
)
group.append(
go.Scatter(
x=[shown],
y=[0.9],
mode="text",
text=[f"{count:,} kept"],
textposition="middle left",
showlegend=False,
hoverinfo="skip",
xaxis=f"x{axis}",
yaxis=f"y{axis}",
)
)
steps.append(group)
fig = make_subplots(
rows=2,
cols=1,
shared_xaxes=True,
vertical_spacing=0.12,
subplot_titles=[repr(text) for text in prompts],
)
fig.update_layout(
height=560,
bargap=0.15,
title="The likeliest 40 next tokens, sorted, at temperature 1",
)
fig.update_yaxes(range=[0, 1], title="probability")
fig.update_xaxes(title="rank", row=2, col=1)
viz.show(viz.with_slider(fig, steps, [str(p) for p in ps], "top_p ", active=2))
generate stops at the model’s end-of-sequence token, which a chat model
emits when its answer is complete. Try a few settings. random.set_seed
makes a sampled run repeatable:
for settings in [
dict(temperature=0),
dict(temperature=0.7, top_p=0.9),
dict(temperature=1.0, top_k=50),
dict(temperature=1.5),
]:
random.set_seed(0)
new = generate(next_logits, story, 24, config.eos_token_id, **settings)
print(f"{settings}\n {tokenizer.decode(new)!r}\n")
{'temperature': 0}
' young woman named Sarah who was struggling to find her place in the world. She had always been a bit of a rebel'
{'temperature': 0.7, 'top_p': 0.9}
' young girl named Lily who was determined to be a better version of herself. She had always been the "perfect" girl'
{'temperature': 1.0, 'top_k': 50}
' young musician who had spent their life playing and composing. He was known as "The Weaver" due to his exceptional skill'
{'temperature': 1.5}
' neighboring realm on your planet, mist. Even then; Unlocking beastellandsis deathshire istsubbralgenaki'
Why does a high temperature drift? Color each generated token by the probability it had in the distribution it was drawn from (temperature 1 for the greedy run), and hover over a chip for the five likeliest alternatives:
def generate_with_probs(
prompt: list[int], n: int, temperature: float
) -> tuple[list[int], list[np.ndarray]]:
"""Generates like `generate`, keeping each step's sampling distribution."""
ids, dists = list(prompt), []
for _ in range(n):
logits = next_logits(ids)
dists.append(F.softmax(logits / (temperature or 1.0)).to_numpy())
token = sample(logits, temperature=temperature)
ids.append(token)
if token == config.eos_token_id:
break
return ids[len(prompt) :], dists
runs = {}
for temperature in [0, 1.5]:
random.set_seed(0)
runs[temperature] = generate_with_probs(story, 24, temperature)
Show chip_title 20 lines
def chip_title(token: int, dist: np.ndarray) -> str:
likeliest = np.argsort(dist)[::-1][:5]
alternatives = ", ".join(
f"{tokenizer.decode([int(i)])!r} {dist[i]:.2f}" for i in likeliest
)
return f"p = {dist[token]:.2f}; likeliest: {alternatives}"
for temperature, (new, dists) in runs.items():
chosen = [float(d[t]) for t, d in zip(new, dists)]
mean = np.mean(chosen)
display(Markdown(f"`temperature={temperature}`, mean p = {mean:.2f}:"))
display(
viz.tokens(
[tokenizer.decode([t]) for t in new],
new,
colors=[f"rgba(42,120,214,{0.08 + 0.72 * p:.2f})" for p in chosen],
titles=[chip_title(t, d) for t, d in zip(new, dists)],
)
)
temperature=0, mean p = 0.56:
temperature=1.5, mean p = 0.01:
Higher temperatures read less predictably. The greedy run picks tokens with
probabilities from about 0.1 to over 0.9. At 1.5 without a filter, nearly
every chip is pale: a flattened distribution gives the long tail of unlikely
tokens enough combined weight that most picks are tokens the model gave
almost no chance, and the text turns to gibberish within a few words. Many chat applications use settings near
temperature=0.7 and top_p=0.9. The handbook’s
inference parameters
page covers these and the other settings servers accept, such as stop sequences
and repetition penalties.
The cost of each token
Generating the token after \(n\) tokens runs the model over all \(n\) of them. Time that step at several lengths, with random token IDs since only the length matters:
rng = np.random.default_rng(0)
lengths = [16, 64, 256, 512, 1024, 2048]
step_ms = []
for length in lengths:
ids = rng.integers(0, config.vocab_size, length).tolist()
next_logits(ids).to_numpy() # warm up this length
times = []
for _ in range(5):
start = time.perf_counter()
next_logits(ids).to_numpy()
times.append((time.perf_counter() - start) * 1000)
step_ms.append(float(np.median(times)))
print(f"{length:5} tokens: {step_ms[-1]:6.1f} ms per new token")
16 tokens: 57.7 ms per new token
64 tokens: 60.1 ms per new token
256 tokens: 70.9 ms per new token
512 tokens: 179.8 ms per new token
1024 tokens: 563.3 ms per new token
2048 tokens: 1839.3 ms per new token
Show the code 17 lines
fig = go.Figure(
go.Scatter(
x=lengths,
y=step_ms,
mode="lines+markers",
line_color=viz.ORANGE,
hovertemplate="%{x} tokens: %{y:.1f} ms<extra></extra>",
)
)
fig.update_layout(
title="Time to generate one token, recomputing the whole sequence",
xaxis_title="sequence length (tokens)",
yaxis_title="ms per token",
yaxis_rangemode="tozero",
height=380,
)
viz.show(fig)
In the run shown here, on an Apple-silicon Mac with a Metal GPU, a step took about 70 to 110 ms up to 256 tokens, about 560 ms at 1,024 tokens, and about 1.8 s at 2,048. Treat these numbers as an example: yours depend on your hardware and vary from run to run. For short sequences the step costs about the same at any length, because fixed work, such as reading every weight and launching each kernel, dominates. Past a few hundred tokens the cost climbs faster than the length itself, because attention compares every position with every earlier one. Other devices give other numbers with the same shape.
The keys and values of earlier positions never change, so all the work of recomputing them is wasted. Section 1.8 caches them. The handbook’s LLM inference metrics page explains why this per-token time, the TPOT from Section 1.1, is the number users feel while a reply streams in.
Exercises
- Sample the same prompt 5 times at
temperature=1.0with and withouttop_p=0.9. How often does each run go off topic? - Implement min-p sampling: keep tokens whose probability is at least
min_ptimes the top token’s probability. Which MAX operations do you need? - Plot how many tokens top-p keeps at each step of a generation. When is the model confident?
Next
Section 1.8 adds a KV cache so each step processes one token instead of the whole sequence.
1.8 The KV cache
Section 1.7 ended with a problem: each generated token reruns the model over the whole sequence, so the cost per token grows with the text. This section fixes it the way inference engines do, with a KV cache, and measures the difference.
Why caching works
To compute attention for a new token, the model needs the keys and values of every earlier position, in every layer. Those keys and values depend only on the tokens up to their own position, so they never change once computed. Recomputing them every step is wasted work. The KV cache stores them instead, which splits generation into two phases:
- Prefill runs the whole prompt at once, like before, and saves every layer’s keys and values.
- Decode runs one new token per step. Each layer computes the new token’s query, key, and value, appends the key and value to its cache, and attends over the whole cache.
The handbook’s How does LLM inference work? page describes the same two phases from the serving side.
Show the setup 27 lines
import logging
import time
import warnings
import numpy as np
import plotly.graph_objects as go
from max.driver import CPU
from plotly.subplots import make_subplots
from max.dtype import DType
from max.graph import TensorType
from max.pipelines import AutoTokenizer
from max.tensor import Tensor
from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.model import load_model
from minillm.sampling import generate
from minillm.weights import load_checkpoint
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
config = LlamaConfig.from_pretrained()
weights = load_checkpoint()
tokenizer = AutoTokenizer(MODEL_ID)
The difference in work is easy to see. Each row below is one generation step for a 6-token prompt; each column is a position whose key and value that step computes. Without a cache, every step recomputes every earlier position. With one, prefill computes the prompt once and each decode step computes one new position:
prompt_len, steps = 6, 10
step_labels = ["prefill"] + [f"decode {i}" for i in range(1, steps)]
recompute_work = np.zeros((steps, prompt_len + steps - 1))
cached_work = np.zeros_like(recompute_work)
for step in range(steps):
end = prompt_len + step
recompute_work[step, :end] = 1
cached_work[step, :end] = 1 if step == 0 else 0
cached_work[step, end - 1] = 1
Show the code 28 lines
fig = make_subplots(
rows=1,
cols=2,
shared_yaxes=True,
horizontal_spacing=0.06,
subplot_titles=[
f"Recompute: {int(recompute_work.sum())} key/value computations",
f"KV cache: {int(cached_work.sum())} key/value computations",
],
)
for col, work in enumerate([recompute_work, cached_work], start=1):
fig.add_trace(
go.Heatmap(
z=np.where(work > 0, 1, np.nan),
y=step_labels,
xgap=2,
ygap=2,
colorscale=[[0, viz.BLUE], [1, viz.BLUE]],
showscale=False,
hovertemplate="%{y}: computes position %{x}<extra></extra>",
),
row=1,
col=col,
)
fig.update_xaxes(title="position", showgrid=False, dtick=2)
fig.update_yaxes(autorange="reversed", showgrid=False)
fig.update_layout(height=380, title="Which keys and values each step computes")
viz.show(fig)
How big is the cache?
Each token stores one key and one value vector per key/value head, per layer. With \(L\) layers, \(H_{kv}\) key/value heads of dimension \(d\), and \(b\) bytes per number:
\[\text{bytes per token} = 2 \cdot L \cdot H_{kv} \cdot d \cdot b = 2 \cdot 30 \cdot 3 \cdot 64 \cdot 4\]
per_token = (
2
* config.num_hidden_layers
* config.num_key_value_heads
* config.head_dim
* 4
)
print(f"{per_token / 1024:.0f} KiB per token in float32")
print(
f"{per_token * 8192 / 2**20:.0f} MiB for SmolLM2's full 8192-token context"
)
print(
f"{per_token * 3 * 8192 / 2**20:.0f} MiB if every query head had its own key/value head"
)
45 KiB per token in float32
360 MiB for SmolLM2's full 8192-token context
1080 MiB if every query head had its own key/value head
The cache grows with the context and with the number of sequences served at once. Compare it with the model’s weights:
weights_mib = sum(w.nbytes for w in weights.values()) / 2**20
two_sequences_mib = per_token * 8192 * 2 / 2**20
print(f"weights: {weights_mib:.0f} MiB")
print(f"cache for two sequences of 8192 tokens: {two_sequences_mib:.0f} MiB")
weights: 513 MiB
cache for two sequences of 8192 tokens: 720 MiB
Move the slider to serve more sequences at once:
Show the code 43 lines
lengths = np.arange(256, 8193, 256)
batches = [1, 2, 4, 16, 64]
steps = []
for batch in batches:
steps.append(
[
go.Scatter(
x=lengths,
y=per_token * lengths * batch / 2**20,
name="KV cache, 3 key/value heads",
line_color=viz.BLUE,
hovertemplate="%{x} tokens: %{y:,.0f} MiB<extra></extra>",
),
go.Scatter(
x=lengths,
y=3 * per_token * lengths * batch / 2**20,
name="KV cache, 9 key/value heads",
line=dict(color=viz.ORANGE, dash="dot"),
hovertemplate="%{x} tokens: %{y:,.0f} MiB<extra></extra>",
),
]
)
fig = go.Figure(
go.Scatter(
x=[lengths[0], lengths[-1]],
y=[weights_mib] * 2,
name="weights",
mode="lines",
line=dict(color=viz.INK_SECONDARY, dash="dash"),
hovertemplate="weights: %{y:,.0f} MiB<extra></extra>",
)
)
fig.update_layout(
title="float32 KV cache memory against context length",
height=460,
xaxis_title="context length (tokens)",
yaxis=dict(title="MiB", type="log", dtick=1),
)
viz.show(
viz.with_slider(
fig, steps, [str(b) for b in batches], "sequences served at once: "
)
)
This is where grouped-query attention
(Section 1.5) pays off: 3
key/value heads instead of 9 make the cache 3 times smaller. Even for SmolLM2 in
float32, two sequences at the full 8,192-token context need a 720 MiB cache,
more than the 513 MiB of weights. For large models serving many users, the cache
often takes more GPU memory than the weights. The handbook’s
GPU memory calculation
adds it to the weights when sizing a deployment.
Attention with a cache
CachedAttention extends
Section 1.5’s Attention. Its forward takes the
layer’s cached keys and values, if any, and returns them extended by the new
positions. start is the position of the first new token, so RoPE rotates
the new queries and keys for where they really are.
Prefill uses the causal mask as before. Decode has one query, the newest token, which may attend to every cached position, so it needs no mask.
Show CachedAttention 37 lines
from __future__ import annotations
import math
import numpy as np
from max import functional as F
from max.driver import Device
from max.dtype import DType
from max.graph import TensorType
from max.nn import ModuleList
from max.nn.kernels import flash_attention
from max.tensor import Tensor
from minillm.attention import Attention
from minillm.config import LlamaConfig
from minillm.model import DecoderLayer, LlamaForCausalLM, LlamaModel
KVPair = tuple[Tensor, Tensor]
class CachedAttention(Attention):
"""Attention that reads and extends a key/value cache."""
def forward(
self, x: Tensor, start: Tensor, past: KVPair | None = None
) -> tuple[Tensor, KVPair]:
q, k, v = self.project(x, start)
if past is None: # prefill: causal attention over the prompt
return self.attend(q, k, v), (k, v)
k = F.concat([past[0], k], axis=1)
v = F.concat([past[1], v], axis=1)
# The one new token may attend to every cached position: no mask.
out = flash_attention(q, k, v, 1 / math.sqrt(self.head_dim))
out = out.rebind_mapping(q.mapping)
batch, seq = q.shape[0], q.shape[1]
out = out.reshape([batch, seq, self.num_heads * self.head_dim])
return self.o_proj(out), (k, v)
Saved to minillm/kv_cache.py.
The layer and model pass each layer’s cache through. The model returns logits for the last position only: generation never needs the others, and skipping them saves most of the LM head’s work during prefill.
Show CachedDecoderLayer, CachedLlamaModel, CachedLlamaForCausalLM 52 lines
class CachedDecoderLayer(DecoderLayer):
def __init__(self, config: LlamaConfig) -> None:
super().__init__(config)
self.self_attn = CachedAttention(config)
def forward(
self, x: Tensor, start: Tensor, past: KVPair | None = None
) -> tuple[Tensor, KVPair]:
attn, kv = self.self_attn(self.input_layernorm(x), start, past)
x = x + attn
return x + self.mlp(self.post_attention_layernorm(x)), kv
class CachedLlamaModel(LlamaModel):
def __init__(self, config: LlamaConfig) -> None:
super().__init__(config)
self.layers = ModuleList(
[
CachedDecoderLayer(config)
for _ in range(config.num_hidden_layers)
]
)
def forward(
self,
input_ids: Tensor,
start: Tensor,
cache: list[KVPair] | None = None,
) -> tuple[Tensor, list[KVPair]]:
x = self.embed_tokens(input_ids)
new_cache = []
for i, layer in enumerate(self.layers):
x, kv = layer(x, start, None if cache is None else cache[i])
new_cache.append(kv)
return self.norm(x), new_cache
class CachedLlamaForCausalLM(LlamaForCausalLM):
"""Returns the last position's logits and the updated cache."""
def __init__(self, config: LlamaConfig) -> None:
super().__init__(config)
self.model = CachedLlamaModel(config)
def forward(
self,
input_ids: Tensor,
start: Tensor,
cache: list[KVPair] | None = None,
) -> tuple[Tensor, list[KVPair]]:
hidden, new_cache = self.model(input_ids, start, cache)
return self.lm_head(hidden[:, -1]), new_cache
Saved to minillm/kv_cache.py.
Two compiled graphs
Prefill and decode have different inputs, so they compile into two graphs
that share the same weights. The cache is a list of 30 (key, value) pairs.
Its length, cache_len, is a named dimension, so one decode graph serves
every step. start is a one-element tensor rather than a Python number, so
that it is an input of the graph instead of a constant baked into it. It is
float32 because RoPE computes its angles in float32, and float32 holds
every position up to 2^24 exactly.
KVCacheDecoder keeps the cache between calls. It has the same interface as
Section 1.7’s next_logits: given the tokens so far, it returns the next
token’s logits, running a prefill for a new prompt and a decode step when
exactly one token was added. It plugs straight into generate.
Show KVCacheDecoder 41 lines
class KVCacheDecoder:
"""Returns next-token logits like a full forward pass, using a cache."""
def __init__(
self, model: CachedLlamaForCausalLM, config: LlamaConfig, device: Device
) -> None:
self.device = device
tokens = TensorType(DType.int64, ("batch", "seq_len"), device=device)
one_token = TensorType(DType.int64, ("batch", 1), device=device)
start = TensorType(DType.float32, [1], device=device)
kv = TensorType(
DType.float32,
("batch", "cache_len", config.num_key_value_heads, config.head_dim),
device=device,
)
self.prefill = model.compile(tokens, start)
self.decode = model.compile(
one_token, start, [(kv, kv)] * config.num_hidden_layers
)
self.cache: list[KVPair] | None = None
self.length = 0
def __call__(self, ids: list[int]) -> Tensor:
if self.cache is not None and len(ids) == self.length + 1:
logits, self.cache = self.decode(
self._tokens(ids[-1:]), self._position(self.length), self.cache
)
else:
logits, self.cache = self.prefill(
self._tokens(ids), self._position(0)
)
self.length = len(ids)
return logits[0]
def _tokens(self, ids: list[int]) -> Tensor:
return Tensor(np.array([ids], dtype=np.int64), device=self.device)
def _position(self, position: int) -> Tensor:
return Tensor(
np.array([position], dtype=np.float32), device=self.device
)
Saved to minillm/kv_cache.py.
start = time.perf_counter()
cached = KVCacheDecoder(
load_model(config, weights, DEVICE, CachedLlamaForCausalLM), config, DEVICE
)
print(f"compiled prefill and decode in {time.perf_counter() - start:.0f} s")
compiled prefill and decode in 30 s
Check it
The cache must not change the result. Generate greedily with Section 1.7’s full recompute and with the cache, and compare the tokens:
full = load_model(config, weights, DEVICE).compile(
TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)
def recompute(ids: list[int]) -> Tensor:
return full(Tensor(np.array([ids]), device=DEVICE))[0, -1]
prompt = tokenizer(["The capital of France is"])["input_ids"][0]
slow = generate(recompute, prompt, 48, temperature=0)
fast = generate(cached, prompt, 48, temperature=0)
print("identical tokens:", slow == fast)
assert slow == fast
print(tokenizer.decode(prompt + fast))
print("cached keys per layer:", [int(d) for d in cached.cache[0][0].shape])
identical tokens: True
The capital of France is Paris. Paris is the largest city in France and the capital of the French department of the Espace de Paris.
The Eiffel Tower, also known as the Eiffel Tower, is a monumental structure that stands 324
cached keys per layer: [1, 52, 3, 64]
Measure it
Time one decode step at several context lengths, with random prompts so that only the length matters:
def ms_per_token(next_logits, prompt: list[int], steps: int = 24) -> float:
ids = list(prompt)
ids.append(int(np.argmax(next_logits(ids).to_numpy()))) # prefill, untimed
start = time.perf_counter()
for _ in range(steps):
ids.append(int(np.argmax(next_logits(ids).to_numpy())))
return (time.perf_counter() - start) / steps * 1000
rng = np.random.default_rng(0)
lengths = [32, 128, 256, 512, 1024]
timings = {"recompute": [], "KV cache": []}
for length in lengths:
prompt = rng.integers(0, config.vocab_size, length).tolist()
timings["recompute"].append(ms_per_token(recompute, prompt))
timings["KV cache"].append(ms_per_token(cached, prompt))
for name, ms in timings.items():
print(f"{name:10}", " ".join(f"{t:6.1f}" for t in ms), "ms")
recompute 100.9 130.0 134.0 209.5 595.8 ms
KV cache 112.9 136.1 110.8 135.5 169.9 ms
Show the code 21 lines
fig = go.Figure(
[
go.Scatter(
x=lengths,
y=ms,
name=name,
mode="lines+markers",
line_color=color,
hovertemplate=name + ", %{x} tokens: %{y:.1f} ms<extra></extra>",
)
for (name, ms), color in zip(timings.items(), [viz.ORANGE, viz.BLUE])
]
)
fig.update_layout(
title="Time per decode step",
height=420,
xaxis_title="context length (tokens)",
yaxis_title="ms per token",
yaxis_rangemode="tozero",
)
viz.show(fig)
Hover over the points to read the timings; yours depend on your hardware and vary from run to run. Recompute grows with the context, because every step reruns attention over the whole sequence. The cached step grows much more slowly. Below a few hundred tokens the cache doesn’t pay off, and can even be slower, because fixed work, such as launching kernels and reading every weight, dominates a step of a model this small.
MAX’s own decode step is far faster: the TPOT measured in Section
1.1 was about 10 ms on the
same machine. Part of the gap is this section’s simple design: every decode call
passes the 60 cache tensors in and gets 60 new ones back, F.concat copies each
layer’s whole cache into a new tensor, and the loop reads the logits back to
NumPy to pick each token on the host. MAX’s engine writes new keys and values
into preallocated cache blocks that its attention kernel reads in place, and it
samples on the GPU. Section
5.3 builds a block-based
cache and explains how production engines avoid the copy.
This simple cache has limits that production engines address:
- Memory is allocated per sequence, as it grows. A server doesn’t know how long each reply will be, and contiguous per-sequence buffers fragment memory. Paged attention stores the cache in fixed-size blocks instead, like virtual memory pages.
- It serves one sequence at a time. A decode step for one token barely uses the hardware. Engines batch many sequences per step, which needs a cache that holds sequences of different lengths.
- Shared prompts are recomputed. Requests that start with the same system prompt could share those cache blocks (prefix caching).
Chapter 5 builds these into a small serving engine: Section 5.2 adds continuous batching, Section 5.3 a paged KV cache, and Section 5.4 prefix caching. The handbook covers each one: PagedAttention, batching, and prefix caching.
Exercises
- Extend the timing chart to context lengths from 16 to 2048. Below what length does the cache stop paying off on your device, and why?
- Make
KVCacheDecoderkeep the cache across two turns of a conversation: prefill only the new turn’s tokens after the first reply. - Store the cache in
bfloat16instead offloat32. How much memory does it save, and do the greedy tokens change?
What you built
Chapter 1 is complete. You have a working LLM inference stack written with MAX: a tokenizer and weight loader, a Llama-architecture model that matches MAX’s own, sampling, and a KV cache. Chapter 2 splits this model across several devices with MAX’s distributed tensors.
2. Distributed tensors
Large models don’t fit on one device, and even small ones serve faster when several devices share the work. In MAX, a tensor that spans several devices behaves like any other tensor: you describe how it is laid out, and MAX runs each operation on every device and inserts the communication it needs.
Distributed code is hard to develop when every test needs a multi-GPU machine. MAX can simulate a mesh of devices on one CPU, with the same code and the same results, so this whole chapter runs on a laptop or in Colab.
| Section | You learn |
|---|---|
| 2.1 Distributed tensors | Device meshes, placements, sharded matmuls, and the collectives they imply. |
| 2.2 A tensor-parallel model | Splitting SmolLM2 across three devices and checking that its logits don’t change. |
The handbook’s parallelism strategies and distributed inference pages explain when each strategy pays off in production.
2.1 Distributed tensors
A distributed tensor is one logical array whose data lives on several devices.
In MAX it is an ordinary Tensor, the same type you used from
Section 1.2 on:
operators, max.functional, and max.nn layers accept it, run on every device
that holds a piece, and add the communication between devices that the math
requires.
This section runs everything on a simulated mesh: four “devices” that are all the same CPU. The program, the per-device pieces, and the results are the same as on four GPUs, so you can develop and test distributed code on a laptop and run it unchanged on real hardware.
Show the setup 23 lines
import logging
import re
import warnings
import numpy as np
import plotly.graph_objects as go
from max import functional as F
from max.driver import CPU
from max.tensor import Tensor
from max.tensor.sharding import (
DeviceMapping,
DeviceMesh,
Partial,
Replicated,
Sharded,
auto_reshard,
visualize,
)
from plotly.subplots import make_subplots
from minillm import viz
logging.getLogger("max").setLevel(logging.ERROR)
Meshes and placements
A DeviceMesh arranges devices on named axes. This one has four devices on
one axis, named tp for tensor parallelism:
mesh = DeviceMesh((CPU(),) * 4, (4,), ("tp",))
def on(*placements) -> DeviceMapping:
"""Places a tensor on the mesh, with one placement per mesh axis."""
return DeviceMapping(mesh, placements)
A DeviceMapping says how a tensor’s data relates to the devices on each
mesh axis. That relation is its placement:
Replicated(): every device holds the whole tensor.Sharded(axis): the tensor is split alongaxis, one piece per device.Partial(): every device holds a full-size tensor, and the real value is their sum.
Split an 8 × 4 matrix by rows:
a = np.arange(32, dtype=np.float32).reshape(8, 4)
x = Tensor(a, device=on(Sharded(0)))
visualize(x)
DType.float32 [8, 4] with DeviceMapping(DeviceMesh(tp=4), [Sharded(axis=0)])
┌──────────┐ tensor[:, :]
│ device 0 │ ├── tp=0 device 0 - tensor[:2, :]
│ [:2, :] │ ├── tp=1 device 1 - tensor[2:4, :]
├──────────┤ ├── tp=2 device 2 - tensor[4:6, :]
│ device 1 │ └── tp=3 device 3 - tensor[6:, :]
│ [2:4, :] │
├──────────┤
│ device 2 │
│ [4:6, :] │
├──────────┤
│ device 3 │
│ [6:, :] │
└──────────┘
x.shape is the global shape, [8, 4], as if the tensor lived on one
device. Each device’s piece, its shard, is in local_shards:
print("global shape:", [int(d) for d in x.shape])
for i, shard in enumerate(x.local_shards):
print(f"device {i}:", [int(d) for d in shard.shape], shard.to_numpy()[:, 0])
global shape: [8, 4]
device 0: [2, 4] [0. 4.]
device 1: [2, 4] [ 8. 12.]
device 2: [2, 4] [16. 20.]
device 3: [2, 4] [24. 28.]
The next helper turns a tensor’s placement into a map of which device holds
each element, using only mapping.placements and the shard shapes. The
figures in this section draw it: one color per device, gray where every
device holds the element, and a darker gray for a Partial tensor, whose
every device holds a term of the sum.
Show owners, owner_heatmap, device_legend 73 lines
ALL, PARTIAL = -1, -2
def owners(t: Tensor) -> np.ndarray:
"""Returns the device that holds each element of ``t`` on a 1-D mesh."""
grid = np.full([int(d) for d in t.shape], ALL)
placement = t.mapping.placements[0]
if isinstance(placement, Partial):
grid[:] = PARTIAL
elif isinstance(placement, Sharded):
start = 0
for device, shard in enumerate(t.local_shards):
size = int(shard.shape[placement.axis])
index = [slice(None)] * grid.ndim
index[placement.axis] = slice(start, start + size)
grid[tuple(index)] = device
start += size
return grid
HOLDER = {ALL: "every device", PARTIAL: "every device, one term each"} | {
d: f"device {d}" for d in range(4)
}
# One band per value from -2 to 3: partial, all, then devices 0 to 3.
BANDS = [
viz.INK_SECONDARY,
viz.GRID,
viz.BLUE,
viz.ORANGE,
viz.AQUA,
viz.YELLOW,
]
OWNER_SCALE = [
[edge, color]
for i, color in enumerate(BANDS)
for edge in (i / 6, (i + 1) / 6)
]
def owner_heatmap(t: Tensor, show_values: bool = True, **trace) -> go.Heatmap:
grid = owners(t)
values = t.to_numpy()
rows, cols = grid.shape
holder = [
[HOLDER[int(grid[r, c])] for c in range(cols)] for r in range(rows)
]
return go.Heatmap(
z=grid,
zmin=-2.5,
zmax=3.5,
colorscale=OWNER_SCALE,
showscale=False,
xgap=2,
ygap=2,
customdata=holder,
text=values.round(1) if show_values else None,
texttemplate="%{text}" if show_values else None,
hovertemplate="row %{y}, column %{x}<br>held by %{customdata}<extra></extra>",
**trace,
)
def device_legend(fig: go.Figure) -> None:
for d in range(4):
fig.add_trace(
go.Scatter(
x=[None],
y=[None],
mode="markers",
name=f"device {d}",
marker=dict(symbol="square", size=12, color=BANDS[d + 2]),
)
)
Move the slider to compare placements of the same 8 × 4 matrix:
placements = {
"Sharded(0)": Sharded(0),
"Sharded(1)": Sharded(1),
"Replicated()": Replicated(),
}
steps = [[owner_heatmap(Tensor(a, device=on(p)))] for p in placements.values()]
fig = go.Figure(
layout=dict(
title="Which device holds each element of an 8 × 4 matrix",
height=520,
xaxis=dict(showgrid=False, dtick=1),
yaxis=dict(showgrid=False, autorange="reversed", dtick=1),
)
)
device_legend(fig)
viz.show(viz.with_slider(fig, steps, list(placements), "placement "))
to_numpy() gathers the shards back into one array, so checking a
distributed result against NumPy is one line:
print(np.array_equal(x.to_numpy(), a))
True
Sharded matrix multiplication
Models spend most of their time in matrix multiplications, and there are two natural ways to split \(y = x W\) across devices. Every tensor-parallel transformer uses both.
In a column-parallel multiplication, every device holds all of \(x\) and a slice of \(W\)’s columns, and computes the matching slice of \(y\). It needs no communication:
w = np.random.default_rng(0).standard_normal((4, 6)).astype(np.float32)
y = Tensor(a, device=on(Replicated())) @ Tensor(w, device=on(Sharded(1)))
print(y.mapping)
print("correct:", np.allclose(y.to_numpy(), a @ w, atol=1e-5))
assert np.allclose(y.to_numpy(), a @ w, atol=1e-5)
DeviceMapping(DeviceMesh(tp=4), [Sharded(axis=1)])
correct: True
In a row-parallel multiplication, each device holds a slice of \(x\)’s
columns and the matching slice of \(W\)’s rows. Each device’s product is one
term of the full sum over the inner dimension, so the result is Partial:
z = Tensor(a, device=on(Sharded(1))) @ Tensor(w, device=on(Sharded(0)))
print(z.mapping)
print("correct:", np.allclose(z.to_numpy(), a @ w, atol=1e-5))
assert np.allclose(z.to_numpy(), a @ w, atol=1e-5)
DeviceMapping(DeviceMesh(tp=4), [Partial()])
correct: True
A Partial tensor is correct only once its terms are summed. Most
operations, such as an activation, need the real value, so MAX must first
combine the shards with a collective, an operation that communicates
between devices. Inside auto_reshard(mode="warn"), MAX reports every
collective it inserts and the reason for it:
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
with auto_reshard(mode="warn"):
activated = F.relu(z)
report = str(next(w.message for w in caught if "reshards" in str(w.message)))
print(report)
print("result:", activated.mapping)
relu(x) reshards x
x : float32[8, 6]@mesh(tp=4){"tp":P}
region : auto_reshard(DEFAULT_TRANSITIONS, mode="warn")
decided per mesh axis
axis x -> out collective bytes status
"tp" x{P->S0} -> S0 reduce_scatter(x) 144 not allowed
x{P->S1} -> S1 reduce_scatter(x) 144 not allowed
x{P->R} -> R allreduce(x) 288 chosen, reshards x
result x{"tp":R} -> {"tp":R}
x = transfer_to(x, DeviceMapping(mesh, (R,)))
with auto_reshard(mode="silent"):
with auto_reshard(ALL_TRANSITIONS, mode="silent"): # also allows the cheaper rows above
at <cell>:4 in <module>()
result: DeviceMapping(DeviceMesh(tp=4), [Replicated()])
MAX chose an all-reduce: every device sends its partial sum and receives
the total. It also considered a reduce-scatter, which sums and leaves each
device one slice, but that transition isn’t in the default set. Outside an
auto_reshard block, MAX raises an error instead of inserting a collective,
so a model can’t add communication you didn’t plan for.
To move a tensor to a placement yourself, use F.transfer_to:
replicated = F.transfer_to(z, on(Replicated()))
print(replicated.mapping)
DeviceMapping(DeviceMesh(tp=4), [Replicated()])
The figure shows what the two collectives do to z, built from its real
shards. Each device in the top row holds a different full-size term, and the
four terms add up to a @ w. Move the slider to see what each device holds
afterward: an all-reduce gives every device the whole sum; a reduce-scatter
gives each device only its two rows of the sum, for half the bytes. The byte
counts come from MAX’s report above.
Show cells 66 lines
terms = [shard.to_numpy() for shard in z.local_shards]
assert np.allclose(sum(terms), a @ w, atol=1e-4)
after = {
"all-reduce": F.transfer_to(z, on(Replicated())),
"reduce-scatter": F.transfer_to(z, on(Sharded(0))),
}
cost = {
name: int(re.search(rf"{op}\(x\)\s+(\d+)", report).group(1))
for name, op in [
("all-reduce", "allreduce"),
("reduce-scatter", "reduce_scatter"),
]
}
limit = np.abs(a @ w).max()
def cells(values: np.ndarray, rows: list[int], **trace) -> go.Heatmap:
return go.Heatmap(
z=values,
y=rows,
zmin=-limit,
zmax=limit,
colorscale=viz.DIVERGING,
showscale=False,
xgap=1,
ygap=1,
texttemplate="%{z:.0f}",
textfont=dict(size=9),
hovertemplate="row %{y}, column %{x}: %{z:.2f}<extra></extra>",
**trace,
)
fig = make_subplots(
rows=2,
cols=4,
vertical_spacing=0.14,
horizontal_spacing=0.03,
subplot_titles=[f"device {d}: its term" for d in range(4)]
+ [f"device {d} afterward" for d in range(4)],
)
for d, term in enumerate(terms):
fig.add_trace(cells(term, list(range(8))), row=1, col=d + 1)
steps = []
for name, tensor in after.items():
group = []
start = 0
for d, shard in enumerate(tensor.local_shards):
values = shard.to_numpy()
rows = (
list(range(start, start + len(values)))
if len(values) < 8
else list(range(8))
)
assert np.allclose(values, (a @ w)[rows], atol=1e-4)
start += len(values)
group.append(cells(values, rows, xaxis=f"x{d + 5}", yaxis=f"y{d + 5}"))
steps.append(group)
fig.update_xaxes(showgrid=False, showticklabels=False)
fig.update_yaxes(showgrid=False, showticklabels=False, range=[7.5, -0.5])
fig.update_layout(
title="What a collective does to a Partial tensor on 4 devices",
height=560,
)
labels = [f"{name}, {cost[name]} bytes per device" for name in after]
viz.show(viz.with_slider(fig, steps, labels, ""))
Column then row: one all-reduce per block
Chaining the two patterns is the core of tensor parallelism, as introduced by the Megatron-LM paper. A column-parallel layer leaves its output split by columns, which is the input a row-parallel layer needs, so the pair runs with no communication between them and a single all-reduce at the end. Here is a small MLP block, checked against NumPy:
rng = np.random.default_rng(1)
w_up = rng.standard_normal((4, 16)).astype(np.float32)
w_down = rng.standard_normal((16, 4)).astype(np.float32)
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
with auto_reshard(mode="warn"):
h = F.relu(
Tensor(a, device=on(Replicated()))
@ Tensor(w_up, device=on(Sharded(1)))
)
partial_out = h @ Tensor(w_down, device=on(Sharded(0)))
reshards = [w for w in caught if "reshards" in str(w.message)]
out = F.transfer_to(partial_out, on(Replicated())) # the one all-reduce
print("hidden:", h.mapping, " block output:", partial_out.mapping)
print("collectives MAX had to insert:", len(reshards))
print(
"correct:",
np.allclose(out.to_numpy(), np.maximum(a @ w_up, 0) @ w_down, atol=1e-3),
)
assert not reshards and np.allclose(
out.to_numpy(), np.maximum(a @ w_up, 0) @ w_down, atol=1e-3
)
hidden: DeviceMapping(DeviceMesh(tp=4), [Sharded(axis=1)]) block output: DeviceMapping(DeviceMesh(tp=4), [Partial()])
collectives MAX had to insert: 0
correct: True
Here is the whole block as placements. Every device holds all of x. The
column-parallel weight and the hidden activation are split by columns in the
same way, so each device multiplies only its own slice. The row-parallel
weight is split by rows to match, and the block’s output, h W_down, is a
partial sum on every device (dark gray) until the all-reduce sums it:
Show the code 28 lines
panels = {
"x": Tensor(a, device=on(Replicated())),
"W_up": Tensor(w_up, device=on(Sharded(1))),
"h": h,
"W_down": Tensor(w_down, device=on(Sharded(0))),
"h W_down": partial_out,
"summed": out,
}
widths = [int(t.shape[1]) for t in panels.values()]
fig = make_subplots(
rows=1,
cols=len(panels),
column_widths=widths,
horizontal_spacing=0.03,
subplot_titles=list(panels),
)
for col, t in enumerate(panels.values(), start=1):
fig.add_trace(owner_heatmap(t, show_values=False), row=1, col=col)
device_legend(fig)
fig.update_xaxes(showgrid=False, showticklabels=False)
fig.update_yaxes(showgrid=False, showticklabels=False, autorange="reversed")
fig.update_layout(
title="A column-parallel, row-parallel MLP block on 4 devices",
height=420,
legend=dict(y=-0.04, yanchor="top"),
margin=dict(t=96),
)
viz.show(fig)
The hidden activation stayed sharded, relu ran on each shard independently,
and the only communication was the explicit all-reduce at the end.
Section 2.2 applies this
pattern to every attention and MLP block of SmolLM2.
Two-dimensional meshes
Real deployments combine strategies. A mesh with two axes can split a batch
of requests across dp (data parallelism) while splitting each layer’s
weights across tp. Each mesh axis gets its own placement:
mesh_2d = DeviceMesh((CPU(),) * 4, (2, 2), ("dp", "tp"))
batch = Tensor(
np.ones((4, 6), np.float32),
device=DeviceMapping(mesh_2d, (Sharded(0), Sharded(1))),
)
visualize(batch)
DType.float32 [4, 6] with DeviceMapping(DeviceMesh(dp=2, tp=2), [Sharded(axis=0), Sharded(axis=1)])
┌──────────┬──────────┐ tensor[:, :]
│ device 0 │ device 1 │ ├── dp=0 - tensor[:2, :]
│ [:2, :3] │ [:2, 3:] │ │ ├── tp=0 device 0 - tensor[:2, :3]
├──────────┼──────────┤ │ └── tp=1 device 1 - tensor[:2, 3:]
│ device 2 │ device 3 │ └── dp=1 - tensor[2:, :]
│ [2:, :3] │ [2:, 3:] │ ├── tp=0 device 2 - tensor[2:, :3]
└──────────┴──────────┘ └── tp=1 device 3 - tensor[2:, 3:]
On real hardware, the cost of each collective depends on how much data it moves and how fast the devices are connected, and that decides the best layout for a given model and cluster. The handbook’s parallelism strategies page compares data, tensor, pipeline, and expert parallelism. MAX’s docs cover distributed tensors in more detail.
Exercises
- Shard
xwithSharded(1)instead ofSharded(0)and visualize it. What happens with 3 devices instead of 4, which don’t divide 4 columns evenly? - Swap the order in the MLP block: row-parallel first, then column-parallel. Which collectives does MAX insert now?
- Count the bytes each device sends in the all-reduce of the MLP block.
How does it scale with the batch size and the hidden size? The
bytescolumn in MAX’s reports is this per-device volume for a ring algorithm, so you can check your count against one.
Next
Section 2.2 splits SmolLM2 across devices and checks that its logits don’t change.
2.2 A tensor-parallel model
This section splits your SmolLM2 from Chapter 1 across three devices with tensor parallelism, the strategy servers use to run a model too big or too slow for one GPU. The model code doesn’t change. You change where its weights live, and MAX runs every layer on all three devices and adds the communication. Then you check that the logits match the single-device model’s.
Show the setup 31 lines
import logging
import time
import warnings
import numpy as np
import plotly.graph_objects as go
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.graph import TensorType
from max.nn.common_layers.linear import col_parallel, row_parallel
from max.pipelines import AutoTokenizer
from max.tensor import Tensor, default_device, default_dtype
from max.tensor.sharding import (
DeviceMapping,
DeviceMesh,
Replicated,
auto_reshard,
visualize,
)
from minillm import viz
from minillm.config import MODEL_ID, LlamaConfig
from minillm.model import LlamaForCausalLM, load_model
from minillm.weights import load_checkpoint
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
config = LlamaConfig.from_pretrained()
weights = load_checkpoint()
The plan
Section 2.1 showed the pattern: a column-parallel layer followed by a row-parallel layer needs one all-reduce. Each SmolLM2 layer has two such pairs:
| Block | Column parallel (split outputs) | Row parallel (split inputs) |
|---|---|---|
| Attention | q_proj, k_proj, v_proj | o_proj |
| MLP | gate_proj, up_proj | down_proj |
Splitting q_proj’s outputs three ways gives each device 3 of the 9 query
heads, and splitting k_proj and v_proj gives each device 1 of the 3
key/value heads, the one those queries share under grouped-query attention
(Section 1.5). So attention runs
on each device independently, on its own heads. The embedding, the norms, and
the LM head stay whole on every device (replicated).
max.nn provides col_parallel and row_parallel, which change a Linear
layer’s weight placement on the mesh axis you name:
from __future__ import annotations
from max.nn.common_layers.linear import col_parallel, row_parallel
from minillm.model import LlamaForCausalLM
def tensor_parallel(
model: LlamaForCausalLM, axis: str = "tp"
) -> LlamaForCausalLM:
"""Shards every layer's projections across mesh axis ``axis``."""
for layer in model.model.layers:
attn, mlp = layer.self_attn, layer.mlp
for linear in (
attn.q_proj,
attn.k_proj,
attn.v_proj,
mlp.gate_proj,
mlp.up_proj,
):
col_parallel(linear, axis)
for linear in (attn.o_proj, mlp.down_proj):
row_parallel(linear, axis)
return model
Saved to minillm/parallel.py.
Build the model on a mesh
Build the model inside default_device(mesh) so its weights are created on
the mesh, apply the plan, and load the checkpoint. load_state_dict cuts
each weight into the shards its placement asks for:
mesh = DeviceMesh((CPU(),) * 3, (3,), ("tp",))
with F.lazy(), default_device(mesh), default_dtype(DType.float32):
tp_model = tensor_parallel(LlamaForCausalLM(config))
tp_model.load_state_dict(
{**weights, "lm_head.weight": weights["model.embed_tokens.weight"]}
)
visualize(tp_model.model.layers[0].self_attn.q_proj.weight)
visualize(tp_model.model.layers[0].self_attn.o_proj.weight)
DType.float32 [576, 576] with DeviceMapping(DeviceMesh(tp=3), [Sharded(axis=0)])
┌──────────────┐ tensor[:, :]
│ device 0 │ ├── tp=0 device 0 - tensor[:192, :]
│ [:192, :] │ ├── tp=1 device 1 - tensor[192:384, :]
├──────────────┤ └── tp=2 device 2 - tensor[384:, :]
│ device 1 │
│ [192:384, :] │
├──────────────┤
│ device 2 │
│ [384:, :] │
└──────────────┘
DType.float32 [576, 576] with DeviceMapping(DeviceMesh(tp=3), [Sharded(axis=1)])
┌──────────────┬──────────────┬──────────────┐ tensor[:, :]
│ device 0 │ device 1 │ device 2 │ ├── tp=0 device 0 - tensor[:, :192]
│ [:, :192] │ [:, 192:384] │ [:, 384:] │ ├── tp=1 device 1 - tensor[:, 192:384]
└──────────────┴──────────────┴──────────────┘ └── tp=2 device 2 - tensor[:, 384:]
Read the head assignment straight from the shards: each device’s slice of
q_proj holds whole query heads, and its slice of k_proj holds the
key/value head those queries share. Hover over a box to see its device:
Show the code 43 lines
attn = tp_model.model.layers[0].self_attn
rows = {
"query heads": attn.q_proj.weight,
"key/value heads": attn.k_proj.weight,
}
device_colors = [viz.BLUE, viz.ORANGE, viz.AQUA]
fig = go.Figure()
for y, (name, weight) in enumerate(rows.items()):
head = 0
for device, shard in enumerate(weight.local_shards):
for _ in range(int(shard.shape[0]) // config.head_dim):
width = (
9 / int(weight.shape[0]) * config.head_dim
) # key/value heads span their group
fig.add_trace(
go.Bar(
x=[width],
base=[head * width],
y=[name],
orientation="h",
width=0.6,
marker=dict(
color=device_colors[device],
line=dict(color=viz.SURFACE, width=3),
),
text=f"{'q' if y == 0 else 'kv'}{head}",
textposition="inside",
insidetextanchor="middle",
name=f"device {device}",
legendgroup=str(device),
showlegend=y == 0 and head % 3 == 0,
hovertemplate=f"{name[:-1]} {head} on device {device}<extra></extra>",
)
)
head += 1
fig.update_layout(
title="Layer 0's attention heads on a 3-device mesh",
height=260,
barmode="overlay",
xaxis=dict(visible=False),
yaxis=dict(autorange="reversed", showgrid=False),
)
viz.show(fig)
Linear stores its weight as [out, in], so column parallel splits axis 0
and row parallel splits axis 1. Each device now holds a third of every
projection, which is most of the model:
def elements_on_device_0(model) -> int:
total = 0
for _, p in model.parameters:
shard = p.local_shards[0] if p.is_distributed else p
total += int(np.prod([int(d) for d in shard.shape]))
return total
total = sum(p.num_elements() for _, p in tp_model.parameters)
print(
f"whole model: {total / 1e6:.0f}M values, device 0 holds {elements_on_device_0(tp_model) / 1e6:.0f}M"
)
whole model: 163M values, device 0 holds 92M
The whole-model count is above SmolLM2’s 135M because the model ties its LM head to the embedding, and the count includes that 28M-value weight twice. Broken down by part, for the single-device model and for each device of the mesh:
Show part 43 lines
def part(name: str) -> str:
for key, label in [
("embed_tokens", "embedding"),
("lm_head", "LM head"),
("self_attn", "attention"),
("mlp", "MLP"),
]:
if key in name:
return label
return "norms"
parts = ["embedding", "LM head", "attention", "MLP", "norms"]
columns = ["one device", "mesh device 0", "mesh device 1", "mesh device 2"]
counts = {p: [0] * 4 for p in parts}
for name, param in tp_model.parameters:
counts[part(name)][0] += param.num_elements()
shards = param.local_shards if param.is_distributed else [param] * 3
for device, shard in enumerate(shards):
counts[part(name)][device + 1] += int(
np.prod([int(d) for d in shard.shape])
)
fig = go.Figure(
[
go.Bar(
x=columns,
y=np.array(values) / 1e6,
name=p,
marker_color=color,
hovertemplate=p + ": %{y:.1f}M values<extra>%{x}</extra>",
)
for (p, values), color in zip(counts.items(), viz.SERIES)
]
)
fig.update_layout(
title="Parameter values per device",
barmode="stack",
height=420,
yaxis_title="millions of values",
legend_traceorder="normal",
)
viz.show(fig)
The replicated embedding and LM head (28M values each) keep device 0 well above a third. Large models also split those across the vocabulary.
Run it
The input token IDs are replicated: every device sees the whole prompt. Run
the model with auto_reshard(mode="warn") to see each collective MAX
inserts:
tokenizer = AutoTokenizer(MODEL_ID)
ids = tokenizer(["The capital of France is"])["input_ids"][0]
tokens = Tensor(np.array([ids]), device=DeviceMapping(mesh, (Replicated(),)))
start = time.perf_counter()
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
with auto_reshard(mode="warn"):
tp_logits = tp_model(tokens).to_numpy()
reshards = [str(w.message) for w in caught if "reshards" in str(w.message)]
print(
f"forward pass in {time.perf_counter() - start:.0f} s, {len(reshards)} collectives inserted"
)
assert len(reshards) == 2 * config.num_hidden_layers
print(reshards[0])
forward pass in 108 s, 60 collectives inserted
add(lhs, rhs) reshards rhs
lhs : float32[1, 5, 576]@mesh(tp=3){"tp":R}
rhs : float32[1, 5, 576]@mesh(tp=3){"tp":P}
region : auto_reshard(DEFAULT_TRANSITIONS, mode="warn")
decided per mesh axis
axis lhs rhs -> out collective bytes status
"tp" lhs{R->P} rhs{P} -> P keep_one_copy(lhs) 60 not allowed
lhs{R->S1} rhs{P->S1} -> S1 local_slice(lhs), reduce_scatter(rhs) 7740 not allowed
lhs{R->S2} rhs{P->S2} -> S2 local_slice(lhs), reduce_scatter(rhs) 7740 not allowed
lhs{R} rhs{P->R} -> R allreduce(rhs) 15360 chosen, reshards rhs
result lhs{"tp":R} rhs{"tp":R} -> {"tp":R}
rhs = transfer_to(rhs, DeviceMapping(mesh, (R,)))
with auto_reshard(mode="silent"):
with auto_reshard(ALL_TRANSITIONS, mode="silent"): # also allows the cheaper rows above
at minillm/model.py:43 in forward()
That makes 60 collectives for 30 layers, two all-reduces per layer: one where
the output of attention’s row-parallel o_proj meets the residual addition,
and one where the MLP’s down_proj output does. Each residual addition needs
the full value, so a column-then-row layout needs both. Everything else,
including attention itself, ran on each device without communication. The
forward pass is slow because the simulated mesh runs eagerly, one operation at
a time, on one CPU; its time says nothing about real hardware.
Check it
The sharded model must produce the same logits as the single-device model from Section 1.6:
single = load_model(config, weights, CPU()).compile(
TensorType(DType.int64, ("batch", "seq_len"), device=CPU())
)
reference = single(Tensor(np.array([ids]), device=CPU())).to_numpy()
print(f"max difference {np.abs(tp_logits - reference).max():.1e}")
print("next token:", repr(tokenizer.decode([int(tp_logits[0, -1].argmax())])))
assert (
np.abs(tp_logits - reference).max() < 1e-3
and tp_logits[0, -1].argmax() == reference[0, -1].argmax()
)
max difference 7.0e-05
next token: ' Paris'
The logits agree to within 1e-4, the size of a change in summation order:
the all-reduce adds three partial sums where one device computed one full
sum. On real hardware, replace the mesh’s CPU() entries with
Accelerator(0), Accelerator(1), and Accelerator(2), and nothing else
changes. MAX’s own models shard their layers with the same col_parallel
and row_parallel helpers; Gemma 4’s attention, for example, uses the
column-then-row layout you built here. max serve runs such a model across
GPUs: pass --devices gpu:0,1,2 to use three.
Simulation answers is it correct? cheaply. Whether a layout is fast depends on the hardware: an all-reduce over NVLink costs microseconds, over PCIe or a network much more, and for a model this small on real GPUs the communication would cost more than it saves. The handbook’s distributed inference and parallelism strategies pages discuss when each layout pays off.
Exercises
- Run on a 9-device mesh, one query head per device. What happens to the key/value heads, which only split three ways?
- Shard the embedding by vocabulary (rows) and the LM head to match. Which collectives does MAX add, and how much memory does device 0 save?
- Add a
dpaxis for a 2 × 3 mesh and run a batch of two prompts with the batch split alongdp. Check both rows against the single-device model.
Next
Chapter 3 goes below the operations: you write a kernel in Mojo and call it from your model.
3. Custom ops in Mojo
MAX’s operations are written in Mojo, and you can write your own the same
way. This chapter writes a kernel in a notebook cell, calls it from your
model, tests and times it, and then looks underneath compile at the graph
API that MAX builds models with.
| Section | You learn |
|---|---|
| 3.1 Your first custom op | Writing a fused SwiGLU kernel in a %%mojo cell, calling it with F.custom, and measuring whether it helps. |
| 3.2 Under the hood: the graph API | What compile builds, max.graph, and MAX’s weight loaders. |
To go deeper into GPU programming in Mojo, work through the Mojo GPU puzzles. The handbook’s kernel optimization chapter explains why fused kernels matter.
3.1 Your first custom op
MAX’s operations, from F.silu to flash attention, run kernels written in Mojo.
You can write your own the same way, compile it from a notebook cell, and call
it from Python like any other MAX operation. This section writes a fused SwiGLU
kernel, the silu(gate) * up inside SmolLM2’s MLP from
Section 1.4, tests it,
runs it inside your full model, and measures whether it helps.
Show the setup 25 lines
import logging
import time
import warnings
from pathlib import Path
import numpy as np
import plotly.graph_objects as go
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.graph import TensorType
from max.pipelines import AutoTokenizer
from max.tensor import Tensor, default_device, default_dtype
from plotly.subplots import make_subplots
from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.layers import MLP
from minillm.model import load_model
from minillm.weights import load_checkpoint
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
Write the kernel
import mojo.notebook adds a %%mojo cell magic to Jupyter. A cell that
starts with %%mojo package -o kernels.mojoc compiles its Mojo code into a
package that MAX can load.
import mojo.notebook
The simplest kind of custom op is elementwise: each output element depends
only on the input elements at the same position. You write a struct with one
static elementwise method that maps input vectors to an output vector, and
MAX handles everything else: splitting the work across threads, loading and
storing memory, and choosing CPU or GPU code.
@extensibility.register("swiglu")names the op; Python calls it by this name.ElementwiseBinaryOpdeclares the shape of the method: two inputs, one output, all of one dtype.SIMD[dtype, width]is a vector ofwidthvalues. MAX picks the width that fills the hardware’s vector registers, so one call processes several elements at once.comptime assertruns at compile time and rejects integer dtypes, for whichexpis undefined.
SiLU is \(x \cdot \sigma(x) = x / (1 + e^{-x})\):
%%mojo package -o kernels.mojoc
import extensibility
from extensibility import ElementwiseBinaryOp
from std.math import exp
@extensibility.register("swiglu")
struct SwiGLU(ElementwiseBinaryOp):
@staticmethod
def elementwise[
dtype: DType, width: SIMDLength
](gate: SIMD[dtype, width], up: SIMD[dtype, width]) -> SIMD[dtype, width]:
comptime assert dtype.is_floating_point(), "SwiGLU needs floats"
return gate / (1 + exp(-gate)) * up
Call it from Python
F.custom calls a custom op by name on eager tensors. It needs the device to
run on, the inputs, the type of each output (here, the same as gate’s), and
the package to load the kernel from:
KERNELS = Path("kernels.mojoc").resolve()
def swiglu(gate: Tensor, up: Tensor) -> Tensor:
return F.custom(
"swiglu",
device=gate.device,
values=[gate, up],
out_types=[gate.type],
custom_extensions=KERNELS,
)[0]
rng = np.random.default_rng(0)
g = rng.standard_normal((4, 8)).astype(np.float32)
u = rng.standard_normal((4, 8)).astype(np.float32)
result = swiglu(Tensor(g, device=DEVICE), Tensor(u, device=DEVICE))
print(result.to_numpy()[0, :4])
print(F.silu(Tensor(g, device=DEVICE)).to_numpy()[0, :4] * u[0, :4])
[-0.01063813 -0.03336792 0.09002377 0.01961606]
[-0.01063813 -0.03336792 0.09002377 0.01961606]
Test it
A kernel needs the same tests as any other code, against a reference you
trust. Check shapes that don’t divide evenly into vectors, a single element,
and inputs extreme enough to overflow exp:
def reference(g: np.ndarray, u: np.ndarray) -> np.ndarray:
return g / (1 + np.exp(-g.astype(np.float64))) * u
cases = {
"1 element": (1,),
"odd length": (1, 7, 1001),
"model size": (2, 32, 1536),
}
for name, shape in cases.items():
g = rng.standard_normal(shape).astype(np.float32) * 4
u = rng.standard_normal(shape).astype(np.float32)
out = swiglu(Tensor(g, device=DEVICE), Tensor(u, device=DEVICE)).to_numpy()
print(f"{name:12} max error {np.abs(out - reference(g, u)).max():.1e}")
assert np.allclose(out, reference(g, u), rtol=1e-4, atol=1e-5)
extremes = np.array([-100.0, -20.0, 0.0, 20.0, 100.0], np.float32)
out = swiglu(
Tensor(extremes, device=DEVICE),
Tensor(np.ones(5, np.float32), device=DEVICE),
)
print("extremes:", out.to_numpy())
assert np.isfinite(out.to_numpy()).all()
1 element max error 1.0e-08
odd length max error 1.8e-06
model size max error 3.0e-06
extremes: [-0.0000000e+00 -4.1223068e-08 0.0000000e+00 2.0000000e+01
1.0000000e+02]
Sweep the input to see the whole function and its error. With up set to
ones, the op is SiLU itself:
sweep = np.linspace(-12, 12, 481, dtype=np.float32)
ones = np.ones_like(sweep)
ours = swiglu(
Tensor(sweep, device=DEVICE), Tensor(ones, device=DEVICE)
).to_numpy()
error = np.abs(ours - reference(sweep, ones))
Show the code 34 lines
fig = make_subplots(
rows=2,
cols=1,
shared_xaxes=True,
vertical_spacing=0.08,
row_heights=[0.6, 0.4],
)
fig.add_trace(
go.Scatter(
x=sweep,
y=ours,
name="your kernel",
line_color=viz.BLUE,
hovertemplate="silu(%{x:.2f}) = %{y:.4f}<extra></extra>",
),
row=1,
col=1,
)
fig.add_trace(
go.Scatter(
x=sweep,
y=np.maximum(error, 1e-12),
name="error against float64 NumPy",
line_color=viz.ORANGE,
hovertemplate="%{x:.2f}: error %{y:.1e}<extra></extra>",
),
row=2,
col=1,
)
fig.update_yaxes(title_text="output", row=1, col=1)
fig.update_yaxes(title_text="absolute error", type="log", dtick=1, row=2, col=1)
fig.update_xaxes(title_text="gate", row=2, col=1)
fig.update_layout(title="The swiglu op with up = 1", height=520)
viz.show(fig)
The error stays at the scale of float32 rounding, about \(10^{-7}\) of the
output or less, across the whole range.
At \(-100\), \(e^{100}\) overflows float32 to infinity, and \(-100 / \infty\) is
\(-0\), the correct limit. A different but equivalent formula, such as
gate * exp(gate) / (1 + exp(gate)), would compute \(\infty / \infty\) and
return NaN at \(+100\). Tests with extreme inputs catch this kind of bug.
Use it in your model
FusedMLP is the MLP from
Section 1.4 with the
custom op in place of F.silu(...) *:
class FusedMLP(MLP):
def forward(self, x: Tensor) -> Tensor:
return self.down_proj(swiglu(self.gate_proj(x), self.up_proj(x)))
config = LlamaConfig.from_pretrained()
weights = load_checkpoint()
model = load_model(config, weights, DEVICE)
for layer in model.model.layers:
with F.lazy(), default_device(DEVICE), default_dtype(DType.float32):
fused = FusedMLP(config)
fused.load_state_dict(dict(layer.mlp.parameters))
layer.mlp = fused
Compile the model as before, with one addition: custom_extensions tells
the compiler where to find the kernel. Then compare the logits with the
model from Section 1.6:
token_type = TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
fused_model = model.compile(token_type, custom_extensions=[KERNELS])
plain_model = load_model(config, weights, DEVICE).compile(token_type)
tokenizer = AutoTokenizer(MODEL_ID)
ids = Tensor(
np.array([tokenizer(["The capital of France is"])["input_ids"][0]]),
device=DEVICE,
)
fused_logits = fused_model(ids).to_numpy()
plain_logits = plain_model(ids).to_numpy()
print(f"max difference {np.abs(fused_logits - plain_logits).max():.1e}")
print(
"next token:", repr(tokenizer.decode([int(fused_logits[0, -1].argmax())]))
)
assert (
np.abs(fused_logits - plain_logits).max() < 1e-2
and fused_logits[0, -1].argmax() == plain_logits[0, -1].argmax()
)
max difference 4.9e-05
next token: ' Paris'
Is it faster?
Time a compiled MLP block with and without the custom op, on a batch of 512 positions:
Show the code 28 lines
with default_device(DEVICE), default_dtype(DType.float32):
plain_mlp = MLP(config)
fused_mlp = FusedMLP(config)
fused_mlp.load_state_dict(dict(plain_mlp.parameters))
block_type = TensorType(
DType.float32, ("batch", "seq_len", config.hidden_size), device=DEVICE
)
runs = {
"MAX ops": plain_mlp.compile(block_type),
"custom op": fused_mlp.compile(block_type, custom_extensions=[KERNELS]),
}
x = Tensor(
rng.standard_normal((1, 512, config.hidden_size)).astype(np.float32),
device=DEVICE,
)
for run in runs.values():
for _ in range(3):
run(x).to_numpy()
times = {name: [] for name in runs}
# Alternate the two, so a change in clock speed or load hits both alike.
for _ in range(30):
for name, run in runs.items():
start = time.perf_counter()
run(x).to_numpy()
times[name].append((time.perf_counter() - start) * 1000)
for name, ms in times.items():
print(f"{name:12} median {np.median(ms):.2f} ms")
MAX ops median 3.58 ms
custom op median 3.41 ms
Show the code 21 lines
fig = go.Figure(
[
go.Box(
y=ms,
name=name,
marker_color=color,
boxpoints="all",
jitter=0.4,
pointpos=0,
hovertemplate="%{y:.2f} ms<extra>" + name + "</extra>",
)
for (name, ms), color in zip(times.items(), [viz.BLUE, viz.ORANGE])
]
)
fig.update_layout(
title="A compiled MLP block on 512 positions, 30 runs each",
height=400,
yaxis=dict(title="ms", rangemode="tozero"),
showlegend=False,
)
viz.show(fig)
In the runs behind this page, on an Apple-silicon Mac with a Metal GPU, the two
boxes overlap almost completely, the medians land within about 10% of each
other, and which one is faster changes from run to run. Treat the output above
as one example: your hardware will give different numbers, so measure on the
hardware you serve on. There is
little to gain because the graph compiler already fuses elementwise operations
into their neighbors: the kernel trace in
Section 4.4 shows silu and the
multiply applied in the epilogue of a matmul, so the plain version never writes
the activation to memory either. Most of the time goes to the three matrix
multiplications.
Write a custom op when MAX doesn’t have the operation you need, or when a better algorithm needs control that a chain of separate operations can’t express. Flash attention (Section 1.5) is the classic case: it never materializes the attention matrix, which no combination of separate operations can avoid. Elementwise ops written as above stay fusible, so the compiler can still merge them with their neighbors, and the same kernel runs on a GPU with no changes.
Kernels that need control over threads and memory, such as a matmul or
attention, implement an execute method instead, written with Mojo’s GPU
programming model. MAX’s guide to
building custom ops shows
the pattern, and the
Mojo GPU puzzles teach GPU
programming in Mojo from first principles. The handbook’s
kernel optimization tools
page compares Mojo with CUDA and Triton, and
GPU architecture fundamentals
explains what a kernel has to work with.
Exercises
- Write a
gelu_tanhcustom op as anElementwiseUnaryOp(unary elementwise ops take one input) and test it againstF.geluwithapproximate="tanh". - Add a
float16case to the tests. How large is the error, and why? - Fuse more: write an op that takes
gate,up, and a scale, and returnssilu(gate) * up * scale. Does the timing change?
Next
Section 3.2 looks underneath
compile: the graph MAX builds from your module, and the graph API you can use
to build one directly.
3.2 Under the hood: the graph API
Since Section 1.2 you
have called compile on modules and gotten a fast callable back. This section
opens that box. compile traces your module into a graph, a complete
description of the computation, and MAX’s compiler turns the graph into
optimized code. You can also build graphs directly with max.graph, which is
how many of MAX’s serving pipelines build their models, and load checkpoints
with MAX’s production weight loader.
Show the setup 25 lines
import logging
import re
from pathlib import Path
import numpy as np
import plotly.graph_objects as go
from huggingface_hub import hf_hub_download
from max.driver import CPU, Buffer
from max.dtype import DType
from max.engine import InferenceSession
from max.graph import DeviceRef, Graph, TensorType, Weight, ops
from max.graph.weights import load_weights
from max.tensor import Tensor, default_device, default_dtype
from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.layers import MLP
from minillm.weights import load_checkpoint, select
logging.getLogger("max").setLevel(logging.ERROR)
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
config = LlamaConfig.from_pretrained()
weights = load_checkpoint()
What compile builds
trace runs the first half of compile: it calls forward with symbolic
inputs and records every operation into a graph, without compiling it. Here is
SmolLM2’s MLP from
Section 1.4:
with default_device(DEVICE), default_dtype(DType.float32):
mlp = MLP(config)
mlp.load_state_dict(select(weights, "model.layers.0.mlp."))
staged = mlp.trace(
TensorType(
DType.float32,
("seq_len", config.hidden_size),
device=DeviceRef.from_device(DEVICE),
)
)
print("\n".join(str(staged.graph).splitlines()[:12]))
mo.graph @MLP<seq_len>(%arg0: !mo.tensor<[seq_len, 576], f32, gpu:0>) -> !mo.tensor<[seq_len, 576], f32, gpu:0> attributes {_kernel_library_paths = [], argument_names = ["input0"], result_names = ["output0"]} {
%0 = mo.chain.create()
%1 = mo.constant.external [name = "gate_proj.weight", align = 1, device = #M.device_ref<"cpu", 0>] : !mo.tensor<[1536, 576], f32>
%result, %outChain = mo.transfer[%0] %1 : !mo.tensor<[1536, 576], f32> to <"gpu", 0>
%2 = mo.constant <#M.dense_array<1, 0> : tensor<2xsi64>> : !mo.tensor<[2], si64>
%3 = mo.transpose(%result, %2) : (!mo.tensor<[1536, 576], f32, gpu:0>, !mo.tensor<[2], si64>) -> !mo.tensor<[576, 1536], f32, gpu:0>
%4 = rmo.matmul(%arg0, %3) : (!mo.tensor<[seq_len, 576], f32, gpu:0>, !mo.tensor<[576, 1536], f32, gpu:0>) -> !mo.tensor<[seq_len, 1536], f32, gpu:0>
%5 = mo.constant <#M.dense_array<0.000000e+00> : tensor<f32>> : !mo.tensor<[], f32, gpu:0>
%6 = rmo.add(%4, %5) : (!mo.tensor<[seq_len, 1536], f32, gpu:0>, !mo.tensor<[], f32, gpu:0>) -> !mo.tensor<[seq_len, 1536], f32, gpu:0>
%7 = mo.silu(%6) : !mo.tensor<[seq_len, 1536], f32, gpu:0>
%8 = mo.constant.external [name = "up_proj.weight", align = 1, device = #M.device_ref<"cpu", 0>] : !mo.tensor<[1536, 576], f32>
%result_0, %outChain_1 = mo.transfer[%0] %8 : !mo.tensor<[1536, 576], f32> to <"gpu", 0>
The text is a dataflow graph: each line defines one value (%4) from
earlier ones. Parse it and draw it, with each operation placed one column
after the latest value it reads. Hover over a node to see its line of IR:
Show kind 85 lines
LINE = re.compile(
r"^\s*(%\w+)(?:, %\w+)* = ([\w.]+)(.*)$"
) # some ops have two results
nodes, edges = {"%arg0": ("input", "input0: [seq_len, 576]")}, []
for line in str(staged.graph).splitlines():
match = LINE.match(line)
if not match or match.group(2) in ("mo.chain.create", "mo.constant"):
continue # skip bookkeeping and small constants
value, op, rest = match.groups()
weight = re.search(r'name = "([^"]+)"', rest)
label = weight.group(1) if weight else op.split(".")[-1]
nodes[value] = (label, line.strip())
edges += [(src, value) for src in re.findall(r"%\w+", rest) if src in nodes]
output = re.search(r"mo.output (%\w+)", str(staged.graph)).group(1)
nodes["output"] = ("output", "output0: [seq_len, 576]")
edges.append((output, "output"))
depth = {}
for value in nodes: # lines are in dataflow order
depth[value] = max(
(depth[src] + 1 for src, dst in edges if dst == value), default=0
)
column = {}
position = {}
for value in nodes:
row = column.get(depth[value], 0)
column[depth[value]] = row + 1
position[value] = (depth[value], -row)
def kind(label: str) -> str:
if label in ("input", "output"):
return "input or output"
if label.endswith(".weight"):
return "weight"
if label == "transfer":
return "copy to device"
return "matmul" if label == "matmul" else "other operation"
colors = {
"input or output": viz.VIOLET,
"weight": viz.ORANGE,
"copy to device": viz.YELLOW,
"matmul": viz.BLUE,
"other operation": viz.AQUA,
}
edge_x, edge_y = [], []
for src, dst in edges:
edge_x += [position[src][0], position[dst][0], None]
edge_y += [position[src][1], position[dst][1], None]
fig = go.Figure(
go.Scatter(
x=edge_x,
y=edge_y,
mode="lines",
line=dict(color=viz.GRID, width=2),
hoverinfo="skip",
showlegend=False,
)
)
for name, color in colors.items():
members = [v for v in nodes if kind(nodes[v][0]) == name]
fig.add_trace(
go.Scatter(
x=[position[v][0] for v in members],
y=[position[v][1] for v in members],
mode="markers+text",
name=name,
text=[nodes[v][0] for v in members],
textposition="top center",
marker=dict(
size=18, color=color, line=dict(color=viz.SURFACE, width=2)
),
customdata=[nodes[v][1][:110] for v in members],
hovertemplate="%{customdata}<extra></extra>",
)
)
fig.update_layout(
title="The MLP's graph, as compile sees it",
height=480,
xaxis=dict(visible=False, range=[-0.8, max(depth.values()) + 0.6]),
yaxis=dict(visible=False, range=[-max(column.values()) + 0.5, 0.6]),
)
viz.show(fig)
The graph is MAX’s intermediate representation (in MLIR). A few things to read in it:
- The input type is
[seq_len, 576]:seq_lenstays symbolic, which is why one compiled graph serves every sequence length. - Each weight is a
mo.constant.externalwith a name such asgate_proj.weight. The graph refers to weights by name and gets their values when it is loaded, so the same graph works with any checkpoint. - On a GPU, each weight also passes through a
mo.transferthat copies it from host memory to the device (yellow in the diagram); on a CPU there is nothing to copy. - Each projection’s
rmo.matmulis followed by anrmo.addof a constant0.0. MAX’sLinearbuilt withbias=Falsestores its bias as the number 0 and still adds it. It costs nothing at run time: the kernel trace in Section 4.4 shows no separate add. - Every line is one operation (
rmo.matmul,mo.silu, …) with fully known types. That is what lets the compiler fuse operations, plan memory for the whole computation, and generate code for the target device ahead of time.
Build a graph yourself
max.graph builds graphs directly. A Graph takes a name, the input types,
and a forward function that combines max.graph.ops operations. Weights
are declared as Weight objects with a name, dtype, shape, and device:
H, I = config.hidden_size, config.intermediate_size
def mlp_forward(x):
def weight(name: str, shape: list[int]) -> Weight:
return Weight(
name, DType.float32, shape, device=DeviceRef.from_device(DEVICE)
)
gate = x @ weight("gate_proj.weight", [I, H]).T
up = x @ weight("up_proj.weight", [I, H]).T
return (ops.silu(gate) * up) @ weight("down_proj.weight", [H, I]).T
graph = Graph(
"mlp",
forward=mlp_forward,
input_types=[
TensorType(
DType.float32, ("seq_len", H), device=DeviceRef.from_device(DEVICE)
)
],
)
An InferenceSession compiles graphs for a set of devices. load compiles
this one and binds each named weight from a registry; execute runs it on
Buffers, the driver’s plain device memory:
session = InferenceSession(devices=[DEVICE])
compiled_mlp = session.load(
graph, weights_registry=select(weights, "model.layers.0.mlp.")
)
x = np.random.default_rng(0).standard_normal((5, H)).astype(np.float32)
(out,) = compiled_mlp.execute(Buffer.from_numpy(x).to(DEVICE))
expected = mlp(Tensor(x, device=DEVICE)).to_numpy()
scale = np.abs(expected).max()
relative = np.abs(out.to_numpy() - expected).max() / scale
print(
f"max difference from your module, relative to the largest output: {relative:.1e}"
)
assert relative < 1e-5
max difference from your module, relative to the largest output: 4.8e-07
The hand-built graph and your module agree because Module.compile builds
this same graph for you, loads it in a session shared by the whole process,
and wraps the result so it takes and returns Tensors.
Custom ops work in graphs too, through ops.custom. Compile the kernel from
Section 3.1 again and use it in
place of silu * up:
import mojo.notebook
Show SwiGLU 14 lines
%%mojo package -o kernels.mojoc
import extensibility
from extensibility import ElementwiseBinaryOp
from std.math import exp
@extensibility.register("swiglu")
struct SwiGLU(ElementwiseBinaryOp):
@staticmethod
def elementwise[
dtype: DType, width: SIMDLength
](gate: SIMD[dtype, width], up: SIMD[dtype, width]) -> SIMD[dtype, width]:
comptime assert dtype.is_floating_point(), "SwiGLU needs floats"
return gate / (1 + exp(-gate)) * up
Show fused_forward 34 lines
def fused_forward(x):
def weight(name: str, shape: list[int]) -> Weight:
return Weight(
name, DType.float32, shape, device=DeviceRef.from_device(DEVICE)
)
gate = x @ weight("gate_proj.weight", [I, H]).T
up = x @ weight("up_proj.weight", [I, H]).T
(h,) = ops.custom(
"swiglu",
device=DeviceRef.from_device(DEVICE),
values=[gate, up],
out_types=[gate.type],
)
return h.tensor @ weight("down_proj.weight", [H, I]).T
fused_graph = Graph(
"fused_mlp",
forward=fused_forward,
input_types=[
TensorType(
DType.float32, ("seq_len", H), device=DeviceRef.from_device(DEVICE)
)
],
custom_extensions=[Path("kernels.mojoc")],
)
fused_mlp = session.load(
fused_graph, weights_registry=select(weights, "model.layers.0.mlp.")
)
(fused_out,) = fused_mlp.execute(Buffer.from_numpy(x).to(DEVICE))
relative = np.abs(fused_out.to_numpy() - expected).max() / scale
print(f"max difference, relative to the largest output: {relative:.1e}")
assert relative < 1e-5
max difference, relative to the largest output: 4.8e-07
Production weight loading
The loader from
Section 1.3 reads one
safetensors file into memory as float32. MAX’s own loader,
max.graph.weights.load_weights, reads safetensors (including checkpoints split
across many files) and GGUF. It maps files into memory instead of reading them,
so a tensor is read only when it’s used, and it keeps each tensor’s original
dtype and, for GGUF, its quantization encoding. Tensors are reached by attribute
path:
path = Path(hf_hub_download(MODEL_ID, "model.safetensors"))
checkpoint = load_weights([path])
gate = checkpoint.model.layers[0].mlp.gate_proj.weight
print(gate.name)
weight = gate.allocate()
print(weight.shape, weight.dtype)
model.layers.0.mlp.gate_proj.weight
[Dim(1536), Dim(576)] DType.bfloat16
allocate returns a Weight like the ones in your hand-built graph, with
the checkpoint’s name, shape, and dtype filled in. A serving pipeline builds
its graph from these, so the graph matches the checkpoint exactly, and the
loader supplies the data when the graph is loaded.
Why it matters
Because the whole computation is known before it runs, MAX can:
- fuse operations and plan memory across the entire model instead of one operation at a time;
- compile once, ahead of time, and cache the result (the reason
max servestarts faster the second time); - on GPUs, record a decode step’s kernel launches and replay them with almost no CPU overhead (device graph capture);
- run the same graph on CPUs and on NVIDIA, AMD, and Apple GPUs.
max.nn modules with compile give you all of this while you write ordinary
Python. Use max.graph directly when you need control over exactly what goes
into a graph, as MAX’s serving pipelines do. MAX’s docs describe the
graph API in full; the handbook’s
kernel optimization
page explains why whole-graph compilation and fusion matter for LLMs.
Exercises
- Print the full traced graph of one
DecoderLayerfrom Section 1.6. Find the flash attention call and the residual additions. - Build the MLP graph with a static input shape,
[5, 576], instead ofseq_len. What happens when you execute it with 6 positions? - Load layer 0’s MLP weights through
load_weightsinstead ofload_checkpoint. What has to change for thefloat32graph?
Next
Chapter 4 turns to correctness and speed: testing layers against references, verifying a whole model, and benchmarking and profiling.
4. Testing and benchmarking
A model has to be correct before it is worth making fast. This chapter applies the tools MAX uses to test and measure its own models to yours.
| Section | You learn |
|---|---|
| 4.1 Testing layers against references | Testing a decoder layer on many shapes against Hugging Face’s PyTorch Llama, with the pattern MAX’s layer benchmarks use. |
| 4.2 Verifying a model’s logits | Comparing a whole model’s logits with a reference, using the metrics MAX’s verification tools use. |
| 4.3 Benchmarking kernels and layers | A benchmark harness, and why decode is limited by memory and prefill by arithmetic. |
| 4.4 Profiling | Where the time goes, from a whole generation run down to the kernels and the parts of your model. |
Related handbook pages: LLM inference metrics and performance benchmarks.
4.1 Testing layers against references
Chapter 1 checked each layer once, on one
input. Real tests check many shapes against an independent reference, with
metrics that tell a rounding difference from a bug. This section builds the
pattern MAX uses for its own layers, in
max/python/layer_benchmarks,
and applies it to your decoder layer, with Hugging Face’s PyTorch Llama as the
reference.
Show the setup 29 lines
import logging
import warnings
import numpy as np
import plotly.graph_objects as go
import torch
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.graph import TensorType
from max.tensor import Tensor, default_device, default_dtype
from transformers import AutoConfig
from transformers.models.llama.modeling_llama import (
LlamaDecoderLayer,
LlamaRotaryEmbedding,
)
from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.model import DecoderLayer
from minillm.weights import load_checkpoint, select
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
config = LlamaConfig.from_pretrained()
layer0 = select(load_checkpoint(), "model.layers.0.")
How to compare outputs
One number is not enough. MAX’s test harness reports three:
- The maximum absolute error catches any single element that is far off.
- The mean absolute error is the typical error, which tells a few outliers from a systematic drift.
- The cosine distance, \(1 - \cos\theta\) between the flattened outputs, is near 0 when the outputs point the same way even if their scale differs slightly, and large for a structural bug such as a wrong head layout.
A test passes when the outputs are allclose, within an absolute and a
relative tolerance, and the cosine distance is below a threshold. The defaults
below, 1e-3 for both tolerances and 1e-5 for the cosine distance, are
tighter than MAX’s (1e-2 and 1e-3) because every test here runs in
float32; MAX’s bfloat16 layer tests loosen them further.
Show CorrectnessResult, compare_outputs 48 lines
from __future__ import annotations
from dataclasses import dataclass
import numpy as np
@dataclass
class CorrectnessResult:
"""How far an output is from its reference, as MAX's layer tests report it."""
label: str
mae: float
max_abs_err: float
cos_dist: float
passed: bool
def __str__(self) -> str:
status = "pass" if self.passed else "FAIL"
return (
f"{self.label:16} mae {self.mae:.1e} max {self.max_abs_err:.1e} "
f"cos dist {self.cos_dist:.1e} {status}"
)
def compare_outputs(
actual: np.ndarray,
expected: np.ndarray,
label: str = "",
atol: float = 1e-3,
rtol: float = 1e-3,
cos_threshold: float = 1e-5,
) -> CorrectnessResult:
"""Compares ``actual`` with ``expected``; passes when they are allclose
and their cosine distance is below ``cos_threshold``."""
a = actual.astype(np.float64).ravel()
e = expected.astype(np.float64).ravel()
cos_dist = 1.0 - float(
a @ e / (np.linalg.norm(a) * np.linalg.norm(e) + 1e-12)
)
allclose = bool(np.allclose(actual, expected, atol=atol, rtol=rtol))
return CorrectnessResult(
label=label,
mae=float(np.mean(np.abs(a - e))),
max_abs_err=float(np.max(np.abs(a - e))),
cos_dist=cos_dist,
passed=allclose and cos_dist < cos_threshold,
)
Saved to minillm/testing.py.
The reference
Hugging Face’s LlamaDecoderLayer is an independent implementation of the
same layer, in PyTorch. Its parameter names match yours, so the same
checkpoint entries load into both. "eager" selects its plain attention
code instead of a fused kernel, so the reference stays simple.
hf_config = AutoConfig.from_pretrained(MODEL_ID, attn_implementation="eager")
reference = LlamaDecoderLayer(hf_config, layer_idx=0).float().eval()
reference.load_state_dict(
{name: torch.from_numpy(w.copy()) for name, w in layer0.items()}
)
rotary = LlamaRotaryEmbedding(hf_config)
def reference_layer(x: np.ndarray) -> np.ndarray:
batch, seq, _ = x.shape
hidden = torch.from_numpy(x)
positions = torch.arange(seq).expand(batch, seq)
mask = torch.full((seq, seq), float("-inf")).triu(1)[None, None]
with torch.no_grad():
out = reference(
hidden,
attention_mask=mask,
position_ids=positions,
position_embeddings=rotary(hidden, positions),
)
return out.numpy()
Compile once, test many shapes
MAX’s harness separates static parameters, which fix the graph (here, the layer’s sizes), from dynamic ones, which vary per run (batch size and sequence length). Compile once with symbolic dimensions, then run every shape through the same compiled layer. The shapes cover the edges: a single token (a decode step), odd lengths, a batch, and a long prefill.
Show compiled_layer, check 34 lines
def compiled_layer(layer_class=DecoderLayer, weights=layer0, cfg=config):
with F.lazy(), default_device(DEVICE), default_dtype(DType.float32):
layer = layer_class(cfg)
layer.load_state_dict(weights)
return layer.compile(
TensorType(
DType.float32, ("batch", "seq_len", cfg.hidden_size), device=DEVICE
)
)
SHAPES = [(1, 1), (1, 7), (2, 33), (1, 512)]
def check(run, shapes=SHAPES) -> list:
rng = np.random.default_rng(0)
results = []
for batch, seq in shapes:
x = rng.standard_normal((batch, seq, config.hidden_size)).astype(
np.float32
)
actual = run(Tensor(x, device=DEVICE)).to_numpy()
results.append(
compare_outputs(
actual, reference_layer(x), label=f"batch {batch}, seq {seq}"
)
)
return results
correct = check(compiled_layer())
for result in correct:
print(result)
assert all(r.passed for r in correct)
batch 1, seq 1 mae 2.7e-07 max 2.9e-06 cos dist 2.4e-14 pass
batch 1, seq 7 mae 6.5e-07 max 1.7e-05 cos dist 1.9e-13 pass
batch 2, seq 33 mae 7.0e-07 max 2.1e-05 cos dist 1.9e-13 pass
batch 1, seq 512 mae 7.1e-07 max 3.8e-05 cos dist 2.0e-13 pass
Every shape passes, with maximum errors of 4e-5 or less and cosine
distances near zero.
A test must be able to fail
A test that never fails proves nothing, so check that this one catches a real
bug. A plausible one is a config read wrong, with the base of RoPE
(Section 1.4) at Llama 2’s
10000 instead of SmolLM2’s 100000:
from dataclasses import replace
buggy = compiled_layer(cfg=replace(config, rope_theta=10000.0))
buggy_results = check(buggy)
for result in buggy_results:
print(result)
assert not all(r.passed for r in buggy_results)
batch 1, seq 1 mae 2.7e-07 max 2.9e-06 cos dist 2.4e-14 pass
batch 1, seq 7 mae 4.9e-04 max 6.6e-03 cos dist 1.2e-07 FAIL
batch 2, seq 33 mae 2.2e-03 max 1.3e-01 cos dist 2.3e-06 FAIL
batch 1, seq 512 mae 5.0e-03 max 2.3e-01 cos dist 1.1e-05 FAIL
Show the code 45 lines
ATOL, COS_THRESHOLD = 1e-3, 1e-5
fig = go.Figure(
go.Scatter(
x=[1e-7, ATOL, ATOL, 1e-7],
y=[1e-15, 1e-15, COS_THRESHOLD, COS_THRESHOLD],
fill="toself",
fillcolor="rgba(27,175,122,0.15)",
line=dict(color=viz.AQUA, width=1, dash="dot"),
name="pass region (about)",
hoverinfo="skip",
)
)
for name, results, color in [
("correct layer", correct, viz.BLUE),
("rope_theta = 10000", buggy_results, viz.ORANGE),
]:
fig.add_trace(
go.Scatter(
x=[r.max_abs_err for r in results],
y=[max(r.cos_dist, 1e-15) for r in results],
mode="markers+text",
name=name,
text=[
r.label.replace("batch ", "b").replace(", seq ", "/s")
for r in results
],
textposition="middle right",
marker=dict(
color=color,
size=11,
symbol=["circle" if r.passed else "x" for r in results],
),
hovertemplate="%{text}: max error %{x:.1e}, cos dist %{y:.1e}"
+ "<extra>"
+ name
+ "</extra>",
)
)
fig.update_layout(
title="Each shape's max error against its cosine distance",
height=440,
xaxis=dict(title="max |error| (log scale)", type="log", dtick=1),
yaxis=dict(title="cosine distance (log scale)", type="log", dtick=2),
)
viz.show(fig)
Each point is one shape, labeled batch/sequence length, and an x marks a
failure. The shaded corner is where both checks pass; its right edge is only
approximate, because allclose also allows a relative error of 1e-3 of each
expected value.
The single-token case still passes: at position 0, RoPE doesn’t rotate at
all, so the bug can’t show there. Every longer input fails. The maximum
error grows by two to four orders of magnitude and breaks the allclose
check. The cosine distance grows by six to eight orders of magnitude, but it
crosses its 1e-5 threshold only on the 512-token input. The two checks
catch different bugs, which is why the test uses both, and why the shape
list must include sequences, not just one token.
Where in the sequence does the bug show? Run the 512-token input through both layers and plot the largest error at each position:
x = (
np.random.default_rng(1)
.standard_normal((1, 512, config.hidden_size))
.astype(np.float32)
)
expected = reference_layer(x)[0]
per_position = {
name: np.abs(run(Tensor(x, device=DEVICE)).to_numpy()[0] - expected).max(-1)
for name, run in [
("correct layer", compiled_layer()),
("rope_theta = 10000", buggy),
]
}
Show the code 22 lines
fig = go.Figure(
[
go.Scatter(
x=np.arange(512),
y=np.maximum(err, 1e-9),
name=name,
mode="lines",
line_color=color,
hovertemplate="position %{x}: %{y:.1e}<extra>" + name + "</extra>",
)
for (name, err), color in zip(
per_position.items(), [viz.BLUE, viz.ORANGE]
)
]
)
fig.update_layout(
title="Largest error at each position of a 512-token input",
height=400,
xaxis_title="position",
yaxis=dict(title="max |error| (log scale)", type="log", dtick=1),
)
viz.show(fig)
At position 0 the two layers agree, because RoPE doesn’t rotate there.
Within a few positions the buggy layer’s error is three to four orders of
magnitude larger, and it stays there for the whole sequence, while the
correct layer’s error stays near 1e-5. A test that only ran one token
would have passed both.
As a pytest file
In a repository, these checks belong in a test file that runs on every
change. %%writefile saves a cell as a file. The test compiles the layer
once per module, as MAX’s own tests do, and checks every shape:
Show the code 45 lines
%%writefile test_decoder_layer.py
import numpy as np
import pytest
import torch
from max.dtype import DType
from max.graph import TensorType
from max.tensor import Tensor, default_device, default_dtype
from transformers import AutoConfig
from transformers.models.llama.modeling_llama import LlamaDecoderLayer, LlamaRotaryEmbedding
from minillm.config import MODEL_ID, LlamaConfig
from minillm.device import pick_device
from minillm.model import DecoderLayer
from minillm.testing import compare_outputs
from minillm.weights import load_checkpoint, select
CONFIG = LlamaConfig.from_pretrained()
@pytest.fixture(scope="module")
def layers():
weights = select(load_checkpoint(), "model.layers.0.")
with default_device(pick_device()), default_dtype(DType.float32):
ours = DecoderLayer(CONFIG)
ours.load_state_dict(weights)
compiled = ours.compile(
TensorType(DType.float32, ("batch", "seq_len", CONFIG.hidden_size), device=pick_device())
)
hf_config = AutoConfig.from_pretrained(MODEL_ID, attn_implementation="eager")
reference = LlamaDecoderLayer(hf_config, layer_idx=0).float().eval()
reference.load_state_dict({k: torch.from_numpy(v.copy()) for k, v in weights.items()})
return compiled, reference, LlamaRotaryEmbedding(hf_config)
@pytest.mark.parametrize("batch, seq", [(1, 1), (1, 7), (2, 33)])
def test_decoder_layer_matches_reference(layers, batch, seq):
compiled, reference, rotary = layers
x = np.random.default_rng(seq).standard_normal((batch, seq, CONFIG.hidden_size)).astype(np.float32)
hidden, positions = torch.from_numpy(x), torch.arange(seq).expand(batch, seq)
mask = torch.full((seq, seq), float("-inf")).triu(1)[None, None]
with torch.no_grad():
expected = reference(hidden, attention_mask=mask, position_ids=positions,
position_embeddings=rotary(hidden, positions)).numpy()
result = compare_outputs(compiled(Tensor(x, device=pick_device())).to_numpy(), expected)
assert result.passed, result
Overwriting test_decoder_layer.py
!python -m pytest -q -p no:cacheprovider test_decoder_layer.py 2>&1 | tail -n 3
... [100%]
3 passed in 4.76s
MAX’s own layer tests follow the same shape, with one more step: each harness also benchmarks the layer on the same shapes, so correctness and speed are measured together. Section 4.3 covers benchmarking.
Exercises
- Add a
bfloat16run: cast the input and the weights, and compare with thefloat32reference. Which tolerances do you need, and which metric moves most? - Test
CachedAttentionfrom Section 1.8: compare a prefill of 8 tokens followed by one decode step with a single 9-token pass. - Plant a different bug, such as swapping
gate_projandup_proj. Which metric catches it first?
Next
Section 4.2 tests the whole model: its logits on real prompts, against the same reference.
4.2 Verifying a model’s logits
Layer tests catch bugs in one layer. A whole model can still go wrong in ways no layer test sees: a weight loaded into the wrong layer, a tied head that isn’t tied, a mask that only fails on long prompts. So model bring-ups end with logit verification: run the same prompts through your model and a trusted reference, and compare the logits at every position.
This section verifies your SmolLM2 against Hugging Face’s PyTorch implementation, with the metrics MAX’s own verification tools use.
Show the setup 36 lines
import logging
import warnings
import numpy as np
import plotly.graph_objects as go
import torch
from max.driver import CPU
from max.dtype import DType
from max.graph import TensorType
from max.pipelines import AutoTokenizer
from max.tensor import Tensor
from transformers import AutoModelForCausalLM
from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.kv_cache import CachedLlamaForCausalLM, KVCacheDecoder
from minillm.model import load_model
from minillm.sampling import generate
from minillm.testing import compare_outputs
from minillm.weights import load_checkpoint
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
config = LlamaConfig.from_pretrained()
weights = load_checkpoint()
tokenizer = AutoTokenizer(MODEL_ID)
ours = load_model(config, weights, DEVICE).compile(
TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)
reference = AutoModelForCausalLM.from_pretrained(
MODEL_ID, torch_dtype=torch.float32
).eval()
Prompts
Verification prompts should exercise different behavior: plain prose, code, numbers, another language, and a prompt long enough to stress attention and RoPE at larger positions.
long_prompt = " ".join(["The quick brown fox jumps over the lazy dog."] * 60)
prompts = {
"prose": "The capital of France is Paris, and the capital of Italy is",
"code": "def fibonacci(n):\n if n < 2:\n return n\n return",
"numbers": "1, 1, 2, 3, 5, 8, 13, 21,",
"french": "Le chat est assis sur",
"long": long_prompt,
}
def logits_pair(text: str) -> tuple[np.ndarray, np.ndarray]:
ids = tokenizer([text])["input_ids"][0]
mine = ours(Tensor(np.array([ids]), device=DEVICE)).to_numpy()[0]
with torch.no_grad():
theirs = reference(torch.tensor([ids])).logits[0].numpy()
return mine, theirs
Metrics for logits
Logits feed a softmax, so on top of the elementwise metrics from Section 4.1, two measures matter for generation.
The KL divergence between the next-token distributions measures how much probability mass moved. It is the standard measure for comparing two models’ predictions, and it ignores differences in logits that softmax removes, such as a constant offset. For the reference distribution \(P\) and yours, \(Q\), at one position:
\[D_{\mathrm{KL}}(P \,\Vert\, Q) = \sum_i P_i \log \frac{P_i}{Q_i}\]
It is 0 only when the two distributions are identical.
Top-1 agreement is the fraction of positions at which both models predict the same most likely token. It is all that greedy decoding sees.
Show softmax, kl_divergence 26 lines
def softmax(z: np.ndarray) -> np.ndarray:
z = z.astype(np.float64) - z.max(-1, keepdims=True)
e = np.exp(z)
return e / e.sum(-1, keepdims=True)
def kl_divergence(p_logits: np.ndarray, q_logits: np.ndarray) -> float:
"""Mean KL(P || Q) over positions, in nats."""
p, q = softmax(p_logits), softmax(q_logits)
return float((p * (np.log(p + 1e-30) - np.log(q + 1e-30))).sum(-1).mean())
print(
f"{'prompt':8} {'positions':>9} {'max err':>8} {'cos dist':>9} {'KL':>9} {'top-1':>6}"
)
errors = {}
for name, text in prompts.items():
mine, theirs = logits_pair(text)
errors[name] = np.abs(mine - theirs).max(-1)
r = compare_outputs(mine, theirs, atol=1e-3, rtol=1e-3)
top1 = (mine.argmax(-1) == theirs.argmax(-1)).mean()
print(
f"{name:8} {len(mine):>9} {r.max_abs_err:>8.1e} {r.cos_dist:>9.1e} "
f"{kl_divergence(theirs, mine):>9.1e} {top1:>6.0%}"
)
assert top1 >= 0.99 and kl_divergence(theirs, mine) < 1e-6
prompt positions max err cos dist KL top-1
prose 13 8.2e-05 2.4e-12 3.2e-11 100%
code 18 8.3e-05 1.5e-12 4.0e-11 100%
numbers 25 1.1e-04 2.1e-12 2.2e-11 100%
french 6 1.0e-04 1.3e-12 8.6e-11 100%
long 600 1.1e-04 2.6e-12 2.5e-12 100%
A summary number can hide a trend. Plot the largest logit error at each position: if a bug in RoPE or the mask grew with position, the long prompt would show it as a rising line.
Show the code 22 lines
fig = go.Figure(
[
go.Scatter(
x=np.arange(1, len(err) + 1),
y=err,
customdata=np.arange(len(err)),
name=name,
mode="lines",
line_color=color,
hovertemplate=name
+ ", position %{customdata}: %{y:.1e}<extra></extra>",
)
for (name, err), color in zip(errors.items(), viz.SERIES)
]
)
fig.update_layout(
title="Largest logit error at each position, against Hugging Face",
height=400,
xaxis=dict(title="position + 1 (log scale)", type="log", dtick=1),
yaxis=dict(title="max |logit difference|", type="log", dtick=1),
)
viz.show(fig)
Every prompt agrees at every position, including the 600-token one, with 100%
top-1 agreement and KL divergences between about 1e-12 and 1e-10 nats. MAX
verifies its own models the same way, comparing MAX’s logits with reference
logits from PyTorch or vLLM by cosine distance, KL divergence, and tolerances.
The
model bring-up workflow
guide covers the rest of bringing a new architecture to MAX.
Greedy generation
Matching logits at the prompt is necessary but not sufficient: generation feeds
the model its own outputs, through your KV cache from
Section 1.8. Compare 48 greedy
tokens from your cached decoder with Hugging Face’s generate:
Show the code 27 lines
cached = KVCacheDecoder(
load_model(config, weights, DEVICE, CachedLlamaForCausalLM), config, DEVICE
)
gaps = {}
for name in ["prose", "code", "numbers"]:
ids = tokenizer([prompts[name]])["input_ids"][0]
mine = generate(cached, ids, 48, temperature=0)
with torch.no_grad():
out = reference.generate(
torch.tensor([ids]),
max_new_tokens=48,
do_sample=False,
output_logits=True,
return_dict_in_generate=True,
)
theirs = out.sequences[0, len(ids) :].tolist()
gaps[name] = [
float(np.diff(np.sort(step[0].numpy())[-2:])[0]) for step in out.logits
]
same = next(
(i for i, (a, b) in enumerate(zip(mine, theirs)) if a != b), len(mine)
)
print(f"{name:8} identical for {same} of {len(mine)} tokens")
assert same >= 24, (
"diverged early; check the gap between the top two logits"
)
prose identical for 48 of 48 tokens
code identical for 48 of 48 tokens
numbers identical for 48 of 48 tokens
At each greedy step, how far ahead was the winning token? output_logits=True
returns the reference’s raw logits at every step; plot the gap between the top
two, with the largest logit error from the first chart as a dotted line:
Show the code 32 lines
fig = go.Figure(
[
go.Scatter(
x=np.arange(len(gap)),
y=gap,
name=name,
mode="lines+markers",
line_color=color,
marker=dict(size=5),
hovertemplate=name + ", step %{x}: gap %{y:.3f}<extra></extra>",
)
for (name, gap), color in zip(gaps.items(), viz.SERIES)
]
)
fig.update_layout(
title="Gap between the top two logits at each greedy step",
height=400,
xaxis_title="generated token",
yaxis=dict(title="logit gap (log scale)", type="log", dtick=1),
)
largest_error = max(err.max() for err in errors.values())
fig.add_trace(
go.Scatter(
x=[0, max(len(gap) for gap in gaps.values()) - 1],
y=[largest_error] * 2,
name=f"largest logit error ({largest_error:.0e})",
mode="lines",
line=dict(color=viz.INK_SECONDARY, dash="dot", width=1),
hoverinfo="skip",
)
)
viz.show(fig)
A rounding difference can change the winner only at a step where the gap is
smaller than the error, that is, where a line dips below the dotted one. Here the smallest gap is above 0.1 and the largest error
about 1e-4, so no step comes close.
If a run does diverge after many tokens, the cause is usually a near tie: at some step two tokens have almost the same logit, and a difference in the fourth or fifth decimal place, from summing in a different order, picks a different winner. From then on the two texts differ, though both are valid greedy continuations. When you see a divergence, look at the gap between the top two logits at that step: a tiny gap means rounding, a large one means a bug.
Beyond logits: accuracy evaluations
Logit verification compares two implementations. It can’t tell you whether the model is good at a task, or whether a change that alters numbers on purpose, such as quantization, costs accuracy. For that, serving teams run task evaluations such as GSM8K or MMLU against the server (Section 7.1 serves your model). The handbook’s quantization page covers how quantized models are evaluated.
Exercises
- Run the comparison in
bfloat16(cast the weights and cast the logits back tofloat32). How do KL divergence and top-1 agreement change? - Find the step where two tokens’ logits are closest during a greedy run. How small is the gap?
- Load one layer’s weights into the wrong layer index and rerun the table. Which prompt and metric show it most clearly?
Next
Section 4.3 measures speed: how to benchmark kernels and layers so the numbers mean something.
4.3 Benchmarking kernels and layers
A benchmark is an experiment, and most quick timings are bad experiments: they include compilation, measure the time to launch work instead of the time to finish it, or report one noisy sample. This section writes a small benchmark harness that avoids those mistakes, then uses it to show why decoding one token is limited by memory traffic and prefill by arithmetic.
Show the setup 23 lines
import logging
import time
import warnings
import numpy as np
import plotly.graph_objects as go
from max.driver import CPU
from max.dtype import DType
from max.graph import TensorType
from max.nn import Linear
from max.tensor import Tensor, default_device, default_dtype
from plotly.subplots import make_subplots
from minillm import viz
from minillm.device import pick_device
from minillm.config import LlamaConfig
from minillm.model import DecoderLayer
from minillm.weights import load_checkpoint, select
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
config = LlamaConfig.from_pretrained()
A harness
The harness follows four rules, in the spirit of MAX’s layer benchmarks, which time on the device with events and report the median and p95:
- Warm up. The first calls compile kernels and fill caches, so don’t time them.
- Wait for the result. Work can run asynchronously, especially on a GPU.
Reading the output (
to_numpy()) forces it to finish. - Repeat. One sample is mostly noise, so take many.
- Report the median and a tail percentile. Outliers pull the mean; the median shows the typical run and p90 shows how slow an unlucky one gets.
Show Timing, benchmark 38 lines
from __future__ import annotations
import time
from collections.abc import Callable
from dataclasses import dataclass
import numpy as np
@dataclass
class Timing:
"""Milliseconds per call over the measured iterations."""
median_ms: float
p90_ms: float
min_ms: float
def __str__(self) -> str:
return f"median {self.median_ms:7.2f} ms p90 {self.p90_ms:7.2f} ms"
def benchmark(
fn: Callable[[], object], warmup: int = 3, iterations: int = 20
) -> Timing:
"""Times ``fn``, which must return a fully computed result (for example,
a NumPy array read back from the device)."""
for _ in range(warmup):
fn()
samples = []
for _ in range(iterations):
start = time.perf_counter()
fn()
samples.append((time.perf_counter() - start) * 1000)
return Timing(
median_ms=float(np.median(samples)),
p90_ms=float(np.percentile(samples, 90)),
min_ms=float(np.min(samples)),
)
Saved to minillm/bench.py.
Memory-bound or compute-bound?
Every projection in the model is a matrix multiplication, x @ W.T, with
x holding one row per token. Its cost has two parts: reading W from
memory, which doesn’t depend on the number of tokens, and the arithmetic,
which grows with it. For \(n\) tokens and a \(K \to N\) projection in
float32 (4 bytes per number), the ratio of the two is the operation’s
arithmetic intensity:
\[I = \frac{\text{FLOPs}}{\text{bytes moved}} = \frac{2nKN}{4\,(KN + nK + nN)}\]
A device can’t run faster than its compute peak \(P_{\text{peak}}\), nor faster than its memory bandwidth \(B\) can feed it, which gives the roofline:
\[P \le \min\left(P_{\text{peak}},\; B \cdot I\right)\]
With few tokens, \(I\) is small and memory sets the limit; with many, \(I\)
grows until compute does. Measure SmolLM2’s gate_proj
(\(K = 576\), \(N = 1536\)) for 1 to 2048 tokens:
Show the code 29 lines
with default_device(DEVICE), default_dtype(DType.float32):
proj = Linear(config.hidden_size, config.intermediate_size, bias=False)
run = proj.compile(
TensorType(DType.float32, ("tokens", config.hidden_size), device=DEVICE)
)
K, N = config.hidden_size, config.intermediate_size
rows = []
for tokens in [1, 4, 16, 64, 256, 1024, 2048]:
x = Tensor(
np.random.default_rng(0)
.standard_normal((tokens, K))
.astype(np.float32),
device=DEVICE,
)
t = benchmark(lambda: run(x).to_numpy())
flops = 2 * tokens * K * N
bytes_moved = 4 * (K * N + tokens * (K + N))
rows.append(
(
tokens,
t.median_ms,
flops / t.median_ms / 1e6,
bytes_moved / t.median_ms / 1e6,
)
)
print(
f"{tokens:5} tokens {t} {rows[-1][2]:7.1f} GFLOP/s {rows[-1][3]:6.1f} GB/s"
)
1 tokens median 0.56 ms p90 0.63 ms 3.1 GFLOP/s 6.3 GB/s
4 tokens median 0.35 ms p90 0.66 ms 20.3 GFLOP/s 10.2 GB/s
16 tokens median 0.39 ms p90 0.47 ms 72.0 GFLOP/s 9.3 GB/s
64 tokens median 0.86 ms p90 1.25 ms 131.0 GFLOP/s 4.7 GB/s
256 tokens median 2.07 ms p90 2.97 ms 218.5 GFLOP/s 2.8 GB/s
1024 tokens median 8.22 ms p90 11.20 ms 220.3 GFLOP/s 1.5 GB/s
2048 tokens median 10.09 ms p90 14.24 ms 359.1 GFLOP/s 2.1 GB/s
Show the code 82 lines
tokens, ms, gflops, gbps = zip(*rows)
intensity = [
2 * t * K * N / (4 * (K * N + t * (K + N))) for t in tokens
] # FLOP per byte moved
fig = make_subplots(
rows=1,
cols=2,
horizontal_spacing=0.14,
subplot_titles=["Time per call", "Arithmetic throughput against intensity"],
)
fig.add_trace(
go.Scatter(
x=tokens,
y=ms,
mode="lines+markers",
line_color=viz.BLUE,
showlegend=False,
hovertemplate="%{x} tokens: %{y:.3f} ms<extra></extra>",
),
row=1,
col=1,
)
fig.add_trace(
go.Scatter(
x=intensity,
y=gflops,
mode="lines+markers+text",
line_color=viz.ORANGE,
text=[str(t) for t in tokens],
textposition="top left",
showlegend=False,
customdata=tokens,
hovertemplate="%{customdata} tokens<br>%{x:.1f} FLOP/byte<br>%{y:.0f} GFLOP/s<extra></extra>",
),
row=1,
col=2,
)
# The roof uses the best bandwidth and throughput this sweep reached, not the
# device's datasheet numbers, so every point sits on or under it.
peak, bandwidth = max(gflops), max(gbps)
roof_x = np.geomspace(min(intensity) / 2, max(intensity) * 2, 200)
fig.add_trace(
go.Scatter(
x=roof_x,
y=np.minimum(peak, bandwidth * roof_x),
mode="lines",
line=dict(color=viz.INK_SECONDARY, dash="dash", width=1.5),
name="measured roofline",
hovertemplate="roof at %{x:.1f} FLOP/byte: %{y:.0f} GFLOP/s<extra></extra>",
),
row=1,
col=2,
)
fig.add_annotation(
x=np.log10(peak / bandwidth),
y=np.log10(peak),
text=f"ridge: {peak / bandwidth:.0f} FLOP/byte",
showarrow=True,
arrowhead=0,
ay=-30,
row=1,
col=2,
)
fig.update_xaxes(title_text="tokens", type="log", dtick=1, row=1, col=1)
fig.update_yaxes(title_text="ms (log scale)", type="log", dtick=1, row=1, col=1)
fig.update_xaxes(
title_text="FLOP per byte moved (log scale)",
type="log",
dtick=1,
row=1,
col=2,
)
fig.update_yaxes(
title_text="GFLOP/s (log scale)", type="log", dtick=1, row=1, col=2
)
fig.update_layout(
title="gate_proj (576 to 1536) from 1 to 2048 tokens; labels are token counts",
height=420,
legend=dict(x=0.58, y=1.12),
)
viz.show(fig)
Read the two plots together, and hover over the points. The dashed line on the right is the roofline drawn from this sweep: its slope is the best bandwidth any point reached, its flat top the best throughput. Points left of the ridge sit under the slanted, memory-bound part of the roof; points to the right sit under the flat, compute-bound part. A device’s datasheet ceilings are higher than these measured ones.
With few tokens, the time barely grows: 1 to 16 tokens all take about half a
millisecond on this page’s machine, an Apple-silicon Mac with a Metal GPU. Treat these
numbers as an example: yours depend on your hardware and vary from run to run.
Each weight is used for only a few multiply-adds after it is loaded, so
arithmetic throughput is low and the operation is memory-bound. At a matrix
this small, though, most of that time is a fixed cost per call, for launching
the work and reading the result back: the single-token call moves W’s 3.5 MB
at under 10 GB/s, far below the GPU’s memory bandwidth. In a large model,
reading the weights dominates instead.
With many tokens, each weight, once loaded, is reused for every token, so arithmetic throughput climbs and the time grows with the number of tokens. The operation moves toward the compute-bound end, where the hardware’s arithmetic units set the limit.
Decoding one token per step runs every weight matrix at the memory-bound end. That is why per-token latency depends on memory bandwidth, and why serving engines batch many requests into each step (Chapter 5): here, 16 tokens cost barely more than one. Prefill processes the whole prompt at once and runs toward the compute-bound end. The handbook’s GPU memory page explains the same trade-off on GPUs.
Benchmark a layer
The same harness measures a whole decoder layer at the shapes that matter in serving: decode steps (one token) for several batch sizes, and prefills of several lengths.
with default_device(DEVICE), default_dtype(DType.float32):
layer = DecoderLayer(config)
layer.load_state_dict(select(load_checkpoint(), "model.layers.0."))
run_layer = layer.compile(
TensorType(
DType.float32, ("batch", "seq_len", config.hidden_size), device=DEVICE
)
)
print("shape (batch x seq) time per call tokens/s")
medians = {}
for batch, seq in [(1, 1), (8, 1), (32, 1), (1, 128), (1, 512)]:
x = Tensor(
np.random.default_rng(0)
.standard_normal((batch, seq, config.hidden_size))
.astype(np.float32),
device=DEVICE,
)
t = benchmark(lambda: run_layer(x).to_numpy(), warmup=5, iterations=50)
medians[batch, seq] = t.median_ms
print(f"{batch:5} x {seq:<5} {t} {batch * seq / t.median_ms * 1000:9.0f}")
print(
f"a decode step for 32 sequences costs {medians[32, 1] / medians[1, 1]:.1f}x one sequence's"
)
shape (batch x seq) time per call tokens/s
1 x 1 median 1.42 ms p90 3.23 ms 704
8 x 1 median 1.47 ms p90 2.30 ms 5429
32 x 1 median 1.68 ms p90 2.35 ms 19027
1 x 128 median 2.65 ms p90 3.15 ms 48343
1 x 512 median 8.16 ms p90 8.84 ms 62743
a decode step for 32 sequences costs 1.2x one sequence's
Show the code 32 lines
labels = [f"{b} x {s}" for b, s in medians]
fig = make_subplots(
rows=1,
cols=2,
horizontal_spacing=0.12,
subplot_titles=["Time per call (ms)", "Tokens per second"],
)
fig.add_trace(
go.Bar(
x=labels,
y=list(medians.values()),
marker_color=viz.BLUE,
showlegend=False,
hovertemplate="%{x}: %{y:.2f} ms<extra></extra>",
),
row=1,
col=1,
)
fig.add_trace(
go.Bar(
x=labels,
y=[b * s / ms * 1000 for (b, s), ms in medians.items()],
marker_color=viz.AQUA,
showlegend=False,
hovertemplate="%{x}: %{y:,.0f} tokens/s<extra></extra>",
),
row=1,
col=2,
)
fig.update_xaxes(title_text="batch x sequence length")
fig.update_layout(title="One decoder layer at serving shapes", height=400)
viz.show(fig)
On this machine, a decode step for 32 sequences costs well under twice a single sequence’s step, so throughput in tokens per second rises sharply with batch size. Timings on a laptop are noisy, which is why the harness reports a percentile alongside the median. A 512-token prefill runs through one layer at tens of thousands of tokens per second; the model has 30 layers, so a whole-model prefill would run at a thirtieth of that, under 2,000 tokens per second.
These numbers describe this machine’s GPU. On another CPU or GPU the curves have
the same shape at different scales, although the time isn’t always monotonic in
size: the kernels behind each operation choose different strategies for
different shapes, so on some devices a smaller batch takes longer than a larger
one (this page’s run doesn’t show it). Measure the shapes you serve. MAX’s own
layer benchmarks
run this kind of sweep for every layer, and its kernel benchmarks do the same
for individual kernels. For whole-server measurements,
Section 7.2 uses
max benchmark. The handbook’s
LLM performance benchmarks
page explains how to read published numbers.
Exercises
- Run the matmul sweep for
down_proj(1536 → 576). Where does it switch from memory-bound to compute-bound? - Benchmark the same layer with
iterations=3five times. How much do the medians vary? How many iterations make the result stable? - Compare the eager and compiled layer at batch 1, sequence 1. Where does eager execution spend its time?
Next
Section 4.4 looks inside a run: which operations take the time, using MAX’s profiling tools.
4.4 Profiling
A benchmark tells you how long something takes. A profile tells you where the time goes, which is what you need before optimizing anything. This section profiles at three levels: a whole generation run, the kernels a compiled graph launches, and the parts of your own model.
Show the setup 21 lines
import logging
import warnings
import numpy as np
import plotly.graph_objects as go
from max.driver import CPU
from max.dtype import DType
from max.graph import TensorType
from max.tensor import Tensor, default_device, default_dtype
from minillm import viz
from minillm.bench import benchmark
from minillm.device import pick_device
from minillm.config import LlamaConfig
from minillm.model import load_model
from minillm.weights import load_checkpoint
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
config = LlamaConfig.from_pretrained()
Profile a whole run
max generate takes a --profile flag. On an NVIDIA GPU with Nsight
Systems installed, it records a kernel trace and prints the kernels that
took the most GPU time. It always profiles the Python side with cProfile
too, which answers the first question to ask of any run: is the time in your
model, or in the code around it?
!max generate --model HuggingFaceTB/SmolLM2-135M-Instruct \
--quantization-encoding float32 --prompt "hello" --max-new-tokens 16 \
--num-warmups 1 --profile 2>&1 | grep -A 8 "=== Top"
=== Top 15 Python/CPU functions (94.4% of CPU time, 307.64 ms total) ===
% total calls function
49.0% 150.85 ms 64 ~:0<built-in method numpy.from_dlpack>
43.0% 132.25 ms 32 max/engine/api.py:118(_Model_execute)
0.3% 1.04 ms 16 max/pipelines/lib/pipeline_variants/text_generation.py:597(execute)
0.3% 970.84 us 16 max/pipelines/sampling/sampling_logits_processor.py:109(_packed_rows_to_device)
0.3% 781.75 us 16 max/pipelines/lib/module_registration.py:763(prepare_initial_token_inputs)
0.2% 589.67 us 16 max/pipelines/sampling/sampling_logits_processor.py:695(create)
0.2% 586.50 us 16 max/pipelines/lib/pipeline_variants/text_generation.py:610(_execute)
The time is in the model: _Model_execute, the call into the compiled graph,
and, on a GPU, numpy.from_dlpack, which reads the logits back to the host and
so waits for the GPU to finish its work. That wait is model time too, a common
trap when reading profiles of asynchronous devices. On this page’s machine, an
Apple-silicon Mac with a Metal GPU, the two take over 90% of the profiled time.
Treat these numbers as an example: yours depend on your hardware and vary from
run to run. The pipeline around the model (tokenization, scheduling, sampling,
and the KV cache bookkeeping) takes under a percent per function. For this
model, any speedup has to come from the compiled graph itself.
See which kernels run
MODULAR_DEBUG=op-log-level=TRACE makes the runtime log every operation it
launches and completes. Trace the MLP from
Section 1.4. The setting
must be in place when the process starts, so the code runs as a separate script:
%%writefile trace_mlp.py
import numpy as np
from max.dtype import DType
from max.graph import TensorType
from max.tensor import Tensor, default_device, default_dtype
from minillm.config import LlamaConfig
from minillm.device import pick_device
from minillm.layers import MLP
DEVICE = pick_device()
with default_device(DEVICE), default_dtype(DType.float32):
mlp = MLP(LlamaConfig())
run = mlp.compile(TensorType(DType.float32, ("seq_len", 576), device=DEVICE))
print("--- run ---", flush=True)
run(Tensor(np.ones((4, 576), np.float32), device=DEVICE)).to_numpy()
Overwriting trace_mlp.py
!MODULAR_DEBUG=op-log-level=TRACE python trace_mlp.py 2>&1 | sed -n '/--- run ---/,$p' | grep LAUNCH | grep -v 'LAUNCH matmul[^ ]'
[OP] LAUNCH matmul [id=22] target=gpu:0
[OP] LAUNCH Epilogue(matmul, Elementwise(silu, mul), operandIndices=[0]) [id=24] target=gpu:0
[OP] LAUNCH matmul [id=26] target=gpu:0
Each matmul logs a second LAUNCH line from inside the kernel. That line names
the kernel and, in parentheses, the graph operation it serves
(matmul(mo.matmul), or matmul(Epilogue(mo.matmul, ...)) for the fused one);
the grep -v drops it, which leaves one line per kernel the graph launches.
The MLP has five operations in your Python (three matmuls, silu, and a
multiply), more in the traced graph you saw in
Section 3.2, and three kernels
at run time. The Epilogue entry is the
second matmul with silu and mul fused into it: the activation is applied as
each tile of the matmul’s output is produced, before it is written to memory.
That fusion is why the hand-written SwiGLU kernel in
Section 3.1 had little left to
fuse, and the trace is how you’d find out. The handbook’s
kernel optimization for LLM inference
page explains fusion and why it matters most on GPUs.
Profile by decomposition
To find where time goes inside your own model, time its parts separately with the harness from Section 4.3. Compile one layer’s attention, MLP, and norm, and the LM head, and time each at two shapes: one decode step (1 token) and a prefill (256 tokens). Each time is multiplied by how often the part runs in the model: 30 attention and MLP blocks, and 61 norms (two per layer plus the final one).
Show the code 31 lines
model = load_model(config, load_checkpoint(), DEVICE)
hidden = TensorType(
DType.float32, ("batch", "seq_len", config.hidden_size), device=DEVICE
)
layer = model.model.layers[0]
parts = {
"attention (x30)": (layer.self_attn.compile(hidden), 30),
"MLP (x30)": (layer.mlp.compile(hidden), 30),
"norms (x61)": (layer.input_layernorm.compile(hidden), 61),
"LM head": (model.lm_head.compile(hidden), 1),
}
shares, all_times = {}, {}
for tokens in [1, 256]:
x = Tensor(
np.random.default_rng(0)
.standard_normal((1, tokens, config.hidden_size))
.astype(np.float32),
device=DEVICE,
)
times = {
name: benchmark(lambda: run(x).to_numpy(), iterations=30).median_ms
* count
for name, (run, count) in parts.items()
}
total = sum(times.values())
shares[tokens] = {name: t / total for name, t in times.items()}
all_times[tokens] = times
print(f"{tokens} token(s): {total:.1f} ms in total")
for name, t in times.items():
print(f" {name:16} {t:7.2f} ms {t / total:5.0%}")
1 token(s): 49.2 ms in total
attention (x30) 23.31 ms 47%
MLP (x30) 10.01 ms 20%
norms (x61) 13.80 ms 28%
LM head 2.05 ms 4%
256 token(s): 156.0 ms in total
attention (x30) 73.32 ms 47%
MLP (x30) 33.32 ms 21%
norms (x61) 20.77 ms 13%
LM head 28.61 ms 18%
Show the code 25 lines
rows = ["decode (1 token)", "prefill (256 tokens)"]
fig = go.Figure(
[
go.Bar(
y=rows,
x=[shares[1][name], shares[256][name]],
name=name,
orientation="h",
marker=dict(color=color, line=dict(color=viz.SURFACE, width=2)),
customdata=[all_times[1][name], all_times[256][name]],
hovertemplate=name
+ ": %{x:.0%} (%{customdata:.2f} ms)<extra>%{y}</extra>",
)
for name, color in zip(parts, viz.SERIES)
]
)
fig.update_layout(
title="Where the model's time goes",
barmode="stack",
height=300,
legend_traceorder="normal",
xaxis=dict(title="share of model time", tickformat=".0%", range=[0, 1]),
yaxis=dict(autorange="reversed", showgrid=False),
)
viz.show(fig)
Compare the two bars. The shares below were measured on this page’s Metal GPU.
Attention takes the largest share at both shapes, about 40%. Its cost is the only one that grows with the square of the sequence length, so its share grows on longer prompts; at 256 tokens the prompt is still short enough that it barely moves.
The LM head’s share grows the most, from a few percent to over a tenth. It holds more weights than all 30 layers’ attention put together (49,152 × 576), and here it runs on every one of the 256 prompt positions. The cached model from Section 1.8 computes it for the last position only, which keeps it small during prefill.
The norms look expensive at decode, over a quarter of the time, but that is mostly an artifact of this method. Each separately compiled call pays a fixed cost to launch its work and read back the result, and the decomposition counts that cost 61 times for the norms; the whole compiled model runs as one graph and pays it once. Time the whole model as one compiled graph to see how large that overhead is:
whole = model.compile(
TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)
for tokens in [1, 256]:
ids = Tensor(np.zeros((1, tokens), np.int64), device=DEVICE)
whole_ms = benchmark(lambda: whole(ids).to_numpy(), iterations=30).median_ms
parts_ms = sum(all_times[tokens].values())
print(
f"{tokens:3} token(s): whole model {whole_ms:6.1f} ms,"
f" sum of parts {parts_ms:6.1f} ms ({parts_ms / whole_ms:.1f}x)"
)
1 token(s): whole model 24.0 ms, sum of parts 49.2 ms (2.0x)
256 token(s): whole model 84.0 ms, sum of parts 156.0 ms (1.9x)
At both shapes the whole model takes less than half the time its parts add up to. The difference is what the 122 separate calls cost: each one launches its work and reads its result back to the host, and the whole graph does both once. So a decomposition ranks the parts, but its shares overstate small operations such as the norms. Check a surprising number with the kernel trace before acting on it.
Every optimization starts here: find the largest share, decide whether it is memory-bound or compute-bound (Section 4.3), then look at its kernels (the trace above).
With Nsight Systems
On NVIDIA GPUs, Nsight Systems answers the same questions: set
MODULAR_ENABLE_PROFILING=detailed and run your program under nsys profile,
or pass --profile to max generate. max benchmark --trace does the same
for a server that already runs under nsys launch, started with
--gpu-profiling detailed. MAX marks its operations as
named ranges in the timeline, so you can match GPU kernels to the operations
that launched them. MAX’s guide to
GPU profiling with Nsight Systems
covers the setup. The handbook’s
kernel optimization tools
page surveys the tools.
Exercises
- Trace a full
DecoderLayerand count its kernels. Which operations are fused into which? - Add a third shape, a decode step for a batch of 32, to the decomposition. Which part’s share changes most?
- Profile the decode step of the
KVCacheDecoderfrom Section 1.8 withcProfilefrom Python. How much of the step is Python overhead at this model size?
Next
Chapter 5 uses what you measured here to build the core of a serving engine that batches many requests into each step.
5. A serving engine
The model from Chapter 1 generates for one prompt at a time. A serving engine takes requests from many users at once and keeps them all moving. This chapter builds the core of such an engine, one technique per section.
| Section | You learn |
|---|---|
| 5.1 The life of a request | A batched model step for an engine, and a first engine that serves one request at a time. |
| 5.2 Continuous batching | A scheduler, and three batching policies compared on the same workload. |
| 5.3 A paged KV cache | Blocks and block tables, the idea behind MAX Serve’s paged KV cache, and the memory they save. |
| 5.4 Prefix caching | Sharing the cache blocks of a common prompt prefix between requests. |
| 5.5 Speculative decoding | Drafting tokens with SmolLM2-135M, verifying them with SmolLM2-360M, and when it pays off. |
Related handbook pages: batching, PagedAttention, prefix caching, and speculative decoding.
5.1 The life of a request
So far your model served one prompt at a time, from a notebook. A serving engine takes requests from many users at once and keeps them all moving: it admits new requests, runs the model, hands each request its next token, and stops each one when it is done. This chapter builds that engine over five sections, on top of a new model interface designed for it.
MAX Serve and other inference engines share this structure:
The handbook’s How does LLM inference work? page describes the same loop.
Show the setup 24 lines
import logging
import time
import warnings
import numpy as np
import plotly.graph_objects as go
from max.dtype import DType
from max.graph import TensorType
from max.pipelines import AutoTokenizer
from max.tensor import Tensor
from plotly.subplots import make_subplots
from minillm import viz
from minillm.config import MODEL_ID, LlamaConfig
from minillm.device import pick_device
from minillm.model import load_model
from minillm.weights import load_checkpoint
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
config = LlamaConfig.from_pretrained()
tokenizer = AutoTokenizer(MODEL_ID)
Show the setup 22 lines
from __future__ import annotations
import math
from collections.abc import Callable
from max import functional as F
from max.dtype import DType
from max.nn import ModuleList
from max.nn.kernels import flash_attention
from max.tensor import Tensor
from minillm.attention import Attention
from minillm.config import LlamaConfig
from minillm.layers import rotate_half
from minillm.model import DecoderLayer, LlamaForCausalLM
KVPair = tuple[Tensor, Tensor]
Step = Callable[..., tuple[Tensor, list[KVPair]]]
# Masked scores use a large finite value instead of -inf: attention kernels
# process keys in blocks, and a row whose first block is fully -inf computes
# exp(-inf - -inf) = NaN, which then spreads through the values to other rows.
MASKED = -1e30
Saved to minillm/step.py.
A model step for an engine
The cached model from Section 1.8 kept its cache inside and handled one sequence. An engine needs one call that advances several sequences, each with its own history, by any number of tokens. The step model does that. Each call takes, for every sequence in the batch:
- the new tokens and their positions,
- an attention mask saying which past and new positions each token may see,
- the past keys and values,
and returns the next-token logits plus only the new keys and values. The model keeps no state. Where the cache lives and how it is laid out is up to the engine, and Section 5.3 and Section 5.4 change exactly that.
Positions now differ per sequence, so RoPE takes a position for each token instead of a start offset:
def rope_at(
x: Tensor, positions: Tensor, head_dim: int, theta: float
) -> Tensor:
"""Rotates ``x``, ``[batch, q, heads, head_dim]``, for per-token
``positions``, ``[batch, q]``."""
inv_freq = Tensor(
[theta ** (-i / head_dim) for i in range(0, head_dim, 2)],
dtype=DType.float32,
device=x.mesh,
)
batch, q = positions.shape
angles = positions.reshape([batch, q, 1]) * inv_freq
angles = F.concat([angles, angles], axis=-1).reshape(
[batch, q, 1, head_dim]
)
return x * F.cos(angles) + rotate_half(x) * F.sin(angles)
Saved to minillm/step.py.
The attention layer projects only the new tokens, appends their keys and values to the past ones, and attends over both. The mask has one column per past and new position:
Show StepAttention 27 lines
class StepAttention(Attention):
"""Attends from new tokens to the past keys and values plus their own."""
def forward(
self, x: Tensor, positions: Tensor, mask: Tensor, past: KVPair
) -> tuple[Tensor, KVPair]:
batch, q, _ = x.shape
query = self.q_proj(x).reshape(
[batch, q, self.num_heads, self.head_dim]
)
key = self.k_proj(x).reshape(
[batch, q, self.num_kv_heads, self.head_dim]
)
value = self.v_proj(x).reshape(
[batch, q, self.num_kv_heads, self.head_dim]
)
query = rope_at(query, positions, self.head_dim, self.rope_theta)
key = rope_at(key, positions, self.head_dim, self.rope_theta)
all_keys = F.concat([past[0], key], axis=1)
all_values = F.concat([past[1], value], axis=1)
# The mask has one column per past and new position.
mask = F.rebind(mask, [batch, q, all_keys.shape[1]])
out = flash_attention(
query, all_keys, all_values, 1 / math.sqrt(self.head_dim), mask
).rebind_mapping(query.mapping)
out = out.reshape([batch, q, self.num_heads * self.head_dim])
return self.o_proj(out), (key, value)
Saved to minillm/step.py.
The decoder layer and the model pass the new arguments through, and the model collects each layer’s new keys and values:
Show StepLayer, StepModel 38 lines
class StepLayer(DecoderLayer):
def __init__(self, config: LlamaConfig) -> None:
super().__init__(config)
self.self_attn = StepAttention(config)
def forward(
self, x: Tensor, positions: Tensor, mask: Tensor, past: KVPair
) -> tuple[Tensor, KVPair]:
attn, kv = self.self_attn(
self.input_layernorm(x), positions, mask, past
)
x = x + attn
return x + self.mlp(self.post_attention_layernorm(x)), kv
class StepModel(LlamaForCausalLM):
"""Advances a batch of sequences: returns each one's next-token logits
and the new tokens' keys and values for every layer."""
def __init__(self, config: LlamaConfig) -> None:
super().__init__(config)
self.model.layers = ModuleList(
[StepLayer(config) for _ in range(config.num_hidden_layers)]
)
def forward(
self,
tokens: Tensor,
positions: Tensor,
mask: Tensor,
past: list[KVPair],
) -> tuple[Tensor, list[KVPair]]:
x = self.model.embed_tokens(tokens)
new_kv = []
for layer, layer_past in zip(self.model.layers, past, strict=True):
x, kv = layer(x, positions, mask, layer_past)
new_kv.append(kv)
return self.lm_head(self.model.norm(x)[:, -1]), new_kv
Saved to minillm/step.py.
Compile it once. Every dimension that changes between calls is symbolic: the
batch size, the number of new tokens q, and the past length:
import numpy as np
from max.driver import Device
from max.graph import TensorType
def compile_step(model: StepModel, config: LlamaConfig, device: Device) -> Step:
"""Compiles ``model`` for any batch size, number of new tokens, and past
length."""
kv = TensorType(
DType.float32,
("batch", "past", config.num_key_value_heads, config.head_dim),
device=device,
)
return model.compile(
TensorType(DType.int64, ("batch", "q"), device=device),
TensorType(DType.float32, ("batch", "q"), device=device),
TensorType(DType.float32, ("batch", "q", "kv"), device=device),
[(kv, kv)] * config.num_hidden_layers,
)
Saved to minillm/step.py.
start = time.perf_counter()
weights = load_checkpoint()
step = compile_step(
load_model(config, weights, DEVICE, StepModel), config, DEVICE
)
print(f"compiled the step model in {time.perf_counter() - start:.0f} s")
compiled the step model in 13 s
The cache and the batch
The engine stores each sequence’s keys and values in a cache object. The
first version is the simplest: one growing array per layer. Every later
cache has the same interface (length, num_layers, read, and append),
so run_step and the step model work with all of them unchanged.
class ContiguousCache:
"""One sequence's keys and values: a growing array per layer."""
def __init__(self, config: LlamaConfig) -> None:
shape = (0, config.num_key_value_heads, config.head_dim)
self.layers = [
[np.zeros(shape, np.float32), np.zeros(shape, np.float32)]
for _ in range(config.num_hidden_layers)
]
@property
def length(self) -> int:
return len(self.layers[0][0])
@property
def num_layers(self) -> int:
return len(self.layers)
def read(self, layer: int) -> tuple[np.ndarray, np.ndarray]:
return self.layers[layer][0], self.layers[layer][1]
def append(self, layer: int, keys: np.ndarray, values: np.ndarray) -> None:
entry = self.layers[layer]
entry[0] = np.concatenate([entry[0], keys])
entry[1] = np.concatenate([entry[1], values])
Saved to minillm/step.py.
To run several sequences in one step, the engine packs them into rectangular arrays. Sequences add different numbers of new tokens, so shorter ones are left-padded, and the mask is built per token: a new token may see its own sequence’s past and the new tokens up to itself, and nothing else. A padding token sees only itself, so its row of the softmax stays well defined; its output is discarded. Masked positions get a large negative number instead of \(-\infty\). If a padding row’s first block of keys were entirely \(-\infty\), a blockwise attention kernel would compute \(e^{-\infty + \infty}\), which is NaN. In the next layer that NaN would become the padding position’s keys and values, and the real tokens of the same sequence would pick it up even though the mask hides that position.
Show run_step 45 lines
def run_step(
step: Step, caches: list, new_tokens: list[list[int]], device: Device
) -> np.ndarray:
"""Advances each sequence by its ``new_tokens``, appends their keys and
values to its cache, and returns next-token logits, ``[batch, vocab]``."""
batch, q = len(caches), max(len(t) for t in new_tokens)
past = max(c.length for c in caches)
tokens = np.zeros((batch, q), np.int64)
positions = np.zeros((batch, q), np.float32)
mask = np.full((batch, q, past + q), MASKED, np.float32)
for i, (cache, new) in enumerate(zip(caches, new_tokens, strict=True)):
pad, n = q - len(new), cache.length
tokens[i, pad:] = new
positions[i, pad:] = np.arange(n, n + len(new))
mask[i, pad:, :n] = 0.0 # every new token sees the sequence's past
mask[i, pad:, past + pad :] = np.triu(
np.full((len(new), len(new)), MASKED), 1
)
mask[i, np.arange(pad), past + np.arange(pad)] = (
0.0 # padding sees itself
)
past_kv = []
for layer in range(caches[0].num_layers):
keys = np.zeros(
(batch, past, *caches[0].read(layer)[0].shape[1:]), np.float32
)
values = np.zeros_like(keys)
for i, cache in enumerate(caches):
k, v = cache.read(layer)
keys[i, : len(k)], values[i, : len(v)] = k, v
past_kv.append(
(Tensor(keys, device=device), Tensor(values, device=device))
)
logits, new_kv = step(
Tensor(tokens, device=device),
Tensor(positions, device=device),
Tensor(mask, device=device),
past_kv,
)
for layer, (k, v) in enumerate(new_kv):
k, v = k.to_numpy(), v.to_numpy()
for i, (cache, new) in enumerate(zip(caches, new_tokens, strict=True)):
pad = q - len(new)
cache.append(layer, k[i, pad:], v[i, pad:])
return logits.to_numpy()
Saved to minillm/step.py.
Check the step against the model from Section 1.6 by generating greedily for two prompts of different lengths in the same steps:
Show greedy_reference 12 lines
full = load_model(config, weights, DEVICE).compile(
TensorType(DType.int64, ("batch", "seq"), device=DEVICE)
)
def greedy_reference(ids: list[int], n: int) -> list[int]:
"""Generates ``n`` tokens greedily with the full model, no cache."""
ids = list(ids)
for _ in range(n):
logits = full(Tensor(np.array([ids]), device=DEVICE)).to_numpy()
ids.append(int(logits[0, -1].argmax()))
return ids[-n:]
prompts = [
tokenizer([t])["input_ids"][0]
for t in ["The capital of France is", "Once upon a time, there was a"]
]
caches = [ContiguousCache(config) for _ in prompts]
outputs = [[int(t)] for t in run_step(step, caches, prompts, DEVICE).argmax(-1)]
for _ in range(11):
next_ids = run_step(
step, caches, [[o[-1]] for o in outputs], DEVICE
).argmax(-1)
for out, t in zip(outputs, next_ids):
out.append(int(t))
for prompt, out in zip(prompts, outputs):
print(out == greedy_reference(prompt, 12), repr(tokenizer.decode(out)))
assert out == greedy_reference(prompt, 12)
True ' Paris. Paris is the largest city in France and the capital'
True ' young woman named Sarah who was struggling to find her place in'
Both sequences match the reference, even though they have different lengths and shared every step.
Here is the mask run_step builds for a mixed step: sequence 0 has 6
tokens cached and decodes 1 new token, while sequence 1 has nothing cached
and prefills a 4-token prompt. A wrapper around step records the inputs it
receives. Blue cells may attend; hover over any cell:
recorded = {}
def recording_step(tokens, positions, mask, past):
recorded.update(
tokens=tokens.to_numpy(),
positions=positions.to_numpy(),
mask=mask.to_numpy(),
)
return step(tokens, positions, mask, past)
caches = [ContiguousCache(config), ContiguousCache(config)]
run_step(
step, caches[:1], [prompts[1][:6]], DEVICE
) # sequence 0 now has 6 cached tokens
run_step(recording_step, caches, [[prompts[1][6]], prompts[0][:4]], DEVICE)
array([[ -2.6938362, -9.929393 , -7.831539 , ..., -7.389453 ,
-0.1006913, -14.242568 ],
[ 13.415854 , 5.296691 , 10.037319 , ..., 2.9147892,
6.7043333, 4.4411554]], shape=(2, 49152), dtype=float32)
Show the code 60 lines
mask, positions = recorded["mask"], recorded["positions"]
past = mask.shape[2] - mask.shape[1]
columns = [f"cached position {j}" for j in range(past)] + [
f"new position {j}" for j in range(mask.shape[1])
]
fig = make_subplots(
rows=1,
cols=2,
horizontal_spacing=0.26,
subplot_titles=[
"sequence 0: 6 cached, decodes 1",
"sequence 1: nothing cached, prefills 4",
],
)
for i, new_length in enumerate([1, 4]):
pad = mask.shape[1] - new_length # left padding, as in run_step
rows = [
f"padding row {r}"
if r < pad
else f"new token at position {int(positions[i, r])}"
for r in range(mask.shape[1])
]
text = [
[
f"{rows[r]} -> {columns[c]}: "
+ ("visible" if mask[i, r, c] == 0 else "masked")
for c in range(mask.shape[2])
]
for r in range(mask.shape[1])
]
fig.add_trace(
go.Heatmap(
z=(mask[i] == 0).astype(int),
zmin=0,
zmax=1,
showscale=False,
xgap=2,
ygap=2,
colorscale=[
[0, viz.GRID],
[0.5, viz.GRID],
[0.5, viz.BLUE],
[1, viz.BLUE],
],
x=columns,
y=rows,
customdata=text,
hovertemplate="%{customdata}<extra></extra>",
),
row=1,
col=i + 1,
)
fig.update_xaxes(showgrid=False, tickangle=-45)
fig.update_yaxes(showgrid=False, autorange="reversed")
fig.update_layout(
title="One step's attention mask for a batch of two sequences",
height=440,
margin=dict(l=8),
)
viz.show(fig)
Sequence 0’s single new token sits in the last row, after three padding rows that each see only themselves. It sees its 6 cached positions and itself. Sequence 1 has no cached tokens, so its 6 cached positions are masked in every row. They hold zeros, there only because the batch’s longest past is 6. Its 4 new tokens form the triangle of the causal mask.
A first engine: one request at a time
A request carries its prompt, how many tokens it may generate, and timestamps for the latency metrics from Section 1.1, such as time to first token (TTFT). Every engine in this chapter serves the same workload: eight requests that arrive at once, each asking for between 8 and 32 new tokens:
from dataclasses import dataclass, field
@dataclass
class Request:
prompt: list[int]
max_new_tokens: int
output: list[int] = field(default_factory=list)
arrived: float = 0.0
first_token: float | None = None
finished: float | None = None
@property
def done(self) -> bool:
return len(self.output) >= self.max_new_tokens
Saved to minillm/step.py.
import time
from collections.abc import Mapping
Tokenizer = Callable[[list[str]], Mapping[str, list[list[int]]]]
WORKLOAD = [
("Tell me about Paris.", 32),
("Write a haiku about the sea.", 24),
("What is 2 + 2?", 8),
("List three colors.", 16),
("Explain gravity in one sentence.", 32),
("Name a fruit.", 8),
("Describe a cat.", 24),
("Say hello in French.", 16),
]
def make_workload(tokenizer: Tokenizer) -> list[Request]:
"""Returns the chapter's eight requests, all arriving now."""
now = time.perf_counter()
return [
Request(tokenizer([text])["input_ids"][0], n, arrived=now)
for text, n in WORKLOAD
]
Saved to minillm/step.py.
The simplest engine serves requests in arrival order, each to completion, one at a time:
Show report 9 lines
def report(requests: list[Request]) -> None:
ttft = [(r.first_token - r.arrived) * 1000 for r in requests]
total = max(r.finished for r in requests) - min(r.arrived for r in requests)
tokens = sum(len(r.output) for r in requests)
print(
f"TTFT: first request {ttft[0]:.0f} ms, "
f"median {np.median(ttft):.0f} ms, worst {max(ttft):.0f} ms"
)
print(f"{tokens} tokens in {total:.1f} s = {tokens / total:.1f} tokens/s")
def serve_one_at_a_time(requests: list[Request]) -> None:
for request in requests:
cache = ContiguousCache(config)
logits = run_step(step, [cache], [request.prompt], DEVICE)
while True:
request.output.append(int(logits[0].argmax()))
request.first_token = request.first_token or time.perf_counter()
if request.done:
break
logits = run_step(step, [cache], [[request.output[-1]]], DEVICE)
request.finished = time.perf_counter()
sequential = make_workload(tokenizer)
serve_one_at_a_time(sequential)
report(sequential)
TTFT: first request 296 ms, median 16553 ms, worst 30666 ms
160 tokens in 35.6 s = 4.5 tokens/s
Eight users sent their requests at the same moment. The first one got a token almost at once. The last one waited for every request ahead of it to finish, so its time to first token is most of the total run time. Section 4.3 measured a decoder layer’s decode step for 32 sequences at about 1.2 to 1.3 times the cost of a step for one, on the Mac the book was built on, so this engine leaves most of the hardware’s capacity unused.
The engines in this chapter favor clarity over speed. On every step,
run_step packs each layer’s past keys and values into NumPy arrays, copies
them to the device, and copies the new ones back, so a step costs more here
than the cached decode step in
Section 1.8. Compare this
chapter’s timings with each other, not with Chapter 1’s.
Section 5.2 runs
several requests in each step.
The handbook’s LLM inference metrics page explains why serving systems track TTFT percentiles as well as averages.
Exercises
- Serve the workload in order of
max_new_tokens, shortest first. What happens to the median and worst TTFT? What would a user with a long request think of this policy? - Run
run_stepwith three prompts in one call and check each againstgreedy_reference. - Print the mask
run_stepbuilds for two sequences with 2 and 4 new tokens and past lengths 3 and 1. Check each row by hand.
Next
Section 5.2 replaces the one-at-a-time loop with continuous batching.
5.2 Continuous batching
The engine from Section 5.1 ran one sequence per step, while Section 4.3 showed that a decode step for many sequences costs little more than a step for one. Batching uses that spare capacity, and the way you batch decides who waits. This section adds a scheduler to the engine and compares three policies on the same workload.
Show the setup 31 lines
import logging
import warnings
import numpy as np
import plotly.graph_objects as go
from max.pipelines import AutoTokenizer
from minillm import viz
from minillm.config import MODEL_ID, LlamaConfig
from minillm.device import pick_device
from minillm.model import load_model
from minillm.step import (
WORKLOAD,
ContiguousCache,
Request,
StepModel,
compile_step,
make_workload,
run_step,
)
from minillm.weights import load_checkpoint
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
config = LlamaConfig.from_pretrained()
tokenizer = AutoTokenizer(MODEL_ID)
step = compile_step(
load_model(config, load_checkpoint(), DEVICE, StepModel), config, DEVICE
)
The engine
The engine holds two lists: requests waiting to start and requests
running. Each call to step does one model step:
- If there is room in the batch and requests are waiting, it admits them and runs their prefill: all their prompt tokens, in one batched step.
- Otherwise it runs a decode step: one new token for every running request, in one batched step.
Finished requests leave the batch after every step. That is continuous
batching: the batch is re-formed at every step, so a waiting request starts
as soon as a slot frees up. Setting continuous=False gives static
batching instead: a batch is admitted together and must finish completely
before the next one starts.
Running prefill and decode in separate steps is a simple policy that gives
new requests priority, at the cost of pausing the running requests’ decode
for a step. Production engines can also split a long prompt into chunks
(chunked prefill) and run prefill chunks and decodes in the same step, so
one big prefill doesn’t stall everyone. MAX Serve chunks long prefills by
default (--enable-chunked-prefill) and batches them with decodes when you
pass --enable-in-flight-batching.
Show Engine 86 lines
from __future__ import annotations
import time
from collections import deque
from collections.abc import Callable
from max.driver import Device
from minillm.config import LlamaConfig
from minillm.step import ContiguousCache, Request, Step, run_step
class Engine:
"""Serves requests with a batching scheduler and greedy decoding."""
def __init__(
self,
step: Step,
config: LlamaConfig,
device: Device,
max_batch: int = 4,
continuous: bool = True,
new_cache: Callable[[LlamaConfig], object] = ContiguousCache,
) -> None:
self.step_fn = step
self.config = config
self.device = device
self.max_batch = max_batch
self.continuous = continuous
self.new_cache = new_cache
self.waiting: deque[Request] = deque()
self.running: list[tuple[Request, object]] = []
self.steps = 0
def add(self, request: Request) -> None:
self.waiting.append(request)
def has_work(self) -> bool:
return bool(self.waiting or self.running)
def step(self) -> None:
room = self.max_batch - len(self.running)
can_admit = (
self.waiting and room > 0 and (self.continuous or not self.running)
)
if can_admit:
admitted = [
self.waiting.popleft()
for _ in range(min(room, len(self.waiting)))
]
batch = [(r, self.new_cache(self.config)) for r in admitted]
inputs = [self.prefill_tokens(r, c) for r, c in batch]
else:
batch = self.running
inputs = [[r.output[-1]] for r, _ in batch]
logits = run_step(
self.step_fn, [c for _, c in batch], inputs, self.device
)
self.steps += 1
now = time.perf_counter()
for (request, _), token in zip(batch, logits.argmax(-1), strict=True):
request.output.append(int(token))
if request.first_token is None:
request.first_token = now
if request.done:
request.finished = now
if can_admit:
for request, cache in batch:
self.prefilled(request, cache)
self.running += batch
self.running = [(r, c) for r, c in self.running if not r.done]
self.release_finished(batch)
def prefill_tokens(self, request: Request, cache: object) -> list[int]:
"""Returns the prompt tokens to compute; Section 5.4 skips cached ones."""
return request.prompt
def prefilled(self, request: Request, cache: object) -> None:
"""Runs after a request's prefill; Section 5.4 uses it."""
def release_finished(self, batch: list) -> None:
"""Frees the caches of finished requests; Section 5.3 needs this."""
def run(self) -> None:
while self.has_work():
self.step()
Saved to minillm/engine.py.
A workload
Serve the eight requests from
Section 5.1 under three
policies. A recording subclass of Engine notes which request holds each batch
slot at every step, for the figures below:
Show RecordingEngine, run_policy, summarize 51 lines
class RecordingEngine(Engine):
"""Records, per step, the request in each batch slot and what it did."""
def step(self) -> None:
before = {id(r): len(r.output) for r in self.requests}
super().step()
ran = [r for r in self.requests if len(r.output) > before[id(r)]]
for r in ran:
if r not in self.slots:
self.slots[self.slots.index(None)] = r
row = []
for r in self.slots:
if r is None:
row.append(None)
elif r not in ran:
row.append((self.requests.index(r), "waits"))
elif before[id(r)] == 0:
row.append((self.requests.index(r), "prefill"))
else:
row.append((self.requests.index(r), "decode"))
self.history.append(row)
self.slots = [None if r is None or r.done else r for r in self.slots]
def run_policy(max_batch: int, continuous: bool) -> tuple[list, Engine]:
engine = RecordingEngine(
step, config, DEVICE, max_batch=max_batch, continuous=continuous
)
engine.requests = make_workload(tokenizer)
engine.slots = [None] * max_batch
engine.history = []
for r in engine.requests:
engine.add(r)
engine.run()
return engine.requests, engine
def summarize(name: str, requests: list[Request], engine: Engine) -> dict:
ttft = [(r.first_token - r.arrived) * 1000 for r in requests]
total = max(r.finished for r in requests) - min(r.arrived for r in requests)
tokens = sum(len(r.output) for r in requests)
print(
f"{name:22} {engine.steps:4} steps {tokens / total:5.1f} tokens/s "
f"TTFT median {np.median(ttft):6.0f} ms worst {max(ttft):6.0f} ms"
)
return {
"ttft": ttft,
"outputs": [r.output for r in requests],
"requests": requests,
"history": engine.history,
}
results = {}
for name, max_batch, continuous in [
("one at a time", 1, True),
("static batches of 4", 4, False),
("continuous, batch 4", 4, True),
]:
results[name] = summarize(name, *run_policy(max_batch, continuous))
one at a time 160 steps 2.9 tokens/s TTFT median 27801 ms worst 50743 ms
static batches of 4 64 steps 3.2 tokens/s TTFT median 6450 ms worst 12693 ms
continuous, batch 4 50 steps 9.3 tokens/s TTFT median 2151 ms worst 8903 ms
Batching changes only the order of the work, so greedy decoding must produce the same tokens for every request under every policy:
reference = results["one at a time"]["outputs"]
print(all(r["outputs"] == reference for r in results.values()))
assert all(r["outputs"] == reference for r in results.values())
True
Show the code 41 lines
steps = []
for name, r in results.items():
start = min(q.arrived for q in r["requests"])
rows = [f"request {i} ({n} tokens)" for i, (_, n) in enumerate(WORKLOAD)]
wait = [(q.first_token - q.arrived) * 1000 for q in r["requests"]]
gen = [(q.finished - q.first_token) * 1000 for q in r["requests"]]
first = [(q.first_token - start) * 1000 for q in r["requests"]]
steps.append(
[
go.Bar(
y=rows,
x=wait,
base=[(q.arrived - start) * 1000 for q in r["requests"]],
orientation="h",
name="waiting, then prefill",
marker_color=viz.MUTED,
legendgroup="wait",
hovertemplate="%{y}: first token after %{x:.0f} ms<extra></extra>",
),
go.Bar(
y=rows,
x=gen,
base=first,
orientation="h",
name="generating",
marker_color=viz.BLUE,
legendgroup="gen",
hovertemplate="%{y}: generating for %{x:.0f} ms<extra></extra>",
),
]
)
fig = go.Figure(
layout=dict(
title="Each request's life under each policy",
height=460,
barmode="overlay",
xaxis_title="milliseconds since the requests arrived",
yaxis=dict(autorange="reversed", showgrid=False),
)
)
viz.show(viz.with_slider(fig, steps, list(results), "policy: ", active=2))
Slide between the policies. One at a time, each request waits for every request before it. Static batches run in two rounds, and the second round can’t start until the slowest request of the first finishes. Continuous batching starts a waiting request as soon as any slot frees up.
The same runs from the engine’s side: each column is one engine step, each row one of the four batch slots, and the color says which request holds the slot. “P” marks a request’s prefill step, and a dot marks a running request that sat out the step because another request was prefilling:
Show the code 54 lines
policies = ["static batches of 4", "continuous, batch 4"]
colors = [viz.GRID] + viz.SERIES # empty, then one color per request
scale = [
[edge, c] for i, c in enumerate(colors) for edge in (i / 9, (i + 1) / 9)
]
steps = []
for name in policies:
history = np.array(
[
[-1 if s is None else s[0] for s in row]
for row in results[name]["history"]
]
).T
marks = {"prefill": "P", "waits": "·", "decode": ""}
text = [
["" if s is None else marks[s[1]] for s in row]
for row in results[name]["history"]
]
hover = [
[
f"step {t + 1}, slot {k}: "
+ ("empty" if s is None else f"request {s[0]}, {s[1]}")
for t, s in enumerate(column)
]
for k, column in enumerate(zip(*results[name]["history"]))
]
steps.append(
[
go.Heatmap(
z=history,
x=np.arange(1, history.shape[1] + 1),
y=[f"slot {k}" for k in range(history.shape[0])],
zmin=-1.5,
zmax=7.5,
colorscale=scale,
showscale=False,
xgap=1,
ygap=3,
text=np.array(text).T,
texttemplate="%{text}",
customdata=hover,
hovertemplate="%{customdata}<extra></extra>",
)
]
)
fig = go.Figure(
layout=dict(
title="Which request holds each batch slot at every engine step",
height=320,
xaxis=dict(title="engine step", showgrid=False),
yaxis=dict(autorange="reversed", showgrid=False),
)
)
viz.show(viz.with_slider(fig, steps, policies, "policy: ", active=1))
Both batching policies cut the number of model steps, from 160 to 64 for static batches and 50 for continuous batching, so they finish the workload sooner and raise throughput. They differ in who waits:
- Static batching admits four requests and runs until the longest of them finishes. The second group waits for that, even after the short requests in the first group are done and their slots sit empty: the grey holes in the slot chart.
- Continuous batching refills a slot as soon as a request finishes, so waiting requests start sooner, and the batch stays full while any request is waiting.
Production engines, MAX Serve among them, use continuous batching with much larger batches, bounded by the memory the KV cache needs. That bound is the subject of Section 5.3. The handbook’s static, dynamic, and continuous batching page compares the policies in production terms.
Exercises
- Run continuous batching with
max_batch2, 4, and 8. How do throughput and worst-case TTFT change? - Make requests arrive over time: add each one after a few engine steps instead of all at the start. Which policy handles bursts better?
- Implement a mixed step: admit new requests’ prefills in the same step as running requests’ decodes. What does it cost the running requests?
Next
Section 5.3 replaces the contiguous cache with a paged one, so the batch can grow without wasting memory.
5.3 A paged KV cache
With continuous batching, memory limits how many requests can be in flight:
every request needs room for its keys and values, and
Section 1.8
measured 45 KiB per token for SmolLM2-135M in float32. The question is how to
hand out that memory. A server doesn’t know how long a reply will be when it
starts, so the simplest scheme reserves room for the maximum length, and most of
that room goes unused. (ContiguousCache avoids the reservation by copying its
arrays on every append, which costs time instead.) This section replaces the
contiguous cache with a paged one, the idea behind vLLM’s PagedAttention and
MAX Serve’s paged KV cache.
Show the setup 24 lines
import logging
import warnings
import numpy as np
import plotly.graph_objects as go
from max.pipelines import AutoTokenizer
from minillm import viz
from minillm.config import MODEL_ID, LlamaConfig
from minillm.device import pick_device
from minillm.engine import Engine
from minillm.model import load_model
from minillm.step import Request, StepModel, compile_step, make_workload
from minillm.weights import load_checkpoint
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
config = LlamaConfig.from_pretrained()
tokenizer = AutoTokenizer(MODEL_ID)
step = compile_step(
load_model(config, load_checkpoint(), DEVICE, StepModel), config, DEVICE
)
Blocks and block tables
Paging works like virtual memory. The cache memory is one pool of
fixed-size blocks, each holding the keys and values of block_size
tokens. A sequence owns a list of block numbers, its block table, and
takes a new block from the pool only when its last block fills. A
sequence’s blocks needn’t be next to each other, so free memory never breaks
into gaps too small to use; the only waste is the unfilled part of each
sequence’s last block. A finished sequence returns its blocks to the pool at
once.
Show BlockPool, PagedCache 79 lines
from __future__ import annotations
import numpy as np
from minillm.config import LlamaConfig
class BlockPool:
"""Fixed-size blocks of KV cache memory, shared by all sequences."""
def __init__(
self, config: LlamaConfig, num_blocks: int, block_size: int = 16
) -> None:
self.block_size = block_size
shape = (
num_blocks,
block_size,
config.num_key_value_heads,
config.head_dim,
)
self.keys = [
np.zeros(shape, np.float32) for _ in range(config.num_hidden_layers)
]
self.values = [
np.zeros(shape, np.float32) for _ in range(config.num_hidden_layers)
]
self.free = list(range(num_blocks))
@property
def num_blocks(self) -> int:
return len(self.keys[0])
def allocate(self) -> int:
if not self.free:
raise MemoryError("the KV cache pool is full")
return self.free.pop()
def available(self) -> int:
"""Returns how many blocks ``allocate`` can still hand out."""
return len(self.free)
def release(self, blocks: list[int]) -> None:
self.free.extend(blocks)
class PagedCache:
"""One sequence's keys and values, stored in pool blocks."""
def __init__(self, pool: BlockPool) -> None:
self.pool = pool
self.blocks: list[int] = []
self.length = 0
@property
def num_layers(self) -> int:
return len(self.pool.keys)
def read(self, layer: int) -> tuple[np.ndarray, np.ndarray]:
keys = self.pool.keys[layer][self.blocks].reshape(
-1, *self.pool.keys[layer].shape[2:]
)
values = self.pool.values[layer][self.blocks].reshape(keys.shape)
return keys[: self.length], values[: self.length]
def append(self, layer: int, keys: np.ndarray, values: np.ndarray) -> None:
size, start = self.pool.block_size, self.length
if layer == 0: # every layer stores the same positions
while len(self.blocks) * size < start + len(keys):
self.blocks.append(self.pool.allocate())
for i in range(len(keys)):
block, slot = self.blocks[(start + i) // size], (start + i) % size
self.pool.keys[layer][block, slot] = keys[i]
self.pool.values[layer][block, slot] = values[i]
if layer == self.num_layers - 1:
self.length += len(keys)
def release(self) -> None:
self.pool.release(self.blocks)
self.blocks = []
Saved to minillm/paged.py.
read gathers a sequence’s blocks into one array for the step model. That copy
keeps the code simple. Production engines skip it: their attention kernel reads
the blocks in place through the block table, which takes a custom kernel like
the ones Section 3.1 shows how
to write. The engine needs three changes. It admits waiting requests in arrival
order while the pool has blocks for their prompts. When a decode step needs new
blocks and the pool is empty, it preempts the newest running request: it frees
that request’s blocks and puts it back at the front of the queue, to recompute
its prompt and the tokens it has generated so far when it runs again. And it
returns a finished request’s blocks to the pool.
Show PagedEngine 68 lines
from minillm.engine import Engine
from minillm.step import Request
class PagedEngine(Engine):
"""An engine whose requests share one pool of KV cache blocks."""
def __init__(self, *args, pool: BlockPool, **kwargs) -> None:
super().__init__(*args, new_cache=lambda _: PagedCache(pool), **kwargs)
self.pool = pool
self.preemptions = 0
def step(self) -> None:
# Admit waiting requests in arrival order while their tokens so far fit.
spare, admitted = self.pool.available(), 0
for request in self.waiting:
if self.blocks_needed(request) > spare:
break
spare -= self.blocks_needed(request)
admitted += 1
held = [self.waiting.pop() for _ in range(len(self.waiting) - admitted)]
held.reverse() # pop() took them from the back
prefill = (
self.waiting
and len(self.running) < self.max_batch
and (self.continuous or not self.running)
)
if not prefill and not self.running and held:
raise MemoryError(
"a waiting request needs more blocks than the pool has"
)
preempted = [] if prefill else self.preempt_until_decode_fits()
super().step()
# Preempted requests go to the front of the queue, as in MAX.
self.waiting.extend(held)
self.waiting.extendleft(reversed(preempted))
def preempt_until_decode_fits(self) -> list[Request]:
"""Frees the newest requests' blocks until every running request can
store its next token, and returns them to recompute later."""
preempted = []
while self.blocks_to_grow() > self.pool.available():
if len(self.running) == 1:
raise MemoryError(
"a request needs more blocks than the pool has"
)
request, cache = self.running.pop()
cache.release()
preempted.insert(0, request)
self.preemptions += 1
return preempted
def blocks_to_grow(self) -> int:
size = self.pool.block_size
return sum(c.length + 1 > len(c.blocks) * size for _, c in self.running)
def blocks_needed(self, request: Request) -> int:
# A preempted request recomputes the tokens it already generated.
tokens = len(request.prompt) + len(request.output)
return -(-tokens // self.pool.block_size)
def prefill_tokens(self, request: Request, cache: PagedCache) -> list[int]:
return request.prompt + request.output
def release_finished(self, batch: list) -> None:
for request, cache in batch:
if request.done:
cache.release()
Saved to minillm/paged.py.
Check it
Paging only changes where the keys and values are stored. Serve the workload from Section 5.2 with a paged cache and compare the tokens:
contiguous_requests = make_workload(tokenizer)
engine = Engine(step, config, DEVICE, max_batch=4)
for r in contiguous_requests:
engine.add(r)
engine.run()
pool = BlockPool(config, num_blocks=32, block_size=16)
paged_requests = make_workload(tokenizer)
paged = PagedEngine(step, config, DEVICE, max_batch=4, pool=pool)
for r in paged_requests:
paged.add(r)
paged.run()
same = [r.output for r in paged_requests] == [
r.output for r in contiguous_requests
]
print("same tokens:", same)
print("blocks free after the run:", len(pool.free), "of", pool.num_blocks)
assert same and len(pool.free) == pool.num_blocks
same tokens: True
blocks free after the run: 32 of 32
The tokens match, and every block is back in the pool after the run. To watch the block tables, run the workload again on a smaller pool of 24 blocks of 8 tokens, and record every running request’s block table after each engine step:
class RecordingEngine(PagedEngine):
def step(self) -> None:
super().step()
self.snapshots.append(
[
(requests.index(r), list(c.blocks), c.length)
for r, c in self.running
]
)
requests = make_workload(tokenizer)
recorder = RecordingEngine(
step, config, DEVICE, max_batch=4, pool=BlockPool(config, 24, 8)
)
recorder.snapshots = []
for r in requests:
recorder.add(r)
recorder.run()
print("preemptions:", recorder.preemptions)
preemptions: 0
On the left, each running request’s block table lists its logical blocks in order, with the fill of its last block. On the right is the pool, and each line links a logical block to the physical block that holds it. Slide through the steps:
Show pool_xy 94 lines
size, pool_x = recorder.pool.block_size, 9.0
def pool_xy(block: int) -> tuple[float, float]:
return pool_x + block % 8, (block // 8) * 1.5
frames = []
for snapshot in recorder.snapshots:
owner = {b: i for i, blocks, _ in snapshot for b in blocks}
xs, ys = zip(*(pool_xy(b) for b in range(recorder.pool.num_blocks)))
traces = [
go.Scatter(
x=xs,
y=ys,
mode="markers+text",
marker=dict(
symbol="square",
size=26,
color=[
viz.SERIES[owner[b]] if b in owner else viz.GRID
for b in range(recorder.pool.num_blocks)
],
),
text=[str(b) for b in range(recorder.pool.num_blocks)],
textfont=dict(color=viz.INK),
hovertext=[
f"physical block {b}: "
+ (f"request {owner[b]}" if b in owner else "free")
for b in range(recorder.pool.num_blocks)
],
hoverinfo="text",
)
]
for row, (i, blocks, length) in enumerate(snapshot):
fills = [min(size, length - j * size) for j in range(len(blocks))]
traces += [
go.Scatter(
x=[
x
for j, b in enumerate(blocks)
for x in (j, pool_xy(b)[0], None)
],
y=[y for b in blocks for y in (row, pool_xy(b)[1], None)],
mode="lines",
line=dict(color=viz.SERIES[i], width=1),
opacity=0.6,
hoverinfo="skip",
),
go.Scatter(
x=[-1.2, *range(len(blocks))],
y=[row] * (len(blocks) + 1),
mode="markers+text",
marker=dict(
symbol="square",
size=[0] + [26] * len(blocks),
color=viz.SERIES[i],
),
text=[f"request {i}"]
+ ["" if f == size else f"{max(f, 0)}/{size}" for f in fills],
textfont=dict(color=viz.INK),
hovertext=[f"request {i}: {length} tokens"]
+ [
f"request {i}, logical block {j} -> physical block {b}, "
f"{max(f, 0)} of {size} slots filled"
for j, (b, f) in enumerate(zip(blocks, fills))
],
hoverinfo="text",
),
]
frames.append(traces)
fig = go.Figure(
go.Scatter(
x=[2.5, pool_x + 3.5],
y=[-0.9, -0.9],
mode="text",
text=["block tables (logical blocks)", "pool (physical blocks)"],
textfont=dict(color=viz.INK_SECONDARY),
hoverinfo="skip",
),
layout=dict(
title="Block tables and the pool after each engine step",
height=400,
showlegend=False,
xaxis=dict(visible=False, range=[-2.6, pool_x + 8]),
yaxis=dict(visible=False, autorange="reversed"),
),
)
labels = [str(i + 1) for i in range(len(frames))]
viz.show(
viz.with_slider(
fig, frames, labels, "after engine step ", active=len(frames) // 3
)
)
Blocks are handed out from wherever they are free, so a request’s blocks end up scattered across the pool, and a finished request’s blocks go straight to the next request. Only the block table records their order. This pool was big enough that no request was preempted; exercise 1 shrinks it until some are.
Why it matters: memory
Compare the memory each scheme ties up for a batch of four of these
requests. A contiguous cache can’t grow in place, so when a request starts
it must reserve room for the longest the request could get. That is
SmolLM2’s full 8192-token context, or at best the request’s max_tokens
limit, which API clients often set well above the length of the replies they
get; the table below assumes 512. Paging allocates only what each sequence
holds, rounded up to whole blocks of \(B\) tokens:
\[\text{blocks for } n \text{ tokens} = \left\lceil \frac{n}{B} \right\rceil\]
per_token = (
2
* config.num_hidden_layers
* config.num_key_value_heads
* config.head_dim
* 4
)
batch = 4
actual = [len(r.prompt) + len(r.output) for r in paged_requests]
reserve_max_context = batch * config.max_position_embeddings
reserve_max_tokens = batch * (max(len(r.prompt) for r in paged_requests) + 512)
paged_tokens = sum(sorted(-(-n // 16) * 16 for n in actual)[-batch:])
rows = [
(name, f"{tokens:,}", f"{tokens * per_token / 2**20:.1f} MiB")
for name, tokens in [
("reserve full context per request", reserve_max_context),
("reserve max_tokens=512 per request", reserve_max_tokens),
("paged, 16-token blocks (4 largest)", paged_tokens),
]
]
header = ("scheme", "tokens reserved", "memory")
widths = [max(len(row[i]) for row in [header, *rows]) for i in range(3)]
for name, tokens, memory in [header, *rows]:
print(f"{name:{widths[0]}} {tokens:>{widths[1]}} {memory:>{widths[2]}}")
scheme tokens reserved memory
reserve full context per request 32,768 1440.0 MiB
reserve max_tokens=512 per request 2,080 91.4 MiB
paged, 16-token blocks (4 largest) 160 7.0 MiB
Show the code 25 lines
schemes = {
"reserve full context": reserve_max_context,
"reserve max_tokens=512": reserve_max_tokens,
"paged, 16-token blocks": paged_tokens,
}
fig = go.Figure(
go.Bar(
y=list(schemes),
x=[t * per_token / 2**20 for t in schemes.values()],
orientation="h",
marker_color=[viz.ORANGE, viz.YELLOW, viz.BLUE],
text=[f"{t * per_token / 2**20:,.1f} MiB" for t in schemes.values()],
textposition="outside",
cliponaxis=False,
hovertemplate="%{y}: %{x:,.1f} MiB<extra></extra>",
)
)
fig.update_layout(
title="KV cache memory for a batch of 4 of these requests",
height=300,
margin=dict(r=96),
xaxis=dict(title="MiB (log scale)", type="log", dtick=1),
yaxis=dict(autorange="reversed", showgrid=False),
)
viz.show(fig)
Saved memory becomes batch capacity. In the memory that one full-context reservation needs (8192 tokens), a paged cache fits about two hundred of these requests at roughly 40 tokens each, which is why most serving engines page their KV cache. The block size is a trade-off: bigger blocks waste more space in each sequence’s last block, and smaller ones mean longer block tables and more bookkeeping per step.
MAX Serve’s paged cache works the same way, and its attention kernels read the
pages through the block table directly. Its scheduler also preempts a running
request when the pool runs out of pages, and logs a message that suggests giving
the cache more memory. max serve exposes the settings. --kv-cache-page-size
sets the block size, 128 tokens by default. --device-memory-utilization sets
the fraction of device memory the process may use, 0.95 by default, and the KV
cache pool gets what is left after the weights and other allocations. The pool
size and --max-batch-size together limit how many requests run at once. The
handbook’s
PagedAttention
page describes the algorithm, and
GPU memory for LLMs
shows how to size the pool.
Exercises
- Shrink the pool to 8 blocks and rerun the paged engine. How does the engine behave when memory runs out, and what happens to time to first token?
- Try block sizes 4, 16, and 64. How much memory goes to partly filled last blocks in each case?
- This engine preempts the newest request. Change it to preempt the request that holds the most blocks instead. Which choice finishes the workload in fewer steps, and which treats long requests more fairly?
Next
Section 5.4 lets requests that share a prompt prefix share its blocks too.
5.4 Prefix caching
Real traffic repeats itself. Requests to a chat product start with the same system prompt, and every turn of a conversation resends the turns before it. Those shared tokens get the same keys and values every time, because a token’s keys and values depend only on that token and the tokens before it. Prefix caching computes them once and shares their cache blocks between requests. With the paged cache from Section 5.3, it takes little code.
Show the setup 26 lines
import logging
import time
import warnings
import numpy as np
import plotly.graph_objects as go
from max.pipelines import AutoTokenizer
from minillm import viz
from minillm.config import MODEL_ID, LlamaConfig
from minillm.device import pick_device
from minillm.model import load_model
from minillm.paged import BlockPool, PagedEngine
from minillm.prefix import PrefixCachingEngine, PrefixCachingPool
from minillm.step import Request, StepModel, compile_step
from minillm.weights import load_checkpoint
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
config = LlamaConfig.from_pretrained()
tokenizer = AutoTokenizer(MODEL_ID)
step = compile_step(
load_model(config, load_checkpoint(), DEVICE, StepModel), config, DEVICE
)
Naming blocks by their content
To share a block, the engine must recognize it. A full block is identified by the tokens it holds and every token before it, since those determine its keys and values. A chained hash captures both: each block’s hash combines the previous block’s hash with the block’s own tokens. Two requests share a block exactly when their prompts agree up to the end of that block.
The pool keeps a table from hash to block, counts how many sequences use each block, and frees a block only when no sequence uses it. A block with no users whose contents are still valid stays in the table, in least-recently-used order, so a later request can still reuse it. Such blocks are evicted only when the pool runs out of free blocks.
Only full blocks are shared, and new tokens always go into a sequence’s own next block, so a shared block is never written to.
Show PrefixCachingPool, PrefixCachingEngine 100 lines
from __future__ import annotations
from collections import OrderedDict
from minillm.config import LlamaConfig
from minillm.paged import BlockPool, PagedCache, PagedEngine
from minillm.step import Request
class PrefixCachingPool(BlockPool):
"""A block pool that remembers full prompt blocks for reuse."""
def __init__(
self, config: LlamaConfig, num_blocks: int, block_size: int = 16
) -> None:
super().__init__(config, num_blocks, block_size)
self.refs = [0] * num_blocks
self.by_hash: dict[int, int] = {}
self.hash_of: dict[int, int] = {}
self.unused: OrderedDict[int, None] = OrderedDict() # cached, no users
def allocate(self) -> int:
if self.free:
block = self.free.pop()
elif self.unused:
block, _ = self.unused.popitem(last=False) # evict the oldest
del self.by_hash[self.hash_of.pop(block)]
else:
raise MemoryError("the KV cache pool is full")
self.refs[block] = 1
return block
def available(self) -> int:
# Cached blocks that no request uses can be evicted and reused.
return len(self.free) + len(self.unused)
def release(self, blocks: list[int]) -> None:
# Tail first: eviction then takes a prefix's last block before its
# first, and lookup can still reuse the rest.
for block in reversed(blocks):
self.refs[block] -= 1
if self.refs[block] == 0:
if block in self.hash_of:
self.unused[block] = None
else:
self.free.append(block)
def block_hashes(self, tokens: list[int]) -> list[int]:
hashes, previous = [], 0
for start in range(
0, len(tokens) - self.block_size + 1, self.block_size
):
previous = hash(
(previous, tuple(tokens[start : start + self.block_size]))
)
hashes.append(previous)
return hashes
def lookup(self, prompt: list[int]) -> list[int]:
"""Returns the cached blocks that start ``prompt``, and claims them.
The last prompt token is always left to compute, since its logits
are needed."""
blocks = []
for h in self.block_hashes(prompt[:-1]):
block = self.by_hash.get(h)
if block is None:
break
self.refs[block] += 1
self.unused.pop(block, None)
blocks.append(block)
return blocks
def register(self, prompt: list[int], blocks: list[int]) -> None:
"""Records a sequence's full prompt blocks for later requests."""
# The last block may be partial; only full blocks have hashes.
for h, block in zip(self.block_hashes(prompt), blocks, strict=False):
if h not in self.by_hash:
self.by_hash[h] = block
self.hash_of[block] = h
class PrefixCachingEngine(PagedEngine):
"""A paged engine that skips computing prompt prefixes it has cached."""
def __init__(self, *args, pool: PrefixCachingPool, **kwargs) -> None:
super().__init__(*args, pool=pool, **kwargs)
self.computed_prompt_tokens = 0
def prefill_tokens(self, request: Request, cache: PagedCache) -> list[int]:
tokens = (
request.prompt + request.output
) # output is empty unless preempted
cache.blocks = self.pool.lookup(tokens)
cache.length = len(cache.blocks) * self.pool.block_size
new = tokens[cache.length :]
self.computed_prompt_tokens += len(new)
return new
def prefilled(self, request: Request, cache: PagedCache) -> None:
self.pool.register(request.prompt, cache.blocks)
Saved to minillm/prefix.py.
prefill_tokens and prefilled are the two hooks the engine from
Section 5.2 calls
around each prefill. The base engine returns the whole prompt from the first and
does nothing in the second. The step model already handles the rest: a prefill
with a non-empty past is a step with several new tokens.
A workload with a shared prefix
Eight users ask different questions of the same assistant. Its system prompt is about 150 tokens long; real ones are often much longer:
Show workload 26 lines
SYSTEM = (
"You are a helpful assistant for a travel company. Answer briefly and politely. "
"Recommend destinations in Europe, mention the best season to visit, give one "
"practical tip about transport, and never invent prices. "
) * 4
QUESTIONS = [
"Where should I go in spring?",
"Is Rome good in winter?",
"How do I get around Paris?",
"What about Lisbon?",
"Best city for museums?",
"Where can I hike?",
"Is Berlin expensive?",
"Suggest a beach town.",
]
def workload() -> list[Request]:
now = time.perf_counter()
return [
Request(tokenizer([SYSTEM + q])["input_ids"][0], 8, arrived=now)
for q in QUESTIONS
]
print("prompt tokens per request:", len(workload()[0].prompt))
prompt tokens per request: 163
Serve the workload twice, without and with prefix caching. The cached engine is a recording subclass that notes, for each request’s prefill, how many prompt tokens it computed and which blocks its block table holds, with their user counts:
Show serve, RecordingPrefixEngine 33 lines
def serve(engine) -> list[Request]:
requests = workload()
for r in requests:
engine.add(r)
start = time.perf_counter()
engine.run()
ttft = [(r.first_token - r.arrived) * 1000 for r in requests]
print(
f"{type(engine).__name__:24} {time.perf_counter() - start:5.1f} s "
f"TTFT median {np.median(ttft):5.0f} ms, worst {max(ttft):5.0f} ms"
)
return requests
class RecordingPrefixEngine(PrefixCachingEngine):
"""Records each request's computed tokens and its blocks' user counts."""
def __init__(self, *args, **kwargs) -> None:
super().__init__(*args, **kwargs)
self.computed, self.tables = {}, {}
def prefill_tokens(self, request: Request, cache: object) -> list[int]:
tokens = super().prefill_tokens(request, cache)
self.computed[id(request)] = len(tokens)
return tokens
def step(self) -> None:
super().step()
for request, cache in self.running:
if id(request) not in self.tables: # just prefilled
self.tables[id(request)] = [
(b, self.pool.refs[b]) for b in cache.blocks
]
plain = serve(
PagedEngine(step, config, DEVICE, max_batch=4, pool=BlockPool(config, 128))
)
prefix_pool = PrefixCachingPool(config, 128)
cached_engine = RecordingPrefixEngine(
step, config, DEVICE, max_batch=4, pool=prefix_pool
)
cached = serve(cached_engine)
PagedEngine 5.1 s TTFT median 2192 ms, worst 3499 ms
RecordingPrefixEngine 4.3 s TTFT median 1617 ms, worst 2961 ms
total_prompt = sum(len(r.prompt) for r in cached)
print(
f"prompt tokens computed: {cached_engine.computed_prompt_tokens} of {total_prompt} "
f"({cached_engine.computed_prompt_tokens / total_prompt:.0%})"
)
print("same tokens:", [r.output for r in cached] == [r.output for r in plain])
assert [r.output for r in cached] == [r.output for r in plain]
assert cached_engine.computed_prompt_tokens < total_prompt
prompt tokens computed: 715 of 1291 (55%)
same tokens: True
Show the code 31 lines
computed = [cached_engine.computed[id(r)] for r in cached]
reused = [len(r.prompt) - c for r, c in zip(cached, computed)]
names = [f"request {i}: {q}" for i, q in enumerate(QUESTIONS)]
fig = go.Figure(
[
go.Bar(
y=names,
x=reused,
name="reused from the cache",
orientation="h",
marker_color=viz.BLUE,
hovertemplate="%{x} tokens reused<extra>%{y}</extra>",
),
go.Bar(
y=names,
x=computed,
name="computed",
orientation="h",
marker_color=viz.ORANGE,
hovertemplate="%{x} tokens computed<extra>%{y}</extra>",
),
]
)
fig.update_layout(
title="Prompt tokens per request with prefix caching",
barmode="stack",
height=420,
xaxis_title="prompt tokens",
yaxis=dict(autorange="reversed", showgrid=False),
)
viz.show(fig)
The block tables show where the reuse comes from. Each row is one request’s block table right after its prefill, each cell one block, labeled with its physical block number. Blue blocks are shared, and ×4 means four requests use the block:
Show the code 44 lines
tables = [cached_engine.tables[id(r)] for r in cached]
size = prefix_pool.block_size
width = max(len(t) for t in tables)
users = np.full((len(tables), width), np.nan)
text = [[""] * width for _ in tables]
hover = [[""] * width for _ in tables]
for i, (request, table) in enumerate(zip(cached, tables)):
for j, (block, refs) in enumerate(table):
users[i, j] = min(refs, 2)
text[i][j] = f"{block}<br>×{refs}" if refs > 1 else str(block)
piece = tokenizer.decode(request.prompt[j * size : (j + 1) * size])
hover[i][j] = (
f"request {i}, logical block {j}: physical block {block}, "
f"{refs} user{'s' * (refs != 1)}<br>{piece!r}"
)
fig = go.Figure(
go.Heatmap(
z=users,
zmin=0.5,
zmax=2.5,
colorscale=[
[0, viz.ORANGE],
[0.5, viz.ORANGE],
[0.5, viz.BLUE],
[1, viz.BLUE],
],
showscale=False,
xgap=3,
ygap=3,
text=text,
texttemplate="%{text}",
customdata=hover,
hovertemplate="%{customdata}<extra></extra>",
x=[f"block {j}" for j in range(width)],
y=[f"request {i}" for i in range(len(tables))],
)
)
fig.update_layout(
title="Block tables after prefill: blue blocks are shared",
height=400,
xaxis=dict(showgrid=False, side="top"),
yaxis=dict(autorange="reversed", showgrid=False),
)
viz.show(fig)
The first batch computes the system prompt once per request, because nothing is cached yet: requests 0 to 3 are prefilled in the same step, so each holds its own copy of the same blocks, and only request 0’s copy enters the table. From the second batch on, each request finds the system prompt’s full blocks in the table, the blue blocks four requests share, and computes only the rest: the system prompt’s last, partly filled block plus its own question. That brings the prompt tokens computed down to 715 of 1,291, or 55%. The outputs are identical, because the reused keys and values are exactly the ones the request would have computed.
The wall-clock gain is modest here: the whole workload took about 4.3 s with prefix caching against 5.1 s without, on this page’s machine (an Apple-silicon Mac with a Metal GPU), because prefilling about 160 prompt tokens on a 135M model is only part of the workload’s time. Treat these numbers as an example: yours depend on your hardware and vary from run to run. The saving grows with the shared prefix. A multi-turn chat resends the whole conversation so far on every turn, often thousands of tokens, and an agent resends long tool descriptions on every call, so for them prefix caching removes most of the prefill.
MAX Serve enables prefix caching by default; --no-enable-prefix-caching turns
it off. The same idea extends across servers: a router can send each request to
the replica that already holds its prefix. The handbook covers
prefix caching
and
inference routing.
Exercises
- Send the workload twice through the same engine. What fraction of prompt tokens does the second pass compute?
- Shrink the pool so cached blocks must be evicted. Which requests lose their prefix, and why does least-recently-used eviction suit chat traffic?
- Why does
lookuphashprompt[:-1]instead of the whole prompt? What would break otherwise?
Next
Section 5.5 speeds up decoding instead: speculative decoding can produce several tokens per step of the large model.
5.5 Speculative decoding
Decoding produces one token per step of the model, and at small batch sizes each step is memory-bound: Section 4.3 showed that a step for several tokens costs little more than a step for one. Speculative decoding uses that spare capacity. A small, fast draft model guesses the next few tokens, and the large target model checks all the guesses in one step. Each guess the target agrees with is an extra token from that one target step, and with greedy decoding the output is exactly what the target alone would produce.
This section implements it with SmolLM2-135M drafting for SmolLM2-360M, a larger model from the same family, and measures when it pays off.
Show the setup 19 lines
import logging
import time
import warnings
import numpy as np
import plotly.graph_objects as go
from max.pipelines import AutoTokenizer
from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.model import load_model
from minillm.step import ContiguousCache, StepModel, compile_step, run_step
from minillm.weights import load_checkpoint
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
TARGET_ID = "HuggingFaceTB/SmolLM2-360M-Instruct"
Two changes to the step model
Verification needs the target’s prediction after every guessed token, not
only the last, so the target’s step returns logits for all positions. When
the target rejects a guess, the keys and values computed for the rejected
tokens must be dropped, so the cache gets a truncate method.
Show AllLogitsStepModel, TruncatableCache, build 30 lines
class AllLogitsStepModel(StepModel):
def forward(self, tokens, positions, mask, past):
x = self.model.embed_tokens(tokens)
new_kv = []
for layer, layer_past in zip(self.model.layers, past, strict=True):
x, kv = layer(x, positions, mask, layer_past)
new_kv.append(kv)
return self.lm_head(self.model.norm(x)), new_kv
class TruncatableCache(ContiguousCache):
def truncate(self, length: int) -> None:
for entry in self.layers:
entry[0], entry[1] = entry[0][:length], entry[1][:length]
def build(model_id: str):
config = LlamaConfig.from_pretrained(model_id)
model = load_model(
config, load_checkpoint(model_id), DEVICE, AllLogitsStepModel
)
return config, compile_step(model, config, DEVICE)
draft_config, draft = build(MODEL_ID)
target_config, target = build(TARGET_ID)
print(
f"draft: {draft_config.num_hidden_layers} layers x {draft_config.hidden_size}, "
f"target: {target_config.num_hidden_layers} layers x {target_config.hidden_size}"
)
draft: 30 layers x 576, target: 32 layers x 960
Both models share a tokenizer, which this kind of speculative decoding requires: the target must read the draft’s token IDs as the same tokens.
The baseline: the target alone
tokenizer = AutoTokenizer(MODEL_ID)
prompt = tokenizer(["The three primary colors are"])["input_ids"][0]
N = 48
def target_alone() -> tuple[list[int], float]:
cache, ids = TruncatableCache(target_config), list(prompt)
start = time.perf_counter()
logits = run_step(target, [cache], [ids], DEVICE)
while True:
ids.append(int(logits[0, -1].argmax()))
if len(ids) - len(prompt) == N:
break
logits = run_step(target, [cache], [[ids[-1]]], DEVICE)
return ids[len(prompt) :], time.perf_counter() - start
reference, reference_time = target_alone()
print(f"{N / reference_time:.1f} tokens/s")
print(tokenizer.decode(reference))
2.5 tokens/s
red, blue, and yellow. These colors can be combined in various ways to create other colors. For example, by mixing red and blue, you get purple. Similarly, by mixing red and yellow, you get orange. By mixing blue
Draft, then verify
Each round:
- The draft model generates
ktokens greedily, one step each. Its first step also feeds in any accepted tokens it hasn’t seen yet. - The target runs one step over the last accepted token plus the
kguesses, and returns its own greedy choice after each of them. - Guesses are accepted from the left for as long as they match the
target’s choice. At the first mismatch, the target’s choice replaces the
guess. If all
kmatch, the target’s choice after the last one comes free. So every round yields between 1 andk + 1tokens. - Both caches are truncated to the accepted tokens, which discards the keys and values of rejected guesses.
Because every emitted token is the target’s own greedy choice, the output is identical to the target alone. Sampling versions accept or reject each guess with a probability test that keeps the output distributed exactly as the target’s own samples would be.
Show speculative 42 lines
def speculative(
k: int, log: list | None = None
) -> tuple[list[int], float, int, int]:
target_cache, draft_cache = (
TruncatableCache(target_config),
TruncatableCache(draft_config),
)
ids = list(prompt)
rounds = accepted = 0
start = time.perf_counter()
logits = run_step(target, [target_cache], [ids], DEVICE)
ids.append(int(logits[0, -1].argmax()))
if log is not None: # for the figure below: where each token came from
log.append(("prefill", ids[-1]))
while len(ids) - len(prompt) < N:
guesses = []
logits = run_step(
draft, [draft_cache], [ids[draft_cache.length :]], DEVICE
)
for i in range(k):
guesses.append(int(logits[0, -1].argmax()))
if i < k - 1:
logits = run_step(draft, [draft_cache], [[guesses[-1]]], DEVICE)
checks = run_step(
target,
[target_cache],
[ids[target_cache.length :] + guesses],
DEVICE,
)[0]
choices = checks[-(k + 1) :].argmax(-1)
n = 0
while n < k and guesses[n] == int(choices[n]):
n += 1
ids += guesses[:n] + [int(choices[n])]
if log is not None:
log += [("accepted", t) for t in guesses[:n]]
log.append(("bonus" if n == k else "corrected", int(choices[n])))
rounds += 1
accepted += n
target_cache.truncate(len(ids) - 1)
draft_cache.truncate(min(draft_cache.length, len(ids) - 1))
return ids[len(prompt) :][:N], time.perf_counter() - start, rounds, accepted
print(
f"{'k':>2} {'same output':>11} {'target steps':>12} {'accepted':>9} {'tokens/s':>9}"
)
print(f"{'-':>2} {'-':>11} {N:>12} {'-':>9} {N / reference_time:>9.1f}")
results = {}
for k in [2, 4, 6]:
out, seconds, rounds, accepted = speculative(k)
results[k] = (rounds, accepted)
print(
f"{k:>2} {str(out == reference):>11} {rounds:>12} {accepted / (rounds * k):>9.0%} {N / seconds:>9.1f}"
)
assert out == reference and rounds < N
k same output target steps accepted tokens/s
- - 48 - 2.5
2 True 21 64% 2.2
4 True 15 58% 1.4
6 True 15 41% 1.2
Look inside the run with k = 4. Each chip is one output token. The first,
in yellow, comes from the target’s prefill. Blue chips are draft guesses the
target accepted, orange ones are the target’s corrections where a guess was
wrong, and green ones are the extra token a round earns when all four
guesses are right. Hover over a chip for its round:
Show the code 23 lines
log = []
speculative(4, log)
log = log[:N]
kinds = {
"prefill": viz.CHIP_COLORS[3],
"accepted": viz.CHIP_COLORS[0],
"corrected": viz.CHIP_COLORS[1],
"bonus": viz.CHIP_COLORS[2],
}
titles, current = [], 0
for kind, _ in log:
titles.append(
"the target's prefill"
if kind == "prefill"
else f"round {current}: {kind}"
)
current += kind != "accepted"
viz.tokens(
[tokenizer.decode([t]) for _, t in log],
[t for _, t in log],
colors=[kinds[kind] for kind, _ in log],
titles=titles,
)
The output matches the target alone, and the target runs far fewer steps: 15 to
21 verification rounds instead of 47 decode steps. Even so, on this page’s
machine (an Apple-silicon Mac with a Metal GPU) every k was slower than the
target alone, at about 1 to 2 tokens/s against about 2.5. Treat these numbers
as an example: yours depend on your hardware and vary from run to run.
A timeline shows where the time goes. Run the target alone and the speculative
loop with k = 4 once more, and time every call to run_step. Each bar is one
step, blue for the target and orange for the draft; hover over a target step in
the second row to see how many tokens its round produced:
Show run_step 19 lines
events, untimed = [], run_step
def run_step(step_fn, caches, new_tokens, device):
start = time.perf_counter()
logits = untimed(step_fn, caches, new_tokens, device)
model = "draft" if step_fn is draft else "target"
events.append((model, start, time.perf_counter(), len(new_tokens[0])))
return logits
try:
target_alone()
baseline_events, events = events, []
round_log = []
speculative(4, round_log)
spec_events = events
finally:
run_step = untimed
Show the code 68 lines
# Tokens each round produced: its accepted guesses plus the target's own token.
per_round, accepted = [], 0
for kind, _ in round_log[1:]:
if kind == "accepted":
accepted += 1
else:
per_round.append(accepted + 1)
accepted = 0
rows = {"target alone": baseline_events, "speculative, k = 4": spec_events}
traces = {
"target": ([], [], [], [], viz.BLUE),
"draft": ([], [], [], [], viz.ORANGE),
}
ends = []
for row, row_events in rows.items():
t0 = row_events[0][1]
verify = 0
for index, (model, start, end, n) in enumerate(row_events):
y, base, width, hover, _ = traces[model]
y.append(row)
base.append(start - t0)
width.append(end - start)
if index == 0:
what = f"target prefill, {n} tokens"
elif model == "draft":
what = "draft step, 1 token"
elif row == "target alone":
what = "target decode step, 1 token"
else:
what = f"target verifies {n} tokens; round {verify + 1} yields {per_round[verify]}"
verify += 1
hover.append(f"{what}: {(end - start) * 1000:.0f} ms")
ends.append((row, row_events[-1][2] - t0))
fig = go.Figure(
[
go.Bar(
y=y,
x=width,
base=base,
orientation="h",
name=f"{model} step",
marker=dict(color=color, line=dict(color=viz.SURFACE, width=1)),
hovertext=hover,
hoverinfo="text",
)
for model, (y, base, width, hover, color) in traces.items()
]
+ [
go.Scatter(
x=[end for _, end in ends],
y=[row for row, _ in ends],
mode="text",
text=[f" {N} tokens in {end:.1f} s" for _, end in ends],
textposition="middle right",
showlegend=False,
hoverinfo="skip",
)
]
)
fig.update_layout(
title=f"Every step of generating {N} tokens, in wall-clock time",
barmode="overlay",
height=300,
xaxis=dict(title="seconds", rangemode="tozero"),
yaxis=dict(autorange="reversed", showgrid=False),
margin=dict(r=140),
)
viz.show(fig)
In the second row, an orange draft step is about as wide as a blue target step, and a target step that verifies five tokens costs about the same as one that decodes a single token, as the cost model below assumes. A round spends four draft steps to earn between one and five tokens, so the steps the target saves are paid back, and more, by the draft.
When it pays off
A round costs k draft steps plus one target step. If a draft step costs a
fraction \(c\) of a target step, and a round yields \(\bar{n}\) tokens on
average, the speedup over the target alone is
\[\text{speedup} \approx \frac{\bar{n}}{1 + k\,c}\]
This assumes the target verifies k + 1 tokens in about the time of a
one-token step. Measure \(c\) for these two models:
def ms_per_decode_step(step, config) -> float:
cache = TruncatableCache(config)
run_step(step, [cache], [prompt], DEVICE)
start = time.perf_counter()
for _ in range(10):
run_step(step, [cache], [[42]], DEVICE)
return (time.perf_counter() - start) / 10 * 1000
c = ms_per_decode_step(draft, draft_config) / ms_per_decode_step(
target, target_config
)
print(f"c = {c:.2f}")
for k, (rounds, accepted) in results.items():
tokens_per_round = (accepted + rounds) / rounds
print(
f"k={k}: {tokens_per_round:.2f} tokens per round -> predicted speedup {tokens_per_round / (1 + k * c):.2f}x"
)
c = 0.69
k=2: 2.29 tokens per round -> predicted speedup 0.96x
k=4: 3.33 tokens per round -> predicted speedup 0.89x
k=6: 3.47 tokens per round -> predicted speedup 0.68x
Plot the predicted speedup against the draft’s relative cost \(c\), using the
acceptance measured for each k. Above the dashed line, speculation is
faster than the target alone; the dotted line marks the \(c\) of these two
models:
Show the code 32 lines
cs = np.logspace(-2, 0, 60)
fig = go.Figure()
for (k, (rounds, accepted)), color in zip(results.items(), viz.SERIES):
per_round = (accepted + rounds) / rounds
fig.add_trace(
go.Scatter(
x=cs,
y=per_round / (1 + k * cs),
name=f"k = {k}",
line_color=color,
hovertemplate=f"k = {k}, c = %{{x:.2f}}: %{{y:.2f}}x<extra></extra>",
)
)
fig.add_hline(y=1, line=dict(color=viz.INK_SECONDARY, dash="dash"))
fig.add_vline(
x=c,
line=dict(color=viz.INK_SECONDARY, dash="dot"),
annotation_text=f"these models, c = {c:.2f}",
annotation_position="top left",
)
fig.update_layout(
title="Predicted speedup of speculative decoding",
height=420,
xaxis=dict(
title="draft step cost / target step cost, c (log scale)",
type="log",
dtick=1,
range=[-2.05, 0.05],
),
yaxis=dict(title="speedup over the target alone", rangemode="tozero"),
)
viz.show(fig)
Here a draft step costs most of a target step, far more than the draft’s 2.7
times fewer parameters suggest. The two models have almost the same depth,
30 and 32 layers, and at this size much of each layer’s time is fixed per-step
work that doesn’t shrink with the layer’s width. Part of that fixed work is this
engine’s own: run_step packs and copies every layer’s keys and values on the
host at each step, so the measured \(c\) includes that host-side packing too. So
the k draft steps in a round cost more than the target steps they save, and
the cost model predicts speculation at or below the speed of the target alone.
The measured runs were slower still, but the conclusion is the same. A draft’s depth matters as much as its parameter count.
Speculative decoding pays off when the draft is cheap (small \(c\)), its
guesses are usually accepted (large \(\bar{n}\)), and verifying k + 1
tokens costs about one target step. A cheap draft can be a 1B model drafting
for a 70B target, or a small head trained on the target’s own hidden states,
as in Medusa, EAGLE, and multi-token prediction (MTP). Cheap verification
holds for memory-bound decoding on GPUs and less so on a busy CPU.
With \(c \approx 0.05\) and the acceptance measured above, the same code would run
about two to three times as fast as the target alone, as the chart shows. MAX
Serve’s speculative decoding uses drafts that read the target model’s hidden
states, such as EAGLE and MTP, selected with --speculative-method. The
handbook’s
speculative decoding
page compares the variants.
Exercises
- Plug different \(c\) values into the formula with the measured acceptance
rates. For which \(c\) does each
kbreak even? - Change the prompt to something more predictable, such as counting from 1
to 50. How do acceptance and the best
kchange? - Make
kadaptive: increase it after a fully accepted round and decrease it after an early rejection. Does it help?
What you built
Chapter 5 is complete. Your engine schedules requests with continuous batching, stores keys and values in a paged cache, and shares cached prefixes between requests, and a separate loop speculates with a draft model for a single sequence. MAX Serve is built from the same parts, with the kernels and scheduling that make them fast on GPUs. Chapter 6 returns to the model and extends it to newer architectures.
6. Bigger models
Your SmolLM2 is a plain Llama. Newer models change a few parts, each for a reason. This chapter extends your model to them, and everything still runs on a laptop.
| Section | You learn |
|---|---|
| 6.1 Qwen3: normalized queries and keys | Bringing up Qwen3-0.6B by subclassing your Llama, and what QK-norm does to attention scores. |
| 6.2 Gemma 3: sliding-window attention | Bringing up Gemma 3 270M: sliding-window attention, GeGLU, and extra normalization. |
| 6.3 Mixture of experts | A Qwen3-MoE layer checked against Hugging Face, routing, and expert parallelism on a simulated mesh. |
Related handbook pages: choosing a model, quantization, and parallelism strategies.
6.1 Qwen3: normalized queries and keys
Most open models are close relatives of the Llama architecture you built in
Chapter 1. To bring up a new one, you find
what changed and why. This section adds Qwen3, here the 0.6-billion-parameter
model Qwen/Qwen3-0.6B. It subclasses your Llama classes, overrides only what
differs, and verifies the result with the tools from
Chapter 4.
Show the setup 26 lines
import logging
import warnings
import numpy as np
import plotly.graph_objects as go
import torch
from max.driver import CPU
from max.dtype import DType
from max.graph import TensorType
from max.pipelines import AutoTokenizer
from max.tensor import Tensor
from transformers import AutoModelForCausalLM
from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.layers import apply_rope, rope_cos_sin
from minillm.model import load_model
from minillm.sampling import generate
from minillm.testing import compare_outputs
from minillm.weights import load_checkpoint
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
QWEN3 = "Qwen/Qwen3-0.6B"
What changed
Compare the two configs:
smol, qwen = (
LlamaConfig.from_pretrained(MODEL_ID),
LlamaConfig.from_pretrained(QWEN3),
)
for field in [
"vocab_size",
"hidden_size",
"num_hidden_layers",
"num_attention_heads",
"num_key_value_heads",
"head_dim",
"intermediate_size",
"rope_theta",
"rms_norm_eps",
]:
if field == "vocab_size":
print(f"{'':22} {'SmolLM2':>10} {'Qwen3':>10}")
print(f"{field:22} {getattr(smol, field):>10} {getattr(qwen, field):>10}")
SmolLM2 Qwen3
vocab_size 49152 151936
hidden_size 576 1024
num_hidden_layers 30 28
num_attention_heads 9 16
num_key_value_heads 3 8
head_dim 64 128
intermediate_size 1536 3072
rope_theta 100000 1000000
rms_norm_eps 1e-05 1e-06
Four differences matter:
- A larger vocabulary, 151,936 tokens, which covers many languages and code. The embedding alone holds 156M parameters, about a quarter of the model’s 596M.
- An explicit
head_dim: 16 heads of 128 dimensions make 2048, wider than the 1024-dimension hidden state.q_projprojects up, ando_projprojects back down. The config class from Section 1.3 readshead_dimwhen a model sets it. - A larger RoPE base, 1,000,000, so the rotations turn more slowly and positions stay distinguishable over longer contexts.
- QK-norm: an RMSNorm on every query and key head, before RoPE.
The first three are config values that your code already handles. QK-norm needs new code.
QK-norm
Attention scores are dot products of queries and keys. During training, nothing stops some heads’ queries and keys from growing until a few scores dominate every softmax, which can destabilize the training of large models. Normalizing the queries and keys, with a learned scale per dimension, bounds the scores. Qwen3 and Gemma 3 normalize each head separately; OLMo 2 normalizes the whole query and key projections.
For each head’s query \(q\) (and likewise each key), with \(d\) = head_dim and
a learned weight \(w\) of size \(d\):
\[\hat{q} = \frac{q}{\sqrt{\tfrac{1}{d}\sum_j q_j^2 + \epsilon}} \odot w\]
The division gives every query the same root mean square, so the length of
\(\hat q\) depends on \(w\) and on \(q\)’s direction, not on \(q\)’s size. A score
\(\hat q \cdot \hat k\) is then bounded however large \(q\) and \(k\) get. In code,
QK-norm is two RMSNorms of size head_dim inside attention, applied to
each head after the projection and before RoPE. The other classes only swap
in the new attention:
Show Qwen3Attention, Qwen3DecoderLayer, Qwen3Model, Qwen3ForCausalLM 58 lines
from __future__ import annotations
from max.graph import Dim
from max.nn import ModuleList, RMSNorm
from max.tensor import Tensor
from minillm.attention import Attention
from minillm.config import LlamaConfig
from minillm.layers import apply_rope, rope_cos_sin
from minillm.model import DecoderLayer, LlamaForCausalLM, LlamaModel
class Qwen3Attention(Attention):
"""Llama attention with an RMSNorm on each query and key head."""
def __init__(self, config: LlamaConfig) -> None:
super().__init__(config)
self.q_norm = RMSNorm(self.head_dim, eps=config.rms_norm_eps)
self.k_norm = RMSNorm(self.head_dim, eps=config.rms_norm_eps)
def project(
self, x: Tensor, start: int | Dim | Tensor
) -> tuple[Tensor, Tensor, Tensor]:
batch, seq, _ = x.shape
q = self.q_proj(x).reshape([batch, seq, self.num_heads, self.head_dim])
k = self.k_proj(x).reshape(
[batch, seq, self.num_kv_heads, self.head_dim]
)
v = self.v_proj(x).reshape(
[batch, seq, self.num_kv_heads, self.head_dim]
)
q, k = self.q_norm(q), self.k_norm(k)
cos, sin = rope_cos_sin(
self.head_dim, self.rope_theta, start, seq, x.mesh
)
return apply_rope(q, cos, sin), apply_rope(k, cos, sin), v
class Qwen3DecoderLayer(DecoderLayer):
def __init__(self, config: LlamaConfig) -> None:
super().__init__(config)
self.self_attn = Qwen3Attention(config)
class Qwen3Model(LlamaModel):
def __init__(self, config: LlamaConfig) -> None:
super().__init__(config)
self.layers = ModuleList(
[Qwen3DecoderLayer(config) for _ in range(config.num_hidden_layers)]
)
class Qwen3ForCausalLM(LlamaForCausalLM):
"""Qwen3: a Llama model whose attention normalizes queries and keys."""
def __init__(self, config: LlamaConfig) -> None:
super().__init__(config)
self.model = Qwen3Model(config)
Saved to minillm/qwen3.py.
The new parameters are named q_norm and k_norm, as in Qwen3’s checkpoint, so
load_model from Section 1.6
loads it unchanged. load_model builds the model inside F.lazy(), so the
Llama layers that the subclasses replace are never allocated.
model = load_model(qwen, load_checkpoint(QWEN3), DEVICE, Qwen3ForCausalLM)
compiled = model.compile(
TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)
n_params = sum(p.num_elements() for _, p in model.parameters)
print(
f"{n_params / 1e6:.0f}M values, including the untied copy of the embedding"
)
752M values, including the untied copy of the embedding
To see what QK-norm guarantees on real activations, run layer 0’s attention on a sentence: its queries and keys, with RoPE, the scores \(q \cdot k / \sqrt{d}\), and their softmax. Then scale the layer’s input up, which scales queries and keys alike, as projection weights that grew during training would. The entropy of a softmax row measures how spread out a query’s attention is: 0 when one key takes all the weight, \(\log n\) when \(n\) keys share it equally.
Show softmax, entropy_per_head 43 lines
attn = model.model.layers[0].self_attn
ids = AutoTokenizer(QWEN3)(
["Normalizing queries and keys keeps attention scores in a stable range."]
)["input_ids"][0]
seq = len(ids)
h = model.model.layers[0].input_layernorm(
model.model.embed_tokens(Tensor(np.array([ids]), device=DEVICE))
)
cos, sin = rope_cos_sin(attn.head_dim, attn.rope_theta, 0, seq, DEVICE)
causal = np.triu(np.full((seq, seq), -np.inf), 1)
def softmax(z: np.ndarray) -> np.ndarray:
e = np.exp(z.astype(np.float64) - z.max(-1, keepdims=True))
return e / e.sum(-1, keepdims=True)
def entropy_per_head(scale: float, qk_norm: bool) -> np.ndarray:
"""Mean attention entropy of each query head, in nats."""
x = h * scale
q = attn.q_proj(x).reshape([1, seq, attn.num_heads, attn.head_dim])
k = attn.k_proj(x).reshape([1, seq, attn.num_kv_heads, attn.head_dim])
if qk_norm:
q, k = attn.q_norm(q), attn.k_norm(k)
q = apply_rope(q, cos, sin).to_numpy()[0]
k = apply_rope(k, cos, sin).to_numpy()[0]
k = np.repeat(k, attn.num_heads // attn.num_kv_heads, axis=1)
scores = np.einsum("qhd,khd->hqk", q, k) / np.sqrt(attn.head_dim)
p = softmax(scores + causal)
# Position 0 sees only itself, so its entropy is always 0.
return -(p * np.log(p + 1e-30)).sum(-1)[:, 1:].mean(-1)
entropy = {
(scale, qk_norm): entropy_per_head(scale, qk_norm)
for scale in [1, 3, 10]
for qk_norm in [False, True]
}
for (scale, qk_norm), e in entropy.items():
label = "with" if qk_norm else "without"
print(
f"input x{scale:<3} {label:7} QK-norm: mean entropy {e.mean():.2f} nats"
)
input x1 without QK-norm: mean entropy 1.87 nats
input x1 with QK-norm: mean entropy 0.77 nats
input x3 without QK-norm: mean entropy 1.39 nats
input x3 with QK-norm: mean entropy 0.77 nats
input x10 without QK-norm: mean entropy 0.18 nats
input x10 with QK-norm: mean entropy 0.77 nats
Show the code 26 lines
steps = [
[
go.Bar(
x=np.arange(attn.num_heads),
y=entropy[scale, qk_norm],
name=f"{'after' if qk_norm else 'before'} QK-norm",
marker_color=viz.BLUE if qk_norm else viz.ORANGE,
hovertemplate="head %{x}: %{y:.2f} nats<extra>%{fullData.name}</extra>",
)
for qk_norm in [False, True]
]
for scale in [1, 3, 10]
]
fig = go.Figure(
layout=dict(
title="Attention entropy per head, Qwen3 layer 0",
height=460,
barmode="group",
xaxis=dict(title="query head", dtick=1),
yaxis=dict(
title="mean entropy (nats)",
range=[0, 1.1 * max(e.max() for e in entropy.values())],
),
)
)
viz.show(viz.with_slider(fig, steps, ["x1", "x3", "x10"], "input scaled "))
Move the slider. Without the norm (orange), the attention depends on the scale.
Scaling the input by 10 makes queries and keys 10 times longer and the scores
100 times larger, and the softmax collapses: the orange bars fall close to 0,
so each query attends to almost a single key. At x1 the orange bars sit higher
than the blue ones, because Qwen3’s raw queries and keys are small; the trained
model always normalizes them. The blue bars don’t move at all. After q_norm and k_norm, a query’s or key’s length depends
only on its direction and the learned per-dimension scale, so no input can make
a score blow up.
Verify it
Compare with Hugging Face’s PyTorch Qwen3 on a few prompts, using the metrics from Section 4.2:
tokenizer = AutoTokenizer(QWEN3)
reference = AutoModelForCausalLM.from_pretrained(
QWEN3, torch_dtype=torch.float32
).eval()
prompts = {
"english": "The capital of France is",
"code": "def add(a, b):\n return",
"french": "Le chat est assis sur",
}
for name, text in prompts.items():
ids = tokenizer([text])["input_ids"][0]
ours = compiled(Tensor(np.array([ids]), device=DEVICE)).to_numpy()[0]
with torch.no_grad():
theirs = reference(torch.tensor([ids])).logits[0].numpy()
r = compare_outputs(ours, theirs, atol=1e-3, rtol=1e-3)
p, q = softmax(theirs), softmax(ours)
kl = float((p * (np.log(p + 1e-30) - np.log(q + 1e-30))).sum(-1).mean())
top1 = (ours.argmax(-1) == theirs.argmax(-1)).mean()
print(
f"{name:8} max err {r.max_abs_err:.1e} KL {kl:.1e} top-1 {top1:.0%}"
)
assert top1 >= 0.99 and kl < 1e-6
english max err 1.4e-04 KL 1.0e-10 top-1 100%
code max err 1.1e-04 KL 1.9e-11 top-1 100%
french max err 8.5e-05 KL 7.8e-11 top-1 100%
def next_logits(ids: list[int]) -> Tensor:
return compiled(Tensor(np.array([ids]), device=DEVICE))[0, -1]
prompt = tokenizer(["The three primary colors are"])["input_ids"][0]
print(
tokenizer.decode(prompt + generate(next_logits, prompt, 24, temperature=0))
)
The three primary colors are red, blue, and yellow. The three secondary colors are red and blue, blue and yellow, and red and yellow
That is the whole bring-up: about 40 lines of model code for a new architecture, checked against the reference to rounding error. MAX’s own Qwen3 implementation serves the same model with a paged KV cache and continuous batching, the techniques that Chapter 5 builds:
max serve --model Qwen/Qwen3-0.6B --quantization-encoding float32 --no-prefer-module-v3
--no-prefer-module-v3 selects MAX’s text-generation implementation of
Qwen3. By default MAX prefers an architecture’s module-based implementation
(Section 7.1 explains
this preference), and while the only module-based one registered under Qwen3’s
architecture name is the embedding model, max serve picks it for this
checkpoint and fails to start.
Qwen3 is also a reasoning model: by default it thinks step by step inside
<think> tags before it answers, which costs tokens and improves answers to
hard questions. Its chat template has an enable_thinking switch that turns
this off. The handbook’s
choosing the right model
page discusses that trade-off.
Exercises
- Add a KV cache to Qwen3: subclass
CachedAttentionfrom Section 1.8 with the sameq_normandk_norm, and check it against the full model. - Remove QK-norm: copy
load_modelfrom Section 1.6, buildLlamaForCausalLMinstead ofQwen3ForCausalLM, and passstrict=Falsetoload_state_dictso that it skips theq_normandk_normweights. Rerun the comparison. What happens to the output? - Count the parameters in the embedding, the attention projections, and the MLPs. How do the proportions compare with SmolLM2’s?
Next
Section 6.2 adds sliding-window attention with Gemma 3, and Section 6.3 builds a mixture of experts.
6.2 Gemma 3: sliding-window attention
Attention’s cost grows with the square of the context, and the KV cache grows linearly with it. Gemma 3 reduces both with sliding-window attention: most of its layers look only at a recent window of tokens (512 in the 270M model), and every sixth layer looks at all of them. This section brings up Gemma 3 270M, which differs from Llama in more places than Qwen3 did in Section 6.1, and verifies it on a prompt long enough for the window to matter.
Gemma’s weights are gated: accept the license on the
model page and make a
Hugging Face token available (on Colab, as the HF_TOKEN secret).
Show the setup 27 lines
import logging
import math
import warnings
from dataclasses import dataclass
import numpy as np
import plotly.graph_objects as go
import torch
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.graph import TensorType
from max.pipelines import AutoTokenizer
from max.tensor import Tensor
from plotly.subplots import make_subplots
from transformers import AutoModelForCausalLM
from minillm import viz
from minillm.device import pick_device
from minillm.model import load_model
from minillm.testing import compare_outputs
from minillm.weights import load_checkpoint
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
GEMMA = "google/gemma-3-270m-it"
What changed
Gemma 3’s config.json has fields that Llama’s doesn’t. A config subclass
adds them, and from_pretrained reads every dataclass field it finds:
Show Gemma3Config 29 lines
from __future__ import annotations
import math
from dataclasses import dataclass, replace
from max import functional as F
from max.driver import Device
from max.dtype import DType
from max.graph import Dim
from max.nn import Embedding, GemmaRMSNorm, Linear, Module, ModuleList
from max.nn.kernels import flash_attention
from max.tensor import Tensor
from max.tensor.sharding import DeviceMesh
from minillm.attention import Attention, causal_mask
from minillm.config import LlamaConfig
from minillm.layers import MLP, apply_rope, rope_cos_sin
@dataclass(frozen=True)
class Gemma3Config(LlamaConfig):
sliding_window: int = 512
layer_types: tuple[str, ...] = ()
rope_local_base_freq: float = 10000.0
query_pre_attn_scalar: float = 256.0
def __post_init__(self) -> None:
super().__post_init__()
object.__setattr__(self, "layer_types", tuple(self.layer_types))
Saved to minillm/gemma3.py.
config = Gemma3Config.from_pretrained(GEMMA)
for field in [
"vocab_size",
"hidden_size",
"num_hidden_layers",
"num_attention_heads",
"num_key_value_heads",
"head_dim",
"intermediate_size",
"rope_theta",
"rope_local_base_freq",
"sliding_window",
"query_pre_attn_scalar",
"rms_norm_eps",
]:
print(f"{field:22} {getattr(config, field):>10}")
pattern = "".join(
"G" if t == "full_attention" else "s" for t in config.layer_types
)
print(f"{'layer types':22} {pattern:>10} (s = sliding, G = global)")
vocab_size 262144
hidden_size 640
num_hidden_layers 18
num_attention_heads 4
num_key_value_heads 1
head_dim 256
intermediate_size 2048
rope_theta 1000000.0
rope_local_base_freq 10000.0
sliding_window 512
query_pre_attn_scalar 256
rms_norm_eps 1e-06
layer types sssssGsssssGsssssG (s = sliding, G = global)
Compared with your Llama, Gemma 3 changes six things:
| Part | Llama | Gemma 3 |
|---|---|---|
| Attention span | every layer sees all past tokens | 5 of 6 layers see only the last 512 |
| RoPE base | one, 100,000 for SmolLM2 | 10,000 for windowed layers, 1,000,000 for global ones |
| Norms per layer | 2, before attention and before the MLP | 4, before and after each |
| RMSNorm | x / rms(x) * w | x / rms(x) * (1 + w) (max.nn.GemmaRMSNorm) |
| MLP activation | SiLU (SwiGLU) | tanh-approximated GELU (GeGLU) |
| Embedding | as stored | multiplied by \(\sqrt{\text{hidden size}}\) |
It also has QK-norm, as in Qwen3, and scales attention scores by one over
the square root of query_pre_attn_scalar instead of the head dimension
(both are 256 in this model). Its vocabulary has 262,144 tokens, so the
embedding holds 168M of the model’s 270M parameters.
Sliding-window attention
A windowed layer adds one more condition to the causal mask: query i may
see key j only if \(i - j < 512\). F.band_part builds that band. Each
layer reads its own RoPE base and window from the config.
The mask uses a large finite number instead of \(-\infty\), for the reason Section 5.1 gave: a query more than 512 tokens in has its whole first block of keys hidden, and a blockwise attention kernel computes \(e^{-\infty + \infty}\) = NaN for such a row.
Show sliding_window_mask, Gemma3Attention 56 lines
def sliding_window_mask(
q_len: int | Dim,
kv_len: int | Dim,
window: int,
device: Device | DeviceMesh,
) -> Tensor:
"""Like ``causal_mask``, but each query also can't see keys ``window`` or
more positions behind it."""
# A finite value, not -inf: a window hides a query's whole first block of
# keys, and a blockwise attention kernel would turn that into NaN.
blocked = F.broadcast_to(
Tensor(-1e30, dtype=DType.float32, device=device), [q_len, kv_len]
)
return F.band_part(
blocked, num_lower=window - 1, num_upper=kv_len - q_len, exclude=True
)
class Gemma3Attention(Attention):
def __init__(self, config: Gemma3Config, layer: int) -> None:
super().__init__(config)
self.sliding = config.layer_types[layer] == "sliding_attention"
self.window = config.sliding_window
self.rope_theta = (
config.rope_local_base_freq if self.sliding else config.rope_theta
)
self.scale = config.query_pre_attn_scalar**-0.5
self.q_norm = GemmaRMSNorm(self.head_dim, eps=config.rms_norm_eps)
self.k_norm = GemmaRMSNorm(self.head_dim, eps=config.rms_norm_eps)
def forward(self, x: Tensor) -> Tensor:
batch, seq, _ = x.shape
q = self.q_norm(
self.q_proj(x).reshape([batch, seq, self.num_heads, self.head_dim])
)
k = self.k_norm(
self.k_proj(x).reshape(
[batch, seq, self.num_kv_heads, self.head_dim]
)
)
v = self.v_proj(x).reshape(
[batch, seq, self.num_kv_heads, self.head_dim]
)
cos, sin = rope_cos_sin(self.head_dim, self.rope_theta, 0, seq, x.mesh)
q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
if self.sliding:
mask = sliding_window_mask(seq, seq, self.window, x.mesh)
else:
mask = causal_mask(seq, seq, x.mesh)
mask = F.broadcast_to(mask, [batch, seq, seq])
out = flash_attention(q, k, v, self.scale, mask).rebind_mapping(
q.mapping
)
return self.o_proj(
out.reshape([batch, seq, self.num_heads * self.head_dim])
)
Saved to minillm/gemma3.py.
Compare the two masks on 16 tokens, with a window small enough to see. The slider changes the window; Gemma 3’s is 512. At a window of 16 the two panels match: once the window covers the whole sequence, a sliding layer is a global one.
seq = 16
full = (causal_mask(seq, seq, DEVICE).to_numpy() == 0).astype(int)
windowed_masks = {
window: (
sliding_window_mask(seq, seq, window, DEVICE).to_numpy() == 0
).astype(int)
for window in [2, 4, 8, 16]
}
print("visible keys per query, window 4:", windowed_masks[4].sum(axis=1))
visible keys per query, window 4: [1 2 3 4 4 4 4 4 4 4 4 4 4 4 4 4]
Show the code 48 lines
steps = []
for window, windowed in windowed_masks.items():
steps.append(
[
go.Heatmap(
z=m,
zmin=0,
zmax=1,
showscale=False,
xgap=2,
ygap=2,
xaxis=f"x{a}",
yaxis=f"y{a}",
colorscale=[
[0, viz.GRID],
[0.5, viz.GRID],
[0.5, viz.BLUE],
[1, viz.BLUE],
],
hovertemplate="query %{y}, key %{x}: "
+ "%{z}"
+ "<extra>"
+ title
+ "</extra>",
)
for m, a, title in [
(full, "", "global layer"),
(windowed, "2", f"window {window}"),
]
]
)
fig = make_subplots(
rows=1,
cols=2,
horizontal_spacing=0.08,
subplot_titles=[
"global layer: causal",
"sliding layer: causal and windowed",
],
)
fig.update_xaxes(title_text="key position", showgrid=False)
fig.update_yaxes(
title_text="query position", showgrid=False, autorange="reversed"
)
fig.update_layout(title="Which keys each query sees (1 = visible)", height=440)
viz.show(
viz.with_slider(fig, steps, ["2", "4", "8", "16"], "window ", active=1)
)
Once the sequence is longer than the window, a sliding layer’s cost per
query stops growing, and so does the KV cache it needs: only the last
window keys. That saving needs an attention kernel that skips fully masked
blocks and a cache that evicts keys older than the window. The model in this
section has neither: its dense mask still visits every key, and it keeps no
cache, so it computes the same function without the savings. In a serving
engine with both, five sliding layers for every global one mean most of
Gemma 3’s KV cache stops growing at 512 tokens.
The rest of the model
GeGLU is the SwiGLU MLP with a different activation. The decoder layer has the four norms, and the model scales its embedding:
Show GeGLU, Gemma3DecoderLayer, Gemma3Model, Gemma3ForCausalLM, ... 64 lines
class GeGLU(MLP):
def forward(self, x: Tensor) -> Tensor:
return self.down_proj(
F.gelu(self.gate_proj(x), approximate="tanh") * self.up_proj(x)
)
class Gemma3DecoderLayer(Module):
def __init__(self, config: Gemma3Config, layer: int) -> None:
eps = config.rms_norm_eps
self.input_layernorm = GemmaRMSNorm(config.hidden_size, eps=eps)
self.self_attn = Gemma3Attention(config, layer)
self.post_attention_layernorm = GemmaRMSNorm(
config.hidden_size, eps=eps
)
self.pre_feedforward_layernorm = GemmaRMSNorm(
config.hidden_size, eps=eps
)
self.mlp = GeGLU(config)
self.post_feedforward_layernorm = GemmaRMSNorm(
config.hidden_size, eps=eps
)
def forward(self, x: Tensor) -> Tensor:
x = x + self.post_attention_layernorm(
self.self_attn(self.input_layernorm(x))
)
return x + self.post_feedforward_layernorm(
self.mlp(self.pre_feedforward_layernorm(x))
)
class Gemma3Model(Module):
def __init__(self, config: Gemma3Config) -> None:
self.embed_scale = math.sqrt(config.hidden_size)
self.embed_tokens = Embedding(config.vocab_size, dim=config.hidden_size)
self.layers = ModuleList(
[
Gemma3DecoderLayer(config, i)
for i in range(config.num_hidden_layers)
]
)
self.norm = GemmaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
def forward(self, input_ids: Tensor) -> Tensor:
x = self.embed_tokens(input_ids) * self.embed_scale
for layer in self.layers:
x = layer(x)
return self.norm(x)
class Gemma3ForCausalLM(Module):
def __init__(self, config: Gemma3Config) -> None:
self.model = Gemma3Model(config)
self.lm_head = Linear(config.hidden_size, config.vocab_size, bias=False)
def forward(self, input_ids: Tensor) -> Tensor:
return self.lm_head(self.model(input_ids))
def without_sliding_window(config: Gemma3Config) -> Gemma3Config:
"""Returns ``config`` with windows too wide to hide any token; each layer
keeps its own RoPE base."""
return replace(config, sliding_window=config.max_position_embeddings)
Saved to minillm/gemma3.py.
weights = load_checkpoint(GEMMA)
model = load_model(config, weights, DEVICE, Gemma3ForCausalLM)
compiled = model.compile(
TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)
Verify it
A short prompt never reaches the window, so verify on two prompts: a short one, and one of about 700 tokens, where windowed layers must ignore the oldest tokens.
Show softmax, verify 38 lines
tokenizer = AutoTokenizer(GEMMA)
reference = AutoModelForCausalLM.from_pretrained(
GEMMA, torch_dtype=torch.float32, attn_implementation="eager"
).eval()
prompts = {
"short": "The capital of France is",
"long (700 tokens)": "The quick brown fox jumps over the lazy dog. " * 70
+ "The capital of France is",
}
def softmax(z: np.ndarray) -> np.ndarray:
e = np.exp(z.astype(np.float64) - z.max(-1, keepdims=True))
return e / e.sum(-1, keepdims=True)
def verify(run) -> dict[str, tuple[np.ndarray, float]]: # noqa: ANN001
# Prints and returns each prompt's KL divergence per position and its
# top-1 agreement.
results = {}
for name, text in prompts.items():
ids = tokenizer([text])["input_ids"][0]
ours = run(Tensor(np.array([ids]), device=DEVICE)).to_numpy()[0]
with torch.no_grad():
theirs = reference(torch.tensor([ids])).logits[0].numpy()
r = compare_outputs(ours, theirs)
p, q = softmax(theirs), softmax(ours)
kl = (p * (np.log(p + 1e-30) - np.log(q + 1e-30))).sum(-1)
top1 = (ours.argmax(-1) == theirs.argmax(-1)).mean()
print(
f"{name:18} {len(ids):4} tokens max err {r.max_abs_err:.1e} KL {kl.mean():.1e} top-1 {top1:.0%}"
)
results[name] = (kl, top1)
return results
matched = verify(compiled)
assert all(kl.mean() < 1e-6 and top1 >= 0.99 for kl, top1 in matched.values())
short 6 tokens max err 1.6e-04 KL 4.8e-11 top-1 100%
long (700 tokens) 706 tokens max err 4.1e-04 KL 6.5e-12 top-1 100%
Both match. To see that the window does real work, build the same model with windows too wide to hide anything, keep each layer’s RoPE base, and verify again:
no_window = load_model(
without_sliding_window(config), weights, DEVICE, Gemma3ForCausalLM
)
unwindowed = verify(
no_window.compile(
TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)
)
long = "long (700 tokens)"
assert unwindowed["short"][0].mean() < 1e-6 # no query is far enough back
assert unwindowed[long][0].mean() > 100 * matched[long][0].mean()
short 6 tokens max err 1.6e-04 KL 4.8e-11 top-1 100%
long (700 tokens) 706 tokens max err 4.9e+01 KL 2.8e-01 top-1 93%
Show the code 30 lines
fig = go.Figure(
[
go.Scatter(
x=np.arange(len(results[long][0])),
y=np.maximum(results[long][0], 1e-16),
name=name,
mode="lines",
line_color=color,
hovertemplate="position %{x}: KL %{y:.1e}<extra>"
+ name
+ "</extra>",
)
for name, results, color in [
("with the window", matched, viz.BLUE),
("without the window", unwindowed, viz.ORANGE),
]
]
)
fig.add_vline(
x=config.sliding_window,
line=dict(color=viz.INK_SECONDARY, dash="dot", width=1),
annotation_text=f"position {config.sliding_window}",
)
fig.update_layout(
title="KL divergence from the reference at each position of the long prompt",
height=420,
xaxis_title="position",
yaxis=dict(title="KL (nats, log scale)", type="log", dtick=2),
)
viz.show(fig)
On the short prompt the two models are identical, because no query is more than 512 tokens from any key. On the long prompt the model without the window diverges from the reference, and the figure shows where: the two curves agree up to position 511 and split at 512, the first position with a key outside its window. The windowed layers do ignore the oldest tokens, and the bring-up must get the window right to match.
With a kernel and cache that take advantage of the window, its attention cost and KV cache stop growing at 512 tokens. With 15 of Gemma 3’s 18 layers windowed, a 32,000-token context needs about a fifth of the cache that a fully global model would need. The handbook’s GPU memory calculation page shows how cache size limits context and batch size.
Exercises
- Compute the KV cache size of this model for 8K and 32K tokens of context, with and without sliding windows.
- Plot attention weights, as in Section 1.5, for a windowed and a global layer on the long prompt. What does the window look like?
- Check the embedding scale: remove
embed_scaleand verify again. Which metric shows it?
Next
Section 6.3 builds a mixture-of-experts layer and spreads its experts across devices.
6.3 Mixture of experts
The largest open models are mixtures of experts (MoE). An MoE layer has many MLPs, the experts, instead of one, and a small router sends each token to only a few of them. The model holds the parameters of all its experts, while each token pays for only a fraction of them. This section builds an MoE layer, checks it against Hugging Face’s Qwen3-MoE implementation, and then spreads the experts across devices.
No real MoE checkpoint fits on a laptop, so this section uses Qwen3-MoE’s architecture at a small size with random weights, a common way to test large-model code. As in Section 4.1, the test compares one layer with a reference implementation.
Show the setup 32 lines
import json
import logging
import warnings
from pathlib import Path
import numpy as np
import plotly.graph_objects as go
import torch
from huggingface_hub import hf_hub_download
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.tensor import Tensor, default_device, default_dtype
from max.tensor.sharding import (
DeviceMapping,
DeviceMesh,
Replicated,
Sharded,
auto_reshard,
)
from plotly.subplots import make_subplots
from transformers import AutoConfig
from transformers.models.qwen3_moe.modeling_qwen3_moe import (
Qwen3MoeSparseMoeBlock,
)
from minillm import viz
from minillm.device import pick_device
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
How big, and how sparse
Qwen3-30B-A3B is a typical MoE model. Its name gives its total parameters, 30 billion, and its active ones, about 3 billion:
real = json.loads(
Path(hf_hub_download("Qwen/Qwen3-30B-A3B", "config.json")).read_text()
)
E, K, H, I = (
real["num_experts"],
real["num_experts_per_tok"],
real["hidden_size"],
real["moe_intermediate_size"],
)
per_expert = 3 * H * I
print(f"{E} experts per layer, {K} chosen per token")
print(
f"expert weights per layer: {E * per_expert / 1e6:.0f}M, used per token: {K * per_expert / 1e6:.0f}M"
)
128 experts per layer, 8 chosen per token
expert weights per layer: 604M, used per token: 38M
Every token uses 8 of the 128 experts, so it touches only a sixteenth of each layer’s expert weights. Adding experts adds parameters without adding work per token. The cost moves to memory, since every expert must be loaded somewhere, which is why MoE models are usually served across several GPUs. The handbook’s parallelism strategies page covers expert parallelism, which the end of this section implements.
The layer
An MoE layer replaces the MLP. For each token:
- The router, a
Linearfrom the hidden size to the number of experts, scores every expert. Softmax turns the scores into probabilities. - The
Kmost probable experts are chosen, and their probabilities are renormalized to sum to 1. Qwen3-MoE’s config asks for this withnorm_topk_prob: true; a model that sets it tofalseweights the experts by their raw probabilities, so reusingMoEfor one means dropping that step. - Each chosen expert, a SwiGLU MLP like the one in Section 1.4, processes the token, and the outputs are summed, weighted by those probabilities.
In symbols, with router weights \(W_r\), the set \(\mathcal{T}\) of the \(K\) most probable experts, and expert \(e\)’s MLP \(E_e\):
\[p = \mathrm{softmax}(W_r x), \qquad y = \sum_{e \in \mathcal{T}} \frac{p_e}{\sum_{j \in \mathcal{T}} p_j} \, E_e(x)\]
The experts’ weights are stored stacked, with the expert as the first axis: one tensor for the fused gate and up projections, and one for the down projection. That is the layout of Hugging Face’s Qwen3-MoE module; the checkpoint files store each expert’s projections separately. This version computes every expert on every token and multiplies the unselected ones by zero. That wastes work, but it is simple and exact, and the last part of this section shows that it is easy to distribute.
Show Experts, MoE 66 lines
from __future__ import annotations
from max import functional as F
from max.dtype import DType
from max.nn import Linear, Module
from max.tensor import Tensor
class Experts(Module):
"""The stacked weights of every expert: gate and up projections fused."""
def __init__(
self, num_experts: int, hidden: int, intermediate: int
) -> None:
self.gate_up_proj = Tensor.zeros(
[num_experts, 2 * intermediate, hidden]
)
self.down_proj = Tensor.zeros([num_experts, hidden, intermediate])
class MoE(Module):
"""A mixture-of-experts layer with top-k routing, as in Qwen3-MoE."""
def __init__(
self, num_experts: int, top_k: int, hidden: int, intermediate: int
) -> None:
self.num_experts, self.top_k, self.intermediate = (
num_experts,
top_k,
intermediate,
)
self.gate = Linear(hidden, num_experts, bias=False) # the router
self.experts = Experts(num_experts, hidden, intermediate)
def routing(self, x: Tensor) -> Tensor:
"""Returns each token's weight for each expert, ``[tokens, experts]``,
zero for experts the token wasn't routed to."""
tokens = x.shape[0]
weights, chosen = F.top_k(F.softmax(self.gate(x)), self.top_k)
weights = weights / weights.sum(axis=-1)
ids = Tensor.arange(self.num_experts, dtype=DType.int64, device=x.mesh)
one_hot = (
chosen.reshape([tokens, self.top_k, 1])
== ids.reshape([1, 1, self.num_experts])
).cast(x.dtype)
return (
(one_hot * weights.reshape([tokens, self.top_k, 1]))
.sum(axis=1)
.reshape([tokens, self.num_experts])
)
def forward(self, x: Tensor) -> Tensor:
"""``x`` is ``[tokens, hidden]``."""
tokens, hidden = x.shape
gate_up = x.reshape(
[1, tokens, hidden]
) @ self.experts.gate_up_proj.transpose(1, 2)
gate, up = (
gate_up[:, :, : self.intermediate],
gate_up[:, :, self.intermediate :],
)
outputs = (F.silu(gate) * up) @ self.experts.down_proj.transpose(
1, 2
) # [E, T, H]
weights = self.routing(x).T.reshape([self.num_experts, tokens, 1])
return (weights * outputs).sum(axis=0).reshape([tokens, hidden])
Saved to minillm/moe.py.
Check it against Qwen3-MoE
Build Hugging Face’s Qwen3-MoE block at a small size, 8 experts with 2 per
token, fill it with random weights, and load the same weights into yours.
The parameter names match, so load_state_dict takes the PyTorch state
directly:
torch.manual_seed(0)
small = AutoConfig.from_pretrained("Qwen/Qwen3-30B-A3B")
(
small.hidden_size,
small.moe_intermediate_size,
small.num_experts,
small.num_experts_per_tok,
) = 64, 32, 8, 2
reference = Qwen3MoeSparseMoeBlock(small).float().eval()
for p in reference.parameters():
torch.nn.init.normal_(p, std=0.2)
state = {name: p.detach().numpy() for name, p in reference.state_dict().items()}
print({name: w.shape for name, w in state.items()})
with default_device(DEVICE), default_dtype(DType.float32):
moe = MoE(num_experts=8, top_k=2, hidden=64, intermediate=32)
moe.load_state_dict(state)
x = np.random.default_rng(0).standard_normal((10, 64)).astype(np.float32)
with torch.no_grad():
expected = reference(torch.from_numpy(x)[None])[0].numpy()
ours = moe(Tensor(x, device=DEVICE)).to_numpy()
print(f"max difference {np.abs(ours - expected).max():.1e}")
assert np.abs(ours - expected).max() < 1e-4
{'experts.gate_up_proj': (8, 64, 64), 'experts.down_proj': (8, 64, 32), 'gate.weight': (8, 64)}
max difference 3.8e-06
Where the tokens go
The routing matrix shows which experts each token uses. Count the tokens per expert for a larger batch, then make the router favor expert 0 by scaling its row of router weights, and count again:
batch = Tensor(
np.random.default_rng(1).standard_normal((512, 64)).astype(np.float32),
device=DEVICE,
)
weights_np = moe.routing(batch).to_numpy()
load = (weights_np > 0).sum(axis=0)
print("tokens per expert:", load, " ideal:", 512 * 2 // 8)
assert load.sum() == 512 * 2
def load_with_bias(factor: float) -> np.ndarray:
router = state["gate.weight"].copy()
router[0] *= factor
with default_device(DEVICE), default_dtype(DType.float32):
biased = MoE(num_experts=8, top_k=2, hidden=64, intermediate=32)
biased.load_state_dict({**state, "gate.weight": router})
return (biased.routing(batch).to_numpy() > 0).sum(axis=0)
loads = {factor: load_with_bias(factor) for factor in [1, 2, 5]}
for factor, counts in loads.items():
print(f"expert 0's row x{factor}: tokens per expert {counts}")
assert (loads[1] == load).all()
tokens per expert: [122 116 157 121 117 134 122 135] ideal: 128
expert 0's row x1: tokens per expert [122 116 157 121 117 134 122 135]
expert 0's row x2: tokens per expert [178 109 144 116 108 130 113 126]
expert 0's row x5: tokens per expert [227 104 136 109 103 121 104 120]
Show the code 60 lines
fig = make_subplots(
rows=1,
cols=2,
column_widths=[0.6, 0.4],
horizontal_spacing=0.2,
subplot_titles=[
"Routing weights of the first 32 tokens",
"Tokens per expert, all 512",
],
)
fig.add_trace(
go.Heatmap(
z=np.where(weights_np[:32] > 0, weights_np[:32], np.nan),
colorscale=viz.SEQUENTIAL,
zmin=0,
zmax=1,
xgap=2,
ygap=2,
colorbar=dict(title="weight", thickness=12, x=0.5),
hovertemplate="token %{y} -> expert %{x}: weight %{z:.2f}<extra></extra>",
),
row=1,
col=1,
)
steps = [
[
go.Bar(
x=np.arange(len(counts)),
y=counts,
marker_color=[viz.ORANGE] + [viz.BLUE] * (len(counts) - 1),
showlegend=False,
xaxis="x2",
yaxis="y2",
hovertemplate="expert %{x}: %{y} tokens<extra></extra>",
)
]
for counts in loads.values()
]
fig.update_xaxes(title_text="expert", dtick=1, showgrid=False)
fig.update_yaxes(
title_text="token", autorange="reversed", showgrid=False, row=1, col=1
)
fig.update_yaxes(
title_text="tokens routed",
range=[0, 1.1 * max(c.max() for c in loads.values())],
row=1,
col=2,
)
fig.update_layout(title="Top-2 routing over 8 experts", height=520)
labels = [f"x{factor}" for factor in loads]
viz.with_slider(fig, steps, labels, "expert 0's router row ")
fig.add_hline(
y=512 * 2 / len(load),
line=dict(color=viz.INK_SECONDARY, dash="dash"),
row=1,
col=2,
annotation_text="even split",
annotation_position="top right",
)
viz.show(fig)
Every token went to exactly two experts. Move the slider to scale expert 0’s row of router weights: its scores grow, so more tokens pick it, and its bar rises to nearly twice the even split at x5 while the others shrink. How evenly a batch spreads across experts matters in practice: an expert that gets far more tokens than the others becomes the slowest part of the layer. MoE models are usually trained with a load-balancing term in the loss that pushes the router to spread tokens evenly, and serving systems monitor the load on each expert.
This dense version computes all 8 experts for every token, 4 times the work that the 2 chosen experts need. Production kernels instead group the tokens by expert and run one matrix multiplication per expert over only its tokens (a grouped GEMM), which removes the waste:
per_expert_small = 3 * 64 * 32 # gate, up, and down projections
dense = 2 * 512 * 8 * per_expert_small
sparse = 2 * 512 * 2 * per_expert_small
print(
f"dense: {dense / 1e6:.1f} MFLOP, routed: {sparse / 1e6:.1f} MFLOP, {dense / sparse:.0f}x less"
)
dense: 50.3 MFLOP, routed: 12.6 MFLOP, 4x less
Expert parallelism
The experts are independent, so they can live on different devices. With 8 experts on 4 devices, each device holds 2. Every device receives every token, runs its own experts, and weights their outputs; the final sum over experts spans devices, which takes one all-reduce. On the simulated mesh from Section 2.1, the only change is to shard the stacked expert weights along the expert axis:
mesh = DeviceMesh((CPU(),) * 4, (4,), ("ep",))
with default_device(mesh), default_dtype(DType.float32):
ep_moe = MoE(num_experts=8, top_k=2, hidden=64, intermediate=32)
by_expert = DeviceMapping(mesh, (Sharded(0),))
ep_moe.experts.gate_up_proj = ep_moe.experts.gate_up_proj.to(by_expert)
ep_moe.experts.down_proj = ep_moe.experts.down_proj.to(by_expert)
ep_moe.load_state_dict(state)
print(ep_moe.experts.down_proj.mapping)
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
with auto_reshard(mode="warn"):
ep_out = ep_moe(Tensor(x, device=DeviceMapping(mesh, (Replicated(),))))
for w in caught:
if "reshards" in str(w.message):
print(w.message)
print(
f"max difference from the reference: {np.abs(ep_out.to_numpy() - expected).max():.1e}"
)
assert np.abs(ep_out.to_numpy() - expected).max() < 1e-4
DeviceMapping(DeviceMesh(ep=4), [Sharded(axis=0)])
mul(lhs, rhs) reshards lhs
lhs : float32[8, 10, 1]@mesh(ep=4){"ep":R}
rhs : float32[8, 10, 64]@mesh(ep=4){"ep":S0}
region : auto_reshard(DEFAULT_TRANSITIONS, mode="warn")
decided per mesh axis
axis lhs rhs -> out collective bytes status
"ep" lhs{R->S0} rhs{S0} -> S0 local_slice(lhs) 1.25 chosen, reshards lhs
lhs{R} rhs{S0->R} -> R allgather(rhs) 15360
lhs{R} rhs{S0->S2} -> S2 all_to_all(rhs) 15360
lhs{R->S1} rhs{S0->S1} -> S1 local_slice(lhs), all_to_all(rhs) 15361.2
lhs{R->P} rhs{S0->R} -> P keep_one_copy(lhs), allgather(rhs) 15361.2 not allowed
result lhs{"ep":S0} rhs{"ep":S0} -> {"ep":S0}
lhs = transfer_to(lhs, DeviceMapping(mesh, (Sharded(0),)))
with auto_reshard(mode="silent"):
at <cell>:67 in forward()
max difference from the reference: 2.9e-06
The output matches. MAX resharded the routing weights to match the
experts’ layout, so each device multiplies only its own experts’ slice, and
the weighted outputs come out as partial sums that to_numpy() all-reduces.
This dense form sends every token to every device. Expert-parallel serving sends each token only to the devices that hold its chosen experts, an all-to-all exchange, and returns the results with a second one. With many experts this moves far less data, but the communication is uneven and depends on the routing. MAX serves MoE models such as GPT-OSS and DeepSeek, and supports expert parallelism across GPUs with all-to-all dispatch and combine steps.
Exercises
- Increase
top_kto 8 (every expert). Does the output equal a weighted sum of 8 dense MLPs? What would the model lose? - Implement the routed version for one device: for each expert, gather its
tokens, run the MLP on them only, and scatter the results back. Check it
against
MoE. - With the experts spread over 4 devices as above, 2 per device, how many token copies does each device process at each slider setting of the routing figure? What does the busiest device do to the time of an expert-parallel step?
What you built
Chapter 6 extended your model with three
features of current open models: normalized queries and keys, sliding-window
attention, and mixtures of experts.
Chapter 7 serves your own architecture with
max serve, benchmarks it, and constrains its output.
7. Production serving
The last chapter packages your own architecture the way MAX packages its own, serves and measures it, and ends on Modular’s cloud.
| Section | You learn |
|---|---|
| 7.1 Serve your architecture | Packaging your model as a MAX architecture and serving it with max serve --custom-architectures. |
| 7.2 Benchmark a server | TTFT, TPOT, and throughput with max benchmark, for MAX’s SmolLM2 and yours, and a concurrency sweep against a latency target. |
| 7.3 Structured output and tools | JSON that follows a schema, and function calling built on top of it. |
| 7.4 Modular’s cloud | Pointing the same client at Modular’s cloud, an inference service that runs state-of-the-art open models on MAX. |
Related handbook pages: OpenAI-compatible API, structured outputs, function calling, and performance benchmarks.
7.1 Serve your architecture
In Section 1.1 you served
SmolLM2 with MAX’s own implementation. This section serves yours: the
LlamaForCausalLM you wrote in
Section 1.6, behind the same
AutoPipeline API and the same OpenAI-compatible server. MAX loads model code
from outside its own package through custom architectures: a Python package
that describes the model, which max serve imports with
--custom-architectures.
You register the model twice. The first registration runs your model unchanged and recomputes the whole sequence at every step. The second wraps your layers in a decoder module that reads keys and values from MAX’s paged KV cache, which is how MAX’s own architectures serve.
Show the setup 13 lines
import logging
import time
import warnings
from max.pipelines import AutoPipeline
from openai import OpenAI
from minillm.config import MODEL_ID
from minillm.server import start_max_serve
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
PROMPT = ["The capital of France is"]
First, the reference: SmolLM2 as MAX implements it, loaded with
AutoPipeline.
reference = AutoPipeline(MODEL_ID, quantization_encoding="float32")
expected = reference(PROMPT, max_new_tokens=16, top_k=1)
reference.close()
expected
['The capital of France is Paris. Paris is the largest city in France and the capital of the French department']
A custom architecture package
A MAX architecture has three parts:
- A config class tells MAX the sizes it needs to plan memory before it loads
anything: the number of layers, key/value heads, and head size. It also
declares the supported encodings; this model computes in
float32. - A pipeline model loads the weights and runs the model. MAX gives it the checkpoint’s weights and, at each step, the request’s tokens, and it returns the next-token logits.
- A
SupportedArchitectureties them together under the name in the checkpoint’sconfig.json, hereLlamaForCausalLM, along with the tokenizer and the weight format.ARCHITECTURESlists it for--custom-architecturesto find.
To keep this first version short, the pipeline model reruns the whole sequence
at each step and serves one request at a time (max_batch_size is 1), like the
generation loop in Section 1.7.
A decoder on the paged KV cache below
removes both limits.
The package is one module, minillm/serve/__init__.py, which the cells below
write part by part. The first imports what the package uses from MAX:
Show the imports 43 lines
from __future__ import annotations
from collections.abc import Callable, Sequence
from dataclasses import dataclass
from typing import ClassVar
import numpy as np
from max import functional as F
from max.driver import CPU, Buffer, Device
from max.dtype import DType
from max.graph import TensorType
from max.graph.weights import Weights, WeightsAdapter, WeightsFormat
from max.nn import Embedding, Linear, Module, ModuleList, RMSNorm
from max.nn.attention_mask import MHAMaskVariant
from max.nn.common_layers.kv_cache import per_device_caches
from max.nn.kernels import flash_attention_ragged, rope_split_store_ragged
from max.nn.kv_cache import KVCacheInputs, KVCacheParams, PagedCacheValues
from max.pipelines import upper_bounded_default
from max.pipelines.context import TextContext
from max.pipelines.lib import (
GraphRole,
MAXModelConfig,
ModelInputs,
ModelOutputs,
PipelineConfig,
PipelineModelWithKVCache,
SupportedArchitecture,
SupportedEncoding,
TextTokenizer,
)
from max.pipelines.lib.interfaces import (
ArchConfigWithAttentionKVCache,
ArchConfigWithStoredKVParams,
)
from max.pipelines.lib.role_contracts import DecoderInputs, DecoderOutputs
from max.pipelines.lib.utils import parse_state_dict_from_weights
from max.pipelines.modeling.types import PipelineTask
from max.tensor import Tensor
from transformers import AutoConfig
from minillm.config import LlamaConfig
from minillm.layers import MLP, rope_cos_sin
from minillm.model import load_model
Saved to minillm/serve/__init__.py.
The config class reads the sizes from the checkpoint’s Hugging Face config.
ArchConfigWithStoredKVParams adds construct_kv_params, which MAX calls to
size the KV cache from those sizes. llama_config converts the same config to
the LlamaConfig from
Section 1.3:
Show MinillmArchConfig, llama_config 38 lines
@dataclass
class MinillmArchConfig(
ArchConfigWithAttentionKVCache, ArchConfigWithStoredKVParams
):
DEFAULT_ENCODING: ClassVar[SupportedEncoding] = "float32"
SUPPORTED_ENCODINGS: ClassVar[set[SupportedEncoding]] = {"float32"}
@property
def num_key_value_heads(self) -> int:
return self.huggingface_config.num_key_value_heads
@property
def head_dim(self) -> int:
return self.get_head_dim(self.huggingface_config)
@property
def num_layers(self) -> int:
return self.huggingface_config.num_hidden_layers
@classmethod
def calculate_max_seq_len(
cls, huggingface_config: AutoConfig, model_config: MAXModelConfig
) -> int:
return upper_bounded_default(
upper_bound=huggingface_config.max_position_embeddings,
default=model_config.max_length,
)
def llama_config(hf: AutoConfig) -> LlamaConfig:
"""Returns the checkpoint's sizes as a ``LlamaConfig``."""
return LlamaConfig(
**{
k: getattr(hf, k)
for k in LlamaConfig.__dataclass_fields__
if hasattr(hf, k)
}
)
Saved to minillm/serve/__init__.py.
The pipeline model’s main job is to build your model. build_model casts the
checkpoint’s bfloat16 weights to float32, builds your LlamaForCausalLM with
load_model from Section 1.6,
and compiles it for any sequence length:
def build_model(
config: LlamaConfig,
pipeline_config: PipelineConfig,
weights: Weights,
adapter: WeightsAdapter | None,
device: Device,
) -> Callable[[Tensor], Tensor]:
"""Builds and compiles minillm's ``LlamaForCausalLM`` from a checkpoint."""
state = parse_state_dict_from_weights(pipeline_config, weights, adapter)
state = {name: w.astype(DType.float32) for name, w in state.items()}
model = load_model(config, state, device)
return model.compile(
TensorType(DType.int64, ("batch", "seq_len"), device=device)
)
Saved to minillm/serve/__init__.py.
MAX constructs the pipeline model with a long list of arguments, which the
constructor passes through before it calls build_model. model_config_cls
names the config class, which MAX asks to size the KV cache.
prepare_initial_token_inputs turns the request’s context into model inputs:
here all of its tokens, since the model keeps no cache. execute runs the
compiled model and returns the last position’s logits:
Show TokenInputs, MinillmPipelineModel 41 lines
@dataclass
class TokenInputs(ModelInputs):
tokens: Buffer
@property
def buffers(self) -> tuple[Buffer, ...]:
return (self.tokens,)
class MinillmPipelineModel(PipelineModelWithKVCache[TextContext]):
model_config_cls: ClassVar[type[MinillmArchConfig]] = MinillmArchConfig
def __init__(self, *args: object, **kwargs: object) -> None:
super().__init__(*args, **kwargs)
self.model = build_model(
llama_config(self.huggingface_config),
self.pipeline_config,
self.weights,
self.adapter,
self.devices[0],
)
def prepare_initial_token_inputs(
self,
replica_batches: Sequence[Sequence[TextContext]],
kv_cache_inputs: KVCacheInputs[Buffer, Buffer] | None = None,
return_n_logits: int = 1,
) -> TokenInputs:
ids = replica_batches[0][0].tokens.all
tokens = np.array([ids], dtype=np.int64)
inputs = TokenInputs(Buffer.from_numpy(tokens).to(self.devices[0]))
inputs.kv_cache_inputs = kv_cache_inputs
return inputs
def execute(self, model_inputs: ModelInputs) -> ModelOutputs:
assert isinstance(model_inputs, TokenInputs)
logits = self.model(Tensor.from_dlpack(model_inputs.tokens))
last = Buffer.from_numpy(
np.ascontiguousarray(logits.to_numpy()[0, -1:, :])
).to(self.devices[0])
return ModelOutputs(logits=last, next_token_logits=last)
Saved to minillm/serve/__init__.py.
Finally, SupportedArchitecture registers the parts under the checkpoint’s
architecture name. required_arguments makes MAX serve one request at a time
and turns off prefix caching, which this model can’t use:
def tie_embeddings(state_dict: dict[str, Weights], **_: object) -> dict:
"""Loads the checkpoint as is, with the LM head sharing the embedding."""
weights = {name: w.data() for name, w in state_dict.items()}
weights.setdefault("lm_head.weight", weights["model.embed_tokens.weight"])
return weights
minillm_arch = SupportedArchitecture(
name="LlamaForCausalLM",
task=PipelineTask.TEXT_GENERATION,
example_repo_ids=["HuggingFaceTB/SmolLM2-135M-Instruct"],
default_weights_format=WeightsFormat.safetensors,
default_encoding=MinillmArchConfig.DEFAULT_ENCODING,
supported_encodings=MinillmArchConfig.SUPPORTED_ENCODINGS,
pipeline_model=MinillmPipelineModel,
tokenizer=TextTokenizer,
context_type=TextContext,
multi_gpu_supported=False,
weight_adapters={WeightsFormat.safetensors: tie_embeddings},
config=MinillmArchConfig,
required_arguments={"enable_prefix_caching": False, "max_batch_size": 1},
supports_overlap_scheduler=False,
supports_device_graph_capture=False,
)
Saved to minillm/serve/__init__.py.
Use it in Python
custom_architectures imports the package and registers its architectures,
as --custom-architectures does for the CLI. A registered architecture with
the same name as one of MAX’s replaces it in this process. MAX registers
module-based implementations under the name with a _ModuleV3 suffix and
prefers them, so prefer_module_v3=False selects the LlamaForCausalLM
registered above:
ours = AutoPipeline(
MODEL_ID,
quantization_encoding="float32",
custom_architectures=["minillm.serve"],
prefer_module_v3=False,
)
result = ours(PROMPT, max_new_tokens=16, top_k=1)
ours.close()
print(result)
print("same as MAX's implementation:", result == expected)
assert result == expected
['The capital of France is Paris. Paris is the largest city in France and the capital of the French department']
same as MAX's implementation: True
The pipeline model is yours, and its greedy output matches MAX’s implementation token for token.
A decoder on the paged KV cache
The pipeline model above recomputes every token at every step and runs one
request at a time. MAX’s own architectures don’t write a pipeline model.
They register a decoder module for the DECODER role, and MAX builds the
pipeline model around it. That pipeline model brings what
Chapter 5 built by hand: continuous
batching, a paged KV cache, and prefix caching.
At each step, the decoder’s forward receives a DecoderInputs:
tokens: the new tokens of every request in the batch, concatenated into one ragged sequence. A request’s prompt arrives whole at its first step, and then one token per step.input_row_offsets: where each request starts intokens, plus the total length at the end.kv: the paged KV cache.per_device_cachesunpacks it into onePagedCacheValuesper device.
Your embedding, norms, and MLP work on ragged tokens unchanged, since they
treat each token on its own. Only attention needs the cache, and two MAX
kernels supply it. rope_split_store_ragged splits the fused Q/K/V
projection, rotates the queries and keys, and writes the keys and values
into the cache’s pages. flash_attention_ragged then attends each new query
to its own request’s cached keys, with a causal mask. The kernels take the
RoPE table packed as (cos, sin) pairs, which rope_table builds from your
rope_cos_sin from
Section 1.4:
Show rope_table, PagedAttention 59 lines
def rope_table(config: LlamaConfig, length: int, device: Device) -> Tensor:
"""Returns the RoPE table as ``[length, head_dim]`` (cos, sin) pairs."""
cos, sin = rope_cos_sin(
config.head_dim, config.rope_theta, 0, length, device
)
half = config.head_dim // 2
pairs = F.stack([cos[:, :half], sin[:, :half]], axis=-1)
return pairs.reshape([length, config.head_dim])
class PagedAttention(Module):
"""Grouped-query attention that reads keys and values from the cache."""
def __init__(
self, config: LlamaConfig, kv_params: KVCacheParams, layer_idx: int
) -> None:
q_dim = config.num_attention_heads * config.head_dim
kv_dim = config.num_key_value_heads * config.head_dim
self.q_proj = Linear(config.hidden_size, q_dim, bias=False)
self.k_proj = Linear(config.hidden_size, kv_dim, bias=False)
self.v_proj = Linear(config.hidden_size, kv_dim, bias=False)
self.o_proj = Linear(q_dim, config.hidden_size, bias=False)
self.n_heads = config.num_attention_heads
self.head_dim = config.head_dim
self.kv_params = kv_params
self.layer_idx = layer_idx
def forward(
self,
x: Tensor,
kv: PagedCacheValues,
rope: Tensor,
input_row_offsets: Tensor,
) -> Tensor:
qkv = F.concat(
[self.q_proj(x), self.k_proj(x), self.v_proj(x)], axis=-1
)
# The kernels read the layer index on the host.
layer = F.constant(self.layer_idx, DType.uint32, device=CPU())
q = rope_split_store_ragged(
kv_params=self.kv_params,
qkv=qkv,
input_row_offsets=input_row_offsets,
freqs_cis=rope,
kv_collection=kv,
layer_idx=layer,
n_heads=self.n_heads,
interleaved=False,
)
out = flash_attention_ragged(
self.kv_params,
input=q.reshape([-1, self.n_heads, self.head_dim]),
input_row_offsets=input_row_offsets,
kv_collection=kv,
layer_idx=layer,
mask_variant=MHAMaskVariant.CAUSAL_MASK,
scale=self.head_dim**-0.5,
)
return self.o_proj(out.reshape([-1, self.n_heads * self.head_dim]))
Saved to minillm/serve/__init__.py.
The layer, the model, and the decoder mirror DecoderLayer, LlamaModel,
and LlamaForCausalLM from
Section 1.6, with the same
attribute names, so the checkpoint loads into them unchanged. MAX constructs
the decoder as MinillmDecoder(config) from the config class. The decoder
returns the logits of each request’s last token, the one its next token
follows:
Show PagedDecoderLayer, PagedLlamaModel, MinillmDecoder 55 lines
class PagedDecoderLayer(Module):
def __init__(
self, config: LlamaConfig, kv_params: KVCacheParams, layer_idx: int
) -> None:
eps = config.rms_norm_eps
self.input_layernorm = RMSNorm(config.hidden_size, eps=eps)
self.self_attn = PagedAttention(config, kv_params, layer_idx)
self.post_attention_layernorm = RMSNorm(config.hidden_size, eps=eps)
self.mlp = MLP(config)
def forward(
self,
x: Tensor,
kv: PagedCacheValues,
rope: Tensor,
input_row_offsets: Tensor,
) -> Tensor:
h = self.input_layernorm(x)
x = x + self.self_attn(h, kv, rope, input_row_offsets)
return x + self.mlp(self.post_attention_layernorm(x))
class PagedLlamaModel(Module):
def __init__(self, config: LlamaConfig, kv_params: KVCacheParams) -> None:
self.embed_tokens = Embedding(config.vocab_size, dim=config.hidden_size)
self.layers = ModuleList(
[
PagedDecoderLayer(config, kv_params, i)
for i in range(config.num_hidden_layers)
]
)
self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
class MinillmDecoder(Module[[DecoderInputs], DecoderOutputs]):
def __init__(self, arch_config: MinillmArchConfig) -> None:
self.config = llama_config(arch_config.huggingface_config)
self.max_seq_len = arch_config.max_seq_len
self.kv_params = arch_config.get_kv_params()
self.model = PagedLlamaModel(self.config, self.kv_params)
self.lm_head = Linear(
self.config.hidden_size, self.config.vocab_size, bias=False
)
def forward(self, inputs: DecoderInputs) -> DecoderOutputs:
(kv,) = per_device_caches(self.kv_params, inputs.kv)
offsets = inputs.input_row_offsets
rope = rope_table(self.config, self.max_seq_len, inputs.tokens.device)
x = self.model.embed_tokens(inputs.tokens)
for layer in self.model.layers:
x = layer(x, kv, rope, offsets)
last = F.gather(x, offsets[1:] - 1, axis=0)
return DecoderOutputs(
next_token_logits=self.lm_head(self.model.norm(last))
)
Saved to minillm/serve/__init__.py.
The registration names the decoder under modules instead of a pipeline
model. Its name, LlamaForCausalLM_ModuleV3, replaces MAX’s own module-based
Llama, so MAX selects it by default:
minillm_decoder_arch = SupportedArchitecture(
name="LlamaForCausalLM_ModuleV3",
task=PipelineTask.TEXT_GENERATION,
example_repo_ids=["HuggingFaceTB/SmolLM2-135M-Instruct"],
default_weights_format=WeightsFormat.safetensors,
default_encoding=MinillmArchConfig.DEFAULT_ENCODING,
supported_encodings=MinillmArchConfig.SUPPORTED_ENCODINGS,
modules={GraphRole.DECODER: MinillmDecoder},
tokenizer=TextTokenizer,
context_type=TextContext,
weight_adapters={WeightsFormat.safetensors: tie_embeddings},
config=MinillmArchConfig,
supports_overlap_scheduler=False,
supports_device_graph_capture=False,
)
ARCHITECTURES = [minillm_arch, minillm_decoder_arch]
Saved to minillm/serve/__init__.py.
This time MAX batches requests. Two prompts run in one batch, and the first still matches MAX’s implementation:
paged = AutoPipeline(
MODEL_ID,
quantization_encoding="float32",
custom_architectures=["minillm.serve"],
max_batch_size=2,
)
batch = paged(PROMPT + ["Once upon a time"], max_new_tokens=16, top_k=1)
paged.close()
for text in batch:
print(text)
print("same as MAX's implementation:", batch[0] == expected[0])
assert batch[0] == expected[0]
The capital of France is Paris. Paris is the largest city in France and the capital of the French department
Once upon a time, I was a young man, a young woman, and a young man again
same as MAX's implementation: True
Serve it
The same package serves over HTTP. --custom-architectures minillm.serve
imports it, and MAX serves the decoder:
start = time.perf_counter()
server = start_max_serve(
MODEL_ID,
"--quantization-encoding",
"float32",
"--custom-architectures",
"minillm.serve",
)
print(f"server ready in {time.perf_counter() - start:.0f} s")
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
completion = client.completions.create(
model=MODEL_ID, prompt=PROMPT[0], max_tokens=16, temperature=0
)
print(PROMPT[0] + completion.choices[0].text)
assert PROMPT[0] + completion.choices[0].text == expected[0]
server.terminate()
_ = server.wait()
server ready in 20 s
The capital of France is Paris. Paris is the largest city in France and the capital of the French department
Your model now answers an OpenAI client exactly as MAX’s implementation does, on MAX’s paged KV cache and scheduler. The handbook’s OpenAI-compatible API page covers the request format. MAX’s guide to serving custom model architectures covers the rest of a production architecture, such as weight adapters for other checkpoint formats and multi-GPU serving.
Exercises
- Register
Qwen3ForCausalLMfrom Section 6.1 as a decoder, and serveQwen/Qwen3-0.6Bthrough it.rope_split_store_raggedtakesq_norm_weight,k_norm_weight, andrms_norm_epsfor Qwen3’s per-head norms. Which parts of the package depend on the model class, and which work unchanged? - Serve each registration in turn,
--no-prefer-module-v3selecting the first, and measure both withmax benchmarkfrom Section 7.2. How does throughput change as you raise--max-concurrency? rope_tablerebuilds the RoPE table at every step. Build it once inMinillmDecoder.__init__instead. Why does MAX then report the table as a weight missing from the checkpoint, and how could the weight adapter supply it?
Next
Section 7.2 measures the
server under load with max benchmark.
7.2 Benchmark a server
A server that answers one request correctly can still fall over under real
traffic. max benchmark measures a running server the way users experience it:
it sends many requests, some at the same time, and records how long each one
waits for its first token and for each token after that. This section benchmarks
MAX’s SmolLM2 and the architecture you served in
Section 7.1, and explains
the numbers.
Show the setup 15 lines
import json
import subprocess
import tempfile
from collections.abc import Iterator
from contextlib import contextmanager
from pathlib import Path
import plotly.graph_objects as go
from plotly.subplots import make_subplots
from minillm import viz
from minillm.config import MODEL_ID
from minillm.server import start_max_serve
WORKDIR = Path(tempfile.mkdtemp())
What to measure
These numbers describe a serving run. The handbook’s LLM inference metrics page defines them in detail:
- Time to first token (TTFT): the time from sending a request to receiving its first token. It includes the wait in the queue and the prefill. Users notice it as the delay before the answer starts.
- Time per output token (TPOT): the average gap between later tokens, one decode step. Users notice it as the reading speed.
- Requests per second and output tokens per second: the throughput of the whole server, which decides how many users one GPU can serve.
Concurrency, the number of requests in flight at once, connects them. More concurrency lets the server batch more requests per step (Chapter 5), which raises throughput, but each request can then wait longer.
Run a benchmark
From a terminal, a benchmark is one command against a running server. This one sends 32 requests of 128 random input tokens each, asks for 32 output tokens per request, and keeps at most 8 in flight:
max benchmark --model HuggingFaceTB/SmolLM2-135M-Instruct --dataset-name random \
--random-input-len 128 --random-output-len 32 \
--num-prompts 32 --max-concurrency 8 --result-filename result.json
The random dataset gives every run the same shape, which suits
comparisons. The default dataset, sharegpt, replays real chat
conversations with varied lengths. The helpers below start a server, run a
benchmark from Python with the same flags, and read the result file, a JSON
object with the summary metrics (request_throughput, mean_ttft_ms,
p95_ttft_ms, and so on) and one record per request (request_records):
Show server, benchmark, summary, show 67 lines
@contextmanager
def server(*flags: str) -> Iterator[None]:
process = start_max_serve(
MODEL_ID, "--quantization-encoding", "float32", *flags
)
try:
yield
finally:
process.terminate()
process.wait()
def benchmark(concurrency: int, *count: str) -> dict:
"""Runs ``max benchmark`` and returns its result file. ``count`` is
``--num-prompts N`` or ``--num-prompts-multiplier M``."""
result = WORKDIR / f"result_{len(list(WORKDIR.glob('*.json')))}.json"
subprocess.run(
[
"max",
"benchmark",
"--model",
MODEL_ID,
"--port",
"8000",
"--dataset-name",
"random",
"--random-input-len",
"128",
"--random-output-len",
"32",
"--max-concurrency",
str(concurrency),
*count,
"--result-filename",
str(result),
"--log-dir",
str(WORKDIR / "logs"),
],
check=True,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
return json.loads(result.read_text())
def summary(run: dict) -> dict[str, float]:
return {
"concurrency": run["max_concurrency"],
"requests/s": run["request_throughput"],
"output tokens/s": run["total_output_tokens"] / run["duration"],
"TTFT (ms)": run["mean_ttft_ms"],
"TPOT (ms)": run["mean_tpot_ms"],
}
def show(runs: list[dict]) -> None:
rows = [summary(run) for run in runs]
print("".join(f"{name:>17}" for name in rows[0]))
for row in rows:
print(
"".join(
f"{value:>17.1f}"
if isinstance(value, float)
else f"{value:>17}"
for value in row.values()
)
)
MAX’s SmolLM2
with server():
max_runs = [
benchmark(1, "--num-prompts", "16"),
benchmark(8, "--num-prompts", "32"),
]
show(max_runs)
concurrency requests/s output tokens/s TTFT (ms) TPOT (ms)
1 1.5 46.6 85.4 19.4
8 8.5 271.8 236.0 22.7
With 8 requests in flight, the server batches them into each decode step. Throughput rises several times, while TPOT barely changes: one decode step costs about the same for 8 sequences as for 1. At this model size, most of a step’s cost is fixed work, such as reading the weights and launching kernels, and little of it is arithmetic per sequence. The handbook’s batching page explains why. TTFT rises because requests that arrive together are prefilled together, and a new request waits for the current step to finish.
The concurrency-8 run sends more requests because max benchmark doesn’t count
all of them. A run this short reports over a trimmed window: at concurrency
above 1 it leaves out the first max_concurrency requests sent and the last
max_concurrency completed, because the server is still filling up or draining
while they run, and divides by the time from the first counted request’s
submission to the last one’s completion. With at least 100 requests at
concurrency above 1, it first looks for a steady-state window automatically and
trims only when it finds none. Either way, the result file says which requests
counted (measured in each request record).
Your architecture
Now run the same benchmark against the package from Section 7.1:
with server("--custom-architectures", "minillm.serve", "--no-prefer-module-v3"):
ours = [
benchmark(1, "--num-prompts", "8"),
benchmark(8, "--num-prompts", "24"),
] # fewer: this server is slow
show(ours)
concurrency requests/s output tokens/s TTFT (ms) TPOT (ms)
1 0.5 14.7 82.3 67.7
8 0.2 6.5 17743.1 94.3
def gain(runs: list[dict]) -> float:
return runs[1]["request_throughput"] / runs[0]["request_throughput"]
def gain_all_requests(runs: list[dict]) -> float:
rates = []
for run in runs:
records = run["request_records"]
start = min(r["submit_time"] for r in records)
end = max(r["complete_time"] for r in records)
rates.append(len(records) / (end - start))
return rates[1] / rates[0]
print(
f"throughput gain from concurrency 8: MAX's {gain(max_runs):.1f}x, "
f"yours {gain(ours):.1f}x"
)
print(
f"over every request, not just the counted ones: "
f"MAX's {gain_all_requests(max_runs):.1f}x, "
f"yours {gain_all_requests(ours):.1f}x"
)
assert gain(max_runs) > 1.5 and gain_all_requests(ours) < 1.5
throughput gain from concurrency 8: MAX's 5.8x, yours 0.4x
over every request, not just the counted ones: MAX's 5.2x, yours 0.7x
Show the code 32 lines
metrics = ["requests/s", "output tokens/s", "TTFT (ms)", "TPOT (ms)"]
fig = make_subplots(
rows=1, cols=4, horizontal_spacing=0.07, subplot_titles=metrics
)
for name, runs, color in [
("MAX's SmolLM2", max_runs, viz.BLUE),
("yours (Section 7.1)", ours, viz.ORANGE),
]:
rows = [summary(run) for run in runs]
for col, metric in enumerate(metrics, start=1):
fig.add_trace(
go.Bar(
x=[f"c = {r['concurrency']}" for r in rows],
y=[r[metric] for r in rows],
name=name,
marker_color=color,
legendgroup=name,
showlegend=col == 1,
hovertemplate=f"{name}, %{{x}}: %{{y:,.1f}} {metric}<extra></extra>",
),
row=1,
col=col,
)
fig.update_yaxes(type="log", title_text="ms (log scale)", row=1, col=3)
fig.update_layout(
title="Serving SmolLM2 at concurrency 1 and 8",
barmode="group",
height=440,
legend=dict(y=-0.12, yanchor="top"),
margin=dict(t=96),
)
viz.show(fig)
The two servers differ in two ways, and earlier sections explain both:
- TPOT is several times higher. The pipeline model from Section 7.1 reruns the whole sequence for every token, as Section 1.7 did, so a step grows with the sequence. The KV cache from Section 1.8 makes it constant.
- Concurrency doesn’t raise throughput. The package declares
max_batch_size1, so the server runs one request at a time and queues the rest. The queue shows up in TTFT: a request waits for the ones ahead of it to finish completely.
The reported throughput even drops, to half the concurrency-1 rate or less,
while counted over every request of the run it drops much less. The trim
explains the difference. Here is every request of your server’s concurrency-8
run, from the result file’s request records. The shaded span is the window that
max benchmark divides by:
Show the code 56 lines
records = sorted(ours[1]["request_records"], key=lambda r: r["submit_time"])
start = records[0]["submit_time"]
counted = [r for r in records if r["measured"]]
window = (
min(r["submit_time"] for r in counted) - start,
max(r["complete_time"] for r in counted) - start,
)
rows = [
f"request {i}" + ("" if r["measured"] else " (not counted)")
for i, r in enumerate(records)
]
fig = go.Figure(
[
go.Bar(
y=rows,
x=[r["ttft_ms"] / 1000 for r in records],
base=[r["submit_time"] - start for r in records],
orientation="h",
name="waiting, then prefill",
marker_color=viz.MUTED,
hovertemplate="%{y}: first token after %{x:.1f} s<extra></extra>",
),
go.Bar(
y=rows,
x=[(r["latency_ms"] - r["ttft_ms"]) / 1000 for r in records],
base=[
r["submit_time"] - start + r["ttft_ms"] / 1000 for r in records
],
orientation="h",
name="generating",
marker_color=[
viz.ORANGE if r["measured"] else viz.GRID for r in records
],
hovertemplate="%{y}: generating for %{x:.1f} s<extra></extra>",
),
]
)
fig.add_vrect(
x0=window[0],
x1=window[1],
fillcolor=viz.ORANGE,
opacity=0.08,
line_width=0,
annotation_text=(
f"counted: {len(counted)} requests in {window[1] - window[0]:.0f} s"
),
annotation_position="top left",
)
fig.update_layout(
title="Your server at concurrency 8: every request of the run",
barmode="overlay",
height=560,
xaxis_title="seconds since the first request was sent",
yaxis=dict(autorange="reversed", showgrid=False),
)
viz.show(fig)
The first 8 requests are sent at once and finish one after another. Each
finished request frees a client slot, so the counted requests, 8 to 15, are
sent while 7 earlier requests still wait in the server’s queue. Their window
also holds the service time of those 7 uncounted requests: about 15 requests’
work, credited to 8. Over every request, from the first submission to the last
completion, throughput stays close to the concurrency-1 rate, as it should for
a server that runs one request at a time. The trim suits a server that batches,
where the ramp-up and the drain distort the numbers. For a server that
serializes requests, read the untrimmed rate, or pass
--skip-first-n-requests 0 --skip-last-n-requests 0.
The engine in Chapter 5 fixed both with a paged KV cache and continuous batching. MAX’s own architectures get them from MAX’s serving stack, which is why a production architecture registers a decoder on MAX’s paged KV cache, as the second registration in Section 7.1 does.
Reading benchmarks well
Measure a server at several concurrency levels, not one, and plot latency
against throughput. --max-concurrency 1,2,4,8,16,32 runs such a sweep in one
command. Pair it with --num-prompts-multiplier 8 instead of a fixed
--num-prompts: each level then sends 8 requests per slot, so the trim never
uses up a level’s requests, as a fixed 32 would at concurrency 32. The cell
below runs the levels one at a time with the same flag, against MAX’s SmolLM2:
with server():
benchmark(1, "--num-prompts", "2") # warm up the new server
sweep = [
benchmark(c, "--num-prompts-multiplier", "8")
for c in [1, 2, 4, 8, 16, 32]
]
show(sweep)
concurrency requests/s output tokens/s TTFT (ms) TPOT (ms)
1 1.4 45.3 109.9 19.2
2 2.8 88.0 95.6 20.4
4 4.8 153.1 148.1 22.2
8 8.3 265.0 280.8 22.1
16 10.8 346.3 609.3 24.2
32 14.2 455.9 999.4 33.3
Show the code 55 lines
TARGET_MS = 500
rows = [summary(run) for run in sweep]
throughput = [r["output tokens/s"] for r in rows]
p95_ttft = [run["p95_ttft_ms"] for run in sweep]
meets = [t <= TARGET_MS for t in p95_ttft]
best = max(
(i for i, ok in enumerate(meets) if ok),
key=lambda i: throughput[i],
default=None,
)
fig = go.Figure(
go.Scatter(
x=throughput,
y=p95_ttft,
mode="lines+markers+text",
text=[f"c = {r['concurrency']}" for r in rows],
textposition="top left",
marker=dict(
size=10, color=[viz.BLUE if ok else viz.RED for ok in meets]
),
line=dict(color=viz.MUTED),
customdata=[
[r["concurrency"], run["mean_tpot_ms"]]
for r, run in zip(rows, sweep)
],
hovertemplate="concurrency %{customdata[0]}: %{x:.0f} tokens/s, "
"p95 TTFT %{y:.0f} ms, mean TPOT %{customdata[1]:.1f} ms<extra></extra>",
)
)
fig.add_hline(
y=TARGET_MS,
line=dict(color=viz.INK_SECONDARY, dash="dash"),
annotation_text=f"target: p95 TTFT {TARGET_MS} ms",
annotation_position="top left",
)
if best is not None:
fig.add_annotation(
x=throughput[best],
y=p95_ttft[best],
text="highest throughput<br>within the target",
showarrow=True,
arrowcolor=viz.INK_SECONDARY,
ax=40,
ay=40,
)
fig.update_layout(
title="MAX's SmolLM2: latency against throughput as concurrency rises",
height=440,
showlegend=False,
xaxis=dict(title="output tokens/s", rangemode="tozero"),
yaxis=dict(
title="p95 time to first token (ms, log scale)", type="log", dtick=1
),
)
viz.show(fig)
Each point is one concurrency level, and red points miss a target of a
95th-percentile TTFT under 500 ms. Throughput keeps rising with concurrency, but
TTFT rises faster once the batch fills. In the run on this page, the highest
throughput within the target came at concurrency 8, and concurrency 16 added
about a third more throughput at more than twice the p95 TTFT. The levels
at 16 and 32 send 128 and 256 requests, enough for max benchmark’s
steady-state window; the others are trimmed. Your numbers depend on your
hardware.
- Benchmark the workload you will serve. Input and output lengths change the results a lot: long prompts stress prefill and TTFT, and long answers stress decode and TPOT.
- Sweep concurrency, as above, with
--num-prompts-multiplierrather than a fixed--num-prompts. Throughput grows ever more slowly once the hardware is busy, while latency keeps rising. - Set a latency target first, such as a TTFT under 500 ms at the 95th
percentile, then find the highest throughput that meets it. The result
file stores percentiles (
p95_ttft_ms,p99_tpot_ms, and so on) next to the means, and--goodput ttft:500 tpot:50adds the share of requests that met every target (goodput_attainment).
The handbook’s
LLM performance benchmarks
page covers benchmark design, and max benchmark --help lists the other
datasets and options.
Exercises
- Extend the sweep of MAX’s SmolLM2 to concurrency 64 and 128 with
--num-prompts-multiplier 4and--goodput ttft:500. Where does throughput stop growing, and at which level doesgoodput_attainmentfall below 90%? - Repeat the MAX run with
--random-input-len 1024. Which metric changes most, and why? - Do the second exercise of
Section 7.1, raise
max_batch_sizein its package to 4, and benchmark it again at concurrency 8.
Next
Section 7.3 constrains what the server generates: JSON that follows a schema, and calls to your own functions.
7.3 Structured output and tools
A chat answer is for people. A program needs output it can parse: a JSON object with the right fields, or the name of a function to call with its arguments. Instructions in the prompt don’t get that reliably, least of all from a small model. This section makes the server guarantee the format with structured output, then builds function calling on top of it.
Show the imports 7 lines
import json
from openai import OpenAI
from pydantic import BaseModel, ConfigDict, Field
from minillm.config import MODEL_ID
from minillm.server import start_max_serve
Structured output is off by default, and --enable-structured-output
turns it on:
server = start_max_serve(
MODEL_ID,
"--quantization-encoding",
"float32",
"--enable-structured-output",
)
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
def chat(messages: list[dict[str, str]], **options: object) -> str:
completion = client.chat.completions.create(
model=MODEL_ID,
messages=messages,
max_tokens=100,
temperature=0,
**options,
)
return completion.choices[0].message.content
JSON that follows a schema
Without a constraint, the model answers in prose:
question = [
{
"role": "user",
"content": "Tell me about Rome: its country and three landmarks.",
}
]
print(chat(question))
Rome, the Eternal City, is a city that has witnessed so much history and transformation. As a helpful AI assistant, I'm here to share with you the fascinating facts about Rome, its country, and three iconic landmarks.
Rome, the Eternal City, is a city that has witnessed so much history and transformation. It's a place where ancient civilizations like the Romans, Greeks, and Etruscans have left their mark, and it's a place where the Romans have built a lasting
Describe the answer you want as a Pydantic model instead. Pydantic turns it
into a JSON schema, which the request passes as response_format:
class City(BaseModel):
model_config = ConfigDict(extra="forbid") # no fields beyond these
name: str
country: str
landmarks: list[str] = Field(min_length=3, max_length=3)
schema = {
"type": "json_schema",
"json_schema": {"name": "City", "schema": City.model_json_schema()},
}
answer = chat(question, response_format=schema)
print(answer)
city = City.model_validate_json(answer)
city
{"name":"Rome","country":"Italy","landmarks":["The Colosseum","The Roman Forum","The Pantheon"]}
City(name='Rome', country='Italy', landmarks=['The Colosseum', 'The Roman Forum', 'The Pantheon'])
model_validate_json raises if the text doesn’t match the schema. With
structured output, the text always matches.
How the server guarantees it
The guarantee comes from the sampler. MAX compiles the schema into a grammar,
and at every decode step the grammar says which tokens may come next: after
{"name": "Rome", , only tokens that continue toward "country". MAX masks
every other token out of the logits before sampling, as the top-p filter in
Section 1.7 did with \(-\infty\),
so the model can only pick a token that keeps the output valid. Generation ends
when the grammar is complete.
The schema fixes the form of the answer and leaves the facts to the model.
The model can skip any requirement that the schema leaves out: without
min_length, a small model may return an empty landmarks list. Put every
requirement you can into the schema, and check the values that matter. The
handbook’s
structured outputs
page covers schema design and the cost of constrained decoding.
Function calling
A tool is a function that the model may ask your program to run: look up the weather, query a database, call an API. The model never runs anything itself. It writes the function’s name and arguments, your code runs the function, and the result goes back into the conversation. A tool call is therefore a structured output whose schema is “one of these functions, with valid arguments for it”.
Here are two tools, each a Python function plus a schema for its arguments:
Show get_weather, convert_currency, arguments 45 lines
RATES = {"USD": 1.0, "EUR": 0.92, "JPY": 151.0}
def get_weather(city: str) -> dict[str, object]:
return {
"city": city,
"temperature_c": 18,
"sky": "light rain",
} # a stand-in for a weather API
def convert_currency(
amount: float, source: str, target: str
) -> dict[str, object]:
return {
"amount": round(amount / RATES[source] * RATES[target], 2),
"currency": target,
}
def arguments(**properties: dict[str, object]) -> dict[str, object]:
return {
"type": "object",
"properties": properties,
"required": list(properties),
"additionalProperties": False,
}
TOOLS = {
"get_weather": (
get_weather,
"Gets the current weather in a city.",
arguments(city={"type": "string"}),
),
"convert_currency": (
convert_currency,
"Converts an amount between currencies.",
arguments(
amount={"type": "number"},
source={"enum": list(RATES)},
target={"enum": list(RATES)},
),
),
}
The schema for a call allows any one of the tools: anyOf lists one object
per tool, each with its fixed name and its own arguments. enum limits
the currencies to the ones RATES knows, so the model can’t ask for one the
function would fail on. The system prompt tells the model what the tools do:
Show call_tool 44 lines
call_schema = {
"anyOf": [
{
"type": "object",
"properties": {"name": {"const": name}, "arguments": args},
"required": ["name", "arguments"],
"additionalProperties": False,
}
for name, (_, _, args) in TOOLS.items()
]
}
system = (
"You can call these functions. Reply with one call as JSON.\n"
+ "\n".join(
f"- {name}: {description} Arguments: {json.dumps(args['properties'])}"
for name, (_, description, args) in TOOLS.items()
)
)
def call_tool(question: str) -> tuple[dict[str, object], dict[str, object]]:
messages = [
{"role": "system", "content": system},
{"role": "user", "content": question},
]
call = json.loads(
chat(
messages,
response_format={
"type": "json_schema",
"json_schema": {"name": "call", "schema": call_schema},
},
)
)
function = TOOLS[call["name"]][0]
return call, function(**call["arguments"])
for question in [
"What's the weather in Tokyo?",
"How much is 20 US dollars in euros?",
]:
call, result = call_tool(question)
print(f"{question}\n call: {call}\n result: {result}")
What's the weather in Tokyo?
call: {'name': 'get_weather', 'arguments': {'city': 'Tokyo'}}
result: {'city': 'Tokyo', 'temperature_c': 18, 'sky': 'light rain'}
How much is 20 US dollars in euros?
call: {'name': 'convert_currency', 'arguments': {'amount': 20, 'source': 'USD', 'target': 'EUR'}}
result: {'amount': 18.4, 'currency': 'EUR'}
Every call parses and runs, since the grammar allows nothing else. Whether the model picks the right tool is up to the model; this one does on these questions.
The last step gives the result back so the model can answer. Here the result goes into the system prompt:
question = "Should I take an umbrella in Paris today?"
call, result = call_tool(question)
print("call:", call)
print(
chat(
[
{
"role": "system",
"content": f"Current weather: {json.dumps(result)}",
},
{"role": "user", "content": question},
]
)
)
call: {'name': 'get_weather', 'arguments': {'city': 'Paris'}}
As a traveler, it's essential to consider the weather forecast and potential weather-related issues before making a decision. Paris is a city known for its rain, and the city's weather can be unpredictable.
If you're planning to visit Paris in the morning, it's best to take an umbrella. The city's rain can be heavy and unpredictable, especially in the mornings. A rain umbrella can provide some protection from the rain, but it's not a guarantee.
If you're
server.terminate()
_ = server.wait()
A 135M-parameter model uses the result loosely. Larger models follow it much more closely, especially ones trained for function calling.
Those models also take tools through the tools parameter of the OpenAI API.
Their chat templates describe the tools to the model in the format it was
trained on, and the model writes calls in its own format. MAX parses those calls
with a tool parser for each model family, such as Llama, Qwen, and DeepSeek,
and returns them as message.tool_calls; tool_choice="required" forces a
call. max serve --tool-parser <name> picks a different parser than the
architecture’s default, and --tool-parser none turns parsing off. SmolLM2’s chat template has no place for tools, which is why this section
built the protocol by hand. The mechanism is the same in both cases: the model
writes a structured call, and your code runs it. The handbook’s
function calling
page covers the API and the agent loops built on it.
Exercises
- Remove
Field(min_length=3, max_length=3)fromCity. What does the model return now? - Add a third tool,
get_time(city), and a question that needs it. Does the model choose it? max benchmarktakes--response-formatwith a JSON schema. Start the server with--enable-structured-output, then benchmark it with and without one, as in Section 7.2. How much does constrained decoding cost per token? Compare TPOT: constrained requests stop at the end of the JSON, so their output lengths differ.
Next
Section 7.4 points the same client at state-of-the-art open models on Modular’s cloud.
7.4 Modular’s cloud
Everything in this book ran on your own machine: models of up to 0.6B parameters, with multi-device meshes simulated on a CPU. Serving state-of-the-art open models to real traffic takes more: the largest models span many GPUs, capacity has to follow demand, and someone has to respond when a node fails. Modular’s cloud is an inference service that runs those models on MAX, the framework you used in this book. This last section points your client code at it by changing the base URL, the API key, and the model name.
import os
from openai import OpenAI
from minillm.config import MODEL_ID
from minillm.server import start_max_serve
Two ways to call a hosted model
Modular’s cloud serves popular open models behind the same OpenAI-compatible API that the earlier sections of this chapter used:
- Shared endpoints are billed per token, and their capacity scales automatically, down to zero when idle.
- Dedicated endpoints reserve GPUs for you and are billed per minute, for consistent latency and guaranteed throughput.
The pricing page compares them.
The same client, a different URL
Every server in Chapter 7 spoke the OpenAI API, and so do Modular’s endpoints. Here is the client from Section 7.1 against a local server again:
server = start_max_serve(MODEL_ID, "--quantization-encoding", "float32")
def ask(client: OpenAI, model: str, question: str) -> str:
completion = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": question}],
max_tokens=60,
temperature=0,
)
return completion.choices[0].message.content
local = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
print(ask(local, MODEL_ID, "What is the capital of France?"))
server.terminate()
_ = server.wait()
The capital of France is Paris.
To call a hosted model instead:
- Sign up at console.modular.com.
- Create an API key with model access on the
API keys page, and set it as the
MODULAR_API_KEYenvironment variable. In Colab, add it as a secret and copy it intoos.environ. - Pick a model on the endpoints
page, and set its model ID as
MODULAR_MODEL.
Only the base URL, the key, and the model name change. The cell skips the request when no key is set:
api_key = os.environ.get("MODULAR_API_KEY")
if api_key is None:
print("Set MODULAR_API_KEY and MODULAR_MODEL to call a hosted model.")
else:
hosted = OpenAI(base_url="https://api.modular.com/v1", api_key=api_key)
print(
ask(
hosted,
os.environ["MODULAR_MODEL"],
"What is the capital of France?",
)
)
Set MODULAR_API_KEY and MODULAR_MODEL to call a hosted model.
The API is the same, so code written against it, such as the structured output
in Section 7.3, works
with hosted models that support the feature. max benchmark measures a hosted
endpoint as it measured your local server. The mcloud backend sends the key
from OPENAI_API_KEY. A hosted endpoint manages its own cache and doesn’t
expose server metrics, so the command skips both. If the model ID isn’t a
Hugging Face repo, add --tokenizer with the repo of its tokenizer:
OPENAI_API_KEY=$MODULAR_API_KEY max benchmark --backend mcloud \
--base-url https://api.modular.com --endpoint /v1/chat/completions \
--no-flush-prefix-cache --no-collect-server-stats \
--model $MODULAR_MODEL \
--dataset-name random --random-input-len 128 --random-output-len 32 \
--num-prompts 32 --max-concurrency 8
Before you choose between shared and dedicated endpoints, benchmark with your real prompt and answer lengths; the latency targets from Section 7.2 apply unchanged. The handbook’s choosing the right model and calculating GPU memory pages help you choose a model and estimate the GPUs it needs.
What you built
You started this book with three lines of AutoPipeline. Since then you:
- wrote SmolLM2 from tensors up, checked it against MAX’s implementation, and added sampling and a KV cache (Chapter 1);
- sharded it across a device mesh (Chapter 2);
- wrote a Mojo kernel and built a graph by hand (Chapter 3);
- tested, benchmarked, and profiled it (Chapter 4);
- built a serving engine with continuous batching, a paged KV cache, prefix caching, and speculative decoding (Chapter 5);
- brought up Qwen3, Gemma 3, and a mixture of experts (Chapter 6);
- served your own architecture, measured it, and constrained its output (Chapter 7).
A model takes the same path into MAX, at a larger size. To keep going:
- Read MAX’s guide to model bring-up and port a model you care about.
- Work through the Mojo GPU puzzles to write the kernels behind it.
- Read the handbook for the ideas behind every chapter.
- Sign up to run state-of-the-art open models on Modular’s cloud, or request a demo to talk about your workload.