4.3 Benchmarking kernels and layers
A benchmark is an experiment, and most quick timings are bad experiments: they include compilation, measure the time to launch work instead of the time to finish it, or report one noisy sample. This section writes a small benchmark harness that avoids those mistakes, then uses it to show why decoding one token is limited by memory traffic and prefill by arithmetic.
import logging
import time
import warnings
import numpy as np
import plotly.graph_objects as go
from max.driver import CPU
from max.dtype import DType
from max.nn import Linear
from max.tensor import Tensor, default_device, default_dtype
from max.tensor.tensor import TensorType
from plotly.subplots import make_subplots
from minillm import viz
from minillm.device import pick_device
from minillm.config import LlamaConfig
from minillm.model import DecoderLayer
from minillm.weights import load_checkpoint, select
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
config = LlamaConfig.from_pretrained()
A harness
The harness follows four rules, as MAX’s layer benchmarks do:
- Warm up. The first calls compile kernels and fill caches, so don’t time them.
- Wait for the result. Work can run asynchronously, especially on a GPU.
Reading the output (
to_numpy()) forces it to finish. - Repeat. One sample is mostly noise, so take many.
- Report the median and a tail percentile. Outliers pull the mean; the median shows the typical run and p90 shows how slow an unlucky one gets.
from __future__ import annotations
import time
from collections.abc import Callable
from dataclasses import dataclass
import numpy as np
@dataclass
class Timing:
"""Milliseconds per call over the measured iterations."""
median_ms: float
p90_ms: float
min_ms: float
def __str__(self) -> str:
return f"median {self.median_ms:7.2f} ms p90 {self.p90_ms:7.2f} ms"
def benchmark(
fn: Callable[[], object], warmup: int = 3, iterations: int = 20
) -> Timing:
"""Times ``fn``, which must return a fully computed result (for example,
a NumPy array read back from the device)."""
for _ in range(warmup):
fn()
samples = []
for _ in range(iterations):
start = time.perf_counter()
fn()
samples.append((time.perf_counter() - start) * 1000)
return Timing(
median_ms=float(np.median(samples)),
p90_ms=float(np.percentile(samples, 90)),
min_ms=float(np.min(samples)),
)
Saved to minillm/bench.py.
Memory-bound or compute-bound?
Every projection in the model is a matrix multiplication, x @ W.T, with
x holding one row per token. Its cost has two parts: reading W from
memory, which doesn’t depend on the number of tokens, and the arithmetic,
which grows with it. For \(n\) tokens and a \(K \to N\) projection in
float32 (4 bytes per number), the ratio of the two is the operation’s
arithmetic intensity:
\[I = \frac{\text{FLOPs}}{\text{bytes moved}} = \frac{2nKN}{4\,(KN + nK + nN)}\]
A device can’t run faster than its compute peak \(P_{\text{peak}}\), nor faster than its memory bandwidth \(B\) can feed it, which gives the roofline:
\[P \le \min\left(P_{\text{peak}},\; B \cdot I\right)\]
With few tokens, \(I\) is small and memory sets the limit; with many, \(I\)
grows until compute does. Measure SmolLM2’s gate_proj
(\(K = 576\), \(N = 1536\)) for 1 to 2048 tokens:
with default_device(DEVICE), default_dtype(DType.float32):
proj = Linear(config.hidden_size, config.intermediate_size, bias=False)
run = proj.compile(
TensorType(DType.float32, ("tokens", config.hidden_size), device=DEVICE)
)
K, N = config.hidden_size, config.intermediate_size
rows = []
for tokens in [1, 4, 16, 64, 256, 1024, 2048]:
x = Tensor(
np.random.default_rng(0)
.standard_normal((tokens, K))
.astype(np.float32),
device=DEVICE,
)
t = benchmark(lambda: run(x).to_numpy())
flops = 2 * tokens * K * N
bytes_moved = 4 * (K * N + tokens * (K + N))
rows.append(
(
tokens,
t.median_ms,
flops / t.median_ms / 1e6,
bytes_moved / t.median_ms / 1e6,
)
)
print(
f"{tokens:5} tokens {t} {rows[-1][2]:7.1f} GFLOP/s {rows[-1][3]:6.1f} GB/s"
)
1 tokens median 0.42 ms p90 0.74 ms 4.2 GFLOP/s 8.4 GB/s
4 tokens median 0.33 ms p90 0.42 ms 21.2 GFLOP/s 10.7 GB/s
16 tokens median 0.31 ms p90 0.40 ms 91.4 GFLOP/s 11.9 GB/s
64 tokens median 0.72 ms p90 0.79 ms 156.8 GFLOP/s 5.6 GB/s
256 tokens median 2.23 ms p90 3.10 ms 203.1 GFLOP/s 2.6 GB/s
1024 tokens median 8.72 ms p90 10.43 ms 207.9 GFLOP/s 1.4 GB/s
2048 tokens median 13.43 ms p90 15.57 ms 269.7 GFLOP/s 1.6 GB/s
tokens, ms, gflops, _ = zip(*rows)
intensity = [
2 * t * K * N / (4 * (K * N + t * (K + N))) for t in tokens
] # FLOP per byte moved
fig = make_subplots(
rows=1,
cols=2,
horizontal_spacing=0.14,
subplot_titles=["Time per call", "Arithmetic throughput against intensity"],
)
fig.add_trace(
go.Scatter(
x=tokens,
y=ms,
mode="lines+markers",
line_color=viz.BLUE,
showlegend=False,
hovertemplate="%{x} tokens: %{y:.3f} ms<extra></extra>",
),
row=1,
col=1,
)
fig.add_trace(
go.Scatter(
x=intensity,
y=gflops,
mode="lines+markers+text",
line_color=viz.ORANGE,
text=[str(t) for t in tokens],
textposition="top left",
showlegend=False,
customdata=tokens,
hovertemplate="%{customdata} tokens<br>%{x:.1f} FLOP/byte<br>%{y:.0f} GFLOP/s<extra></extra>",
),
row=1,
col=2,
)
fig.update_xaxes(title_text="tokens", type="log", dtick=1, row=1, col=1)
fig.update_yaxes(title_text="ms (log scale)", type="log", dtick=1, row=1, col=1)
fig.update_xaxes(
title_text="FLOP per byte moved (log scale)",
type="log",
dtick=1,
row=1,
col=2,
)
fig.update_yaxes(
title_text="GFLOP/s achieved", rangemode="tozero", row=1, col=2
)
fig.update_layout(
title="gate_proj (576 to 1536) from 1 to 2048 tokens; labels are token counts",
height=420,
)
viz.show(fig)
Read the two plots together, and hover over the points.
With few tokens, the time barely grows: 1 to 16 tokens all take about 0.3 to 0.4
ms on this page’s machine, an Apple-silicon Mac with a Metal GPU. Treat these
numbers as an example: yours depend on your hardware and vary from run to run.
Each weight is used for only a few multiply-adds after it is loaded, so
arithmetic throughput is low and the operation is memory-bound. At a matrix
this small, though, most of that time is a fixed cost per call, for launching
the work and reading the result back: the single-token call moves W’s 3.5 MB
at only about 10 GB/s, far below the GPU’s memory bandwidth. In a large model,
reading the weights dominates instead.
With many tokens, each weight, once loaded, is reused for every token, so arithmetic throughput climbs and the time grows with the number of tokens. The operation moves toward the compute-bound end, where the hardware’s arithmetic units set the limit.
Decoding one token per step runs every weight matrix at the memory-bound end. That is why per-token latency depends on memory bandwidth, and why serving engines batch many requests into each step (Chapter 5): here, 16 tokens cost barely more than one. Prefill processes the whole prompt at once and runs toward the compute-bound end. The handbook’s GPU memory page explains the same trade-off on GPUs.
Benchmark a layer
The same harness measures a whole decoder layer at the shapes that matter in serving: decode steps (one token) for several batch sizes, and prefills of several lengths.
with default_device(DEVICE), default_dtype(DType.float32):
layer = DecoderLayer(config)
layer.load_state_dict(select(load_checkpoint(), "model.layers.0."))
run_layer = layer.compile(
TensorType(
DType.float32, ("batch", "seq_len", config.hidden_size), device=DEVICE
)
)
print("shape (batch x seq) time per call tokens/s")
medians = {}
for batch, seq in [(1, 1), (8, 1), (32, 1), (1, 128), (1, 512)]:
x = Tensor(
np.random.default_rng(0)
.standard_normal((batch, seq, config.hidden_size))
.astype(np.float32),
device=DEVICE,
)
t = benchmark(lambda: run_layer(x).to_numpy(), warmup=5, iterations=50)
medians[batch, seq] = t.median_ms
print(f"{batch:5} x {seq:<5} {t} {batch * seq / t.median_ms * 1000:9.0f}")
print(
f"a decode step for 32 sequences costs {medians[32, 1] / medians[1, 1]:.1f}x one sequence's"
)
labels = [f"{b} x {s}" for b, s in medians]
fig = make_subplots(
rows=1,
cols=2,
horizontal_spacing=0.12,
subplot_titles=["Time per call (ms)", "Tokens per second"],
)
fig.add_trace(
go.Bar(
x=labels,
y=list(medians.values()),
marker_color=viz.BLUE,
showlegend=False,
hovertemplate="%{x}: %{y:.2f} ms<extra></extra>",
),
row=1,
col=1,
)
fig.add_trace(
go.Bar(
x=labels,
y=[b * s / ms * 1000 for (b, s), ms in medians.items()],
marker_color=viz.AQUA,
showlegend=False,
hovertemplate="%{x}: %{y:,.0f} tokens/s<extra></extra>",
),
row=1,
col=2,
)
fig.update_xaxes(title_text="batch x sequence length")
fig.update_layout(title="One decoder layer at serving shapes", height=400)
viz.show(fig)
shape (batch x seq) time per call tokens/s
1 x 1 median 0.67 ms p90 0.86 ms 1490
8 x 1 median 0.80 ms p90 1.07 ms 9971
32 x 1 median 0.88 ms p90 1.16 ms 36358
1 x 128 median 2.57 ms p90 3.25 ms 49748
1 x 512 median 7.92 ms p90 9.58 ms 64663
a decode step for 32 sequences costs 1.3x one sequence's
On this machine, a decode step for 32 sequences costs about 1.2 to 1.3 times a single sequence’s step, so throughput in tokens per second rises sharply with batch size. Timings on a laptop are noisy, which is why the harness reports a percentile alongside the median. A 512-token prefill runs through one layer at about 60,000 tokens per second; the model has 30 layers, so a whole-model prefill would run at roughly 2,000 tokens per second.
These numbers describe this machine’s GPU. On another CPU or GPU the curves have
the same shape at different scales, although the time isn’t always monotonic in
size: the kernels behind each operation choose different strategies for
different shapes, so on some devices a smaller batch takes longer than a larger
one (this page’s run doesn’t show it). Measure the shapes you serve. MAX’s own
layer benchmarks
run this kind of sweep for every layer, and its kernel benchmarks do the same
for individual kernels. For whole-server measurements,
Section 7.2 uses
max benchmark. The handbook’s
LLM performance benchmarks
page explains how to read published numbers.
Exercises
- Run the matmul sweep for
down_proj(1536 → 576). Where does it switch from memory-bound to compute-bound? - Benchmark the same layer with
iterations=3five times. How much do the medians vary? How many iterations make the result stable? - Compare the eager and compiled layer at batch 1, sequence 1. Where does eager execution spend its time?
Next
Section 4.4 looks inside a run: which operations take the time, using MAX’s profiling tools.