4.3 Benchmarking kernels and layers

Open in Colab Open in Kaggle View on GitHub

A benchmark is an experiment, and most quick timings are bad experiments: they include compilation, measure the time to launch work instead of the time to finish it, or report one noisy sample. This section writes a small benchmark harness that avoids those mistakes, then uses it to show why decoding one token is limited by memory traffic and prefill by arithmetic.

import logging
import time
import warnings

import numpy as np
import plotly.graph_objects as go
from max.driver import CPU
from max.dtype import DType
from max.nn import Linear
from max.tensor import Tensor, default_device, default_dtype
from max.tensor.tensor import TensorType
from plotly.subplots import make_subplots

from minillm import viz
from minillm.device import pick_device
from minillm.config import LlamaConfig
from minillm.model import DecoderLayer
from minillm.weights import load_checkpoint, select

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU
config = LlamaConfig.from_pretrained()

A harness

The harness follows four rules, as MAX’s layer benchmarks do:

  1. Warm up. The first calls compile kernels and fill caches, so don’t time them.
  2. Wait for the result. Work can run asynchronously, especially on a GPU. Reading the output (to_numpy()) forces it to finish.
  3. Repeat. One sample is mostly noise, so take many.
  4. Report the median and a tail percentile. Outliers pull the mean; the median shows the typical run and p90 shows how slow an unlucky one gets.
from __future__ import annotations

import time
from collections.abc import Callable
from dataclasses import dataclass

import numpy as np


@dataclass
class Timing:
    """Milliseconds per call over the measured iterations."""

    median_ms: float
    p90_ms: float
    min_ms: float

    def __str__(self) -> str:
        return f"median {self.median_ms:7.2f} ms  p90 {self.p90_ms:7.2f} ms"


def benchmark(
    fn: Callable[[], object], warmup: int = 3, iterations: int = 20
) -> Timing:
    """Times ``fn``, which must return a fully computed result (for example,
    a NumPy array read back from the device)."""
    for _ in range(warmup):
        fn()
    samples = []
    for _ in range(iterations):
        start = time.perf_counter()
        fn()
        samples.append((time.perf_counter() - start) * 1000)
    return Timing(
        median_ms=float(np.median(samples)),
        p90_ms=float(np.percentile(samples, 90)),
        min_ms=float(np.min(samples)),
    )

Saved to minillm/bench.py.

Memory-bound or compute-bound?

Every projection in the model is a matrix multiplication, x @ W.T, with x holding one row per token. Its cost has two parts: reading W from memory, which doesn’t depend on the number of tokens, and the arithmetic, which grows with it. For \(n\) tokens and a \(K \to N\) projection in float32 (4 bytes per number), the ratio of the two is the operation’s arithmetic intensity:

\[I = \frac{\text{FLOPs}}{\text{bytes moved}} = \frac{2nKN}{4\,(KN + nK + nN)}\]

A device can’t run faster than its compute peak \(P_{\text{peak}}\), nor faster than its memory bandwidth \(B\) can feed it, which gives the roofline:

\[P \le \min\left(P_{\text{peak}},\; B \cdot I\right)\]

With few tokens, \(I\) is small and memory sets the limit; with many, \(I\) grows until compute does. Measure SmolLM2’s gate_proj (\(K = 576\), \(N = 1536\)) for 1 to 2048 tokens:

with default_device(DEVICE), default_dtype(DType.float32):
    proj = Linear(config.hidden_size, config.intermediate_size, bias=False)
run = proj.compile(
    TensorType(DType.float32, ("tokens", config.hidden_size), device=DEVICE)
)

K, N = config.hidden_size, config.intermediate_size
rows = []
for tokens in [1, 4, 16, 64, 256, 1024, 2048]:
    x = Tensor(
        np.random.default_rng(0)
        .standard_normal((tokens, K))
        .astype(np.float32),
        device=DEVICE,
    )
    t = benchmark(lambda: run(x).to_numpy())
    flops = 2 * tokens * K * N
    bytes_moved = 4 * (K * N + tokens * (K + N))
    rows.append(
        (
            tokens,
            t.median_ms,
            flops / t.median_ms / 1e6,
            bytes_moved / t.median_ms / 1e6,
        )
    )
    print(
        f"{tokens:5} tokens  {t}  {rows[-1][2]:7.1f} GFLOP/s  {rows[-1][3]:6.1f} GB/s"
    )
    1 tokens  median    0.42 ms  p90    0.74 ms      4.2 GFLOP/s     8.4 GB/s
    4 tokens  median    0.33 ms  p90    0.42 ms     21.2 GFLOP/s    10.7 GB/s
   16 tokens  median    0.31 ms  p90    0.40 ms     91.4 GFLOP/s    11.9 GB/s
   64 tokens  median    0.72 ms  p90    0.79 ms    156.8 GFLOP/s     5.6 GB/s
  256 tokens  median    2.23 ms  p90    3.10 ms    203.1 GFLOP/s     2.6 GB/s
 1024 tokens  median    8.72 ms  p90   10.43 ms    207.9 GFLOP/s     1.4 GB/s
 2048 tokens  median   13.43 ms  p90   15.57 ms    269.7 GFLOP/s     1.6 GB/s
tokens, ms, gflops, _ = zip(*rows)
intensity = [
    2 * t * K * N / (4 * (K * N + t * (K + N))) for t in tokens
]  # FLOP per byte moved

fig = make_subplots(
    rows=1,
    cols=2,
    horizontal_spacing=0.14,
    subplot_titles=["Time per call", "Arithmetic throughput against intensity"],
)
fig.add_trace(
    go.Scatter(
        x=tokens,
        y=ms,
        mode="lines+markers",
        line_color=viz.BLUE,
        showlegend=False,
        hovertemplate="%{x} tokens: %{y:.3f} ms<extra></extra>",
    ),
    row=1,
    col=1,
)
fig.add_trace(
    go.Scatter(
        x=intensity,
        y=gflops,
        mode="lines+markers+text",
        line_color=viz.ORANGE,
        text=[str(t) for t in tokens],
        textposition="top left",
        showlegend=False,
        customdata=tokens,
        hovertemplate="%{customdata} tokens<br>%{x:.1f} FLOP/byte<br>%{y:.0f} GFLOP/s<extra></extra>",
    ),
    row=1,
    col=2,
)
fig.update_xaxes(title_text="tokens", type="log", dtick=1, row=1, col=1)
fig.update_yaxes(title_text="ms (log scale)", type="log", dtick=1, row=1, col=1)
fig.update_xaxes(
    title_text="FLOP per byte moved (log scale)",
    type="log",
    dtick=1,
    row=1,
    col=2,
)
fig.update_yaxes(
    title_text="GFLOP/s achieved", rangemode="tozero", row=1, col=2
)
fig.update_layout(
    title="gate_proj (576 to 1536) from 1 to 2048 tokens; labels are token counts",
    height=420,
)
viz.show(fig)

Read the two plots together, and hover over the points.

With few tokens, the time barely grows: 1 to 16 tokens all take about 0.3 to 0.4 ms on this page’s machine, an Apple-silicon Mac with a Metal GPU. Treat these numbers as an example: yours depend on your hardware and vary from run to run. Each weight is used for only a few multiply-adds after it is loaded, so arithmetic throughput is low and the operation is memory-bound. At a matrix this small, though, most of that time is a fixed cost per call, for launching the work and reading the result back: the single-token call moves W’s 3.5 MB at only about 10 GB/s, far below the GPU’s memory bandwidth. In a large model, reading the weights dominates instead.

With many tokens, each weight, once loaded, is reused for every token, so arithmetic throughput climbs and the time grows with the number of tokens. The operation moves toward the compute-bound end, where the hardware’s arithmetic units set the limit.

Decoding one token per step runs every weight matrix at the memory-bound end. That is why per-token latency depends on memory bandwidth, and why serving engines batch many requests into each step (Chapter 5): here, 16 tokens cost barely more than one. Prefill processes the whole prompt at once and runs toward the compute-bound end. The handbook’s GPU memory page explains the same trade-off on GPUs.

Benchmark a layer

The same harness measures a whole decoder layer at the shapes that matter in serving: decode steps (one token) for several batch sizes, and prefills of several lengths.

with default_device(DEVICE), default_dtype(DType.float32):
    layer = DecoderLayer(config)
layer.load_state_dict(select(load_checkpoint(), "model.layers.0."))
run_layer = layer.compile(
    TensorType(
        DType.float32, ("batch", "seq_len", config.hidden_size), device=DEVICE
    )
)

print("shape (batch x seq)       time per call       tokens/s")
medians = {}
for batch, seq in [(1, 1), (8, 1), (32, 1), (1, 128), (1, 512)]:
    x = Tensor(
        np.random.default_rng(0)
        .standard_normal((batch, seq, config.hidden_size))
        .astype(np.float32),
        device=DEVICE,
    )
    t = benchmark(lambda: run_layer(x).to_numpy(), warmup=5, iterations=50)
    medians[batch, seq] = t.median_ms
    print(f"{batch:5} x {seq:<5}  {t}  {batch * seq / t.median_ms * 1000:9.0f}")
print(
    f"a decode step for 32 sequences costs {medians[32, 1] / medians[1, 1]:.1f}x one sequence's"
)

labels = [f"{b} x {s}" for b, s in medians]
fig = make_subplots(
    rows=1,
    cols=2,
    horizontal_spacing=0.12,
    subplot_titles=["Time per call (ms)", "Tokens per second"],
)
fig.add_trace(
    go.Bar(
        x=labels,
        y=list(medians.values()),
        marker_color=viz.BLUE,
        showlegend=False,
        hovertemplate="%{x}: %{y:.2f} ms<extra></extra>",
    ),
    row=1,
    col=1,
)
fig.add_trace(
    go.Bar(
        x=labels,
        y=[b * s / ms * 1000 for (b, s), ms in medians.items()],
        marker_color=viz.AQUA,
        showlegend=False,
        hovertemplate="%{x}: %{y:,.0f} tokens/s<extra></extra>",
    ),
    row=1,
    col=2,
)
fig.update_xaxes(title_text="batch x sequence length")
fig.update_layout(title="One decoder layer at serving shapes", height=400)
viz.show(fig)
shape (batch x seq)       time per call       tokens/s
    1 x 1      median    0.67 ms  p90    0.86 ms       1490
    8 x 1      median    0.80 ms  p90    1.07 ms       9971
   32 x 1      median    0.88 ms  p90    1.16 ms      36358
    1 x 128    median    2.57 ms  p90    3.25 ms      49748
    1 x 512    median    7.92 ms  p90    9.58 ms      64663
a decode step for 32 sequences costs 1.3x one sequence's

On this machine, a decode step for 32 sequences costs about 1.2 to 1.3 times a single sequence’s step, so throughput in tokens per second rises sharply with batch size. Timings on a laptop are noisy, which is why the harness reports a percentile alongside the median. A 512-token prefill runs through one layer at about 60,000 tokens per second; the model has 30 layers, so a whole-model prefill would run at roughly 2,000 tokens per second.

These numbers describe this machine’s GPU. On another CPU or GPU the curves have the same shape at different scales, although the time isn’t always monotonic in size: the kernels behind each operation choose different strategies for different shapes, so on some devices a smaller batch takes longer than a larger one (this page’s run doesn’t show it). Measure the shapes you serve. MAX’s own layer benchmarks run this kind of sweep for every layer, and its kernel benchmarks do the same for individual kernels. For whole-server measurements, Section 7.2 uses max benchmark. The handbook’s LLM performance benchmarks page explains how to read published numbers.

Exercises

  1. Run the matmul sweep for down_proj (1536 → 576). Where does it switch from memory-bound to compute-bound?
  2. Benchmark the same layer with iterations=3 five times. How much do the medians vary? How many iterations make the result stable?
  3. Compare the eager and compiled layer at batch 1, sequence 1. Where does eager execution spend its time?

Next

Section 4.4 looks inside a run: which operations take the time, using MAX’s profiling tools.