1.7 Generation and sampling
Your model turns tokens into next-token logits. This section turns logits into
text: the generation loop, and the sampling strategies behind the temperature,
top_k, and top_p settings from
Section 1.1. Everything runs
on your own model from
Section 1.6.
import logging
import time
import warnings
import numpy as np
import plotly.graph_objects as go
from plotly.subplots import make_subplots
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.functional import random
from max.pipelines import AutoModel, AutoTokenizer
from max.tensor import Tensor
from max.tensor.tensor import TensorType
from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.model import load_model
from minillm.weights import load_checkpoint
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
config = LlamaConfig.from_pretrained()
tokenizer = AutoTokenizer(MODEL_ID)
model = load_model(config, load_checkpoint(), DEVICE)
compiled = model.compile(
TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)
The generation loop needs one thing from the model: the logits for the next token, given the tokens so far.
def next_logits(ids: list[int]) -> Tensor:
logits = compiled(Tensor(np.array([ids]), device=DEVICE))
return logits[0, -1] # [vocab_size]
Greedy decoding
The simplest strategy picks the most likely token every time:
prompt = tokenizer(["The capital of France is"])["input_ids"][0]
ids = list(prompt)
for _ in range(24):
ids.append(int(F.argmax(next_logits(ids)).item()))
print(tokenizer.decode(ids))
The capital of France is Paris. Paris is the largest city in France and the capital of the French department of the Espace de Paris.
Greedy decoding is deterministic, so it is the one strategy you can compare
token for token with MAX’s own pipeline. top_k=1 makes AutoModel greedy
too:
reference = AutoModel(MODEL_ID, quantization_encoding="float32")
expected = reference.generate(
input_ids=[prompt], max_new_tokens=24, top_k=1
).sequences[0]
reference.close()
print("same tokens as AutoModel:", list(expected[-24:]) == ids[-24:])
assert list(expected[-24:]) == ids[-24:]
same tokens as AutoModel: True
Greedy text is coherent but tends to repeat itself and to pick the blandest continuation. Sampling picks tokens at random, in proportion to their probability, to get more varied text.
Temperature
Softmax turns logits \(z\) into probabilities. Dividing the logits by a temperature \(T\) first reshapes the distribution:
\[p_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}\]
Below 1, the likely tokens get likelier; above 1, the distribution flattens toward uniform. As \(T \to 0\), sampling becomes greedy. Move the slider to see it on the twelve likeliest next tokens of a story:
story = tokenizer(["Once upon a time, there was a"])["input_ids"][0]
logits = next_logits(story)
top = np.argsort(logits.to_numpy())[::-1][:12]
labels = [repr(tokenizer.decode([int(i)])) for i in top]
temperatures = [0.1, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5, 2.0]
steps = []
for temperature in temperatures:
probs = F.softmax(logits / temperature).to_numpy()
steps.append(
[
go.Bar(
x=labels,
y=probs[top],
marker_color=viz.BLUE,
hovertemplate="%{x}: %{y:.3f}<extra></extra>",
)
]
)
fig = go.Figure(
layout=dict(
title="Next-token probabilities after 'Once upon a time, there was a'",
yaxis=dict(title="probability", range=[0, 1]),
height=460,
bargap=0.25,
)
)
viz.show(
viz.with_slider(
fig, steps, [str(t) for t in temperatures], "temperature ", active=4
)
)
Sampling with MAX operations
To sample from a distribution, you could compute the softmax and its cumulative sum, then search it with a random number. The Gumbel-max trick does it with operations you already have: add Gumbel noise \(-\log(-\log u)\), with \(u\) uniform in \((0, 1)\), to the scaled logits, and take the argmax:
\[\text{token} = \arg\max_i \left( \frac{z_i}{T} - \log(-\log u_i) \right), \qquad u_i \sim \mathrm{Uniform}(0, 1)\]
The sampled tokens follow the softmax distribution exactly, without computing the softmax. A quick check against the probabilities:
def gumbel_argmax(scores: Tensor) -> Tensor:
u = random.uniform(
scores.shape,
range=(1e-6, 1.0),
dtype=scores.dtype,
device=scores.device,
)
return F.argmax(scores - F.log(-F.log(u)))
toy = Tensor(np.array([2.0, 1.0, 0.5, 0.1, -1.0], np.float32), device=DEVICE)
counts = np.bincount(
[int(gumbel_argmax(toy).item()) for _ in range(2000)], minlength=5
)
print("sampled:", (counts / 2000).round(3))
print("softmax:", F.softmax(toy).to_numpy().round(3))
assert np.abs(counts / 2000 - F.softmax(toy).to_numpy()).max() < 0.05
sampled: [0.556 0.205 0.133 0.08 0.026]
softmax: [0.559 0.205 0.125 0.084 0.028]
Top-k and top-p
Even at a sensible temperature, the long tail of unlikely tokens adds up, and one bad draw can derail the text. Two filters cut the tail before sampling:
- Top-k keeps the \(k\) most likely tokens.
- Top-p (nucleus sampling) keeps the smallest set of tokens whose probabilities add up to at least \(p\), so it keeps few tokens when the model is confident and many when it isn’t.
Both filters work on tokens sorted from most to least likely. F.top_k
returns the \(k\) largest scores already sorted, with their token IDs. For
top-p alone, F.argsort sorts the whole vocabulary and F.gather puts the
scores in that order. F.cumsum over the sorted probabilities then gives
the running total, and F.where drops every token that the total had
already passed \(p\) before reaching. The filtered scores go through the
Gumbel-max trick, and the ID map turns the winning position back into a
token. Without a filter, nothing needs sorting.
from __future__ import annotations
from collections.abc import Callable
from max import functional as F
from max.functional import random
from max.tensor import Tensor
def sample(
logits: Tensor,
temperature: float = 1.0,
top_k: int | None = None,
top_p: float | None = None,
) -> int:
"""Picks a token ID from 1-D ``logits``; ``temperature=0`` is greedy."""
if temperature == 0:
return int(F.argmax(logits).item())
scores, token_ids = logits / temperature, None
if top_k is not None:
scores, token_ids = F.top_k(scores, top_k) # sorted, largest first
if top_p is not None:
if token_ids is None:
token_ids = F.argsort(scores, ascending=False)
scores = F.gather(scores, token_ids, axis=0)
probs = F.softmax(scores)
before = F.cumsum(probs, axis=-1) - probs # mass of likelier tokens
scores = F.where(before < top_p, scores, float("-inf"))
u = random.uniform(
scores.shape,
range=(1e-6, 1.0),
dtype=scores.dtype,
device=scores.device,
)
choice = int(F.argmax(scores - F.log(-F.log(u))).item())
return choice if token_ids is None else int(token_ids.to_numpy()[choice])
def generate(
next_logits: Callable[[list[int]], Tensor],
prompt: list[int],
max_new_tokens: int,
eos_token_id: int | None = None,
**sampling: float | None,
) -> list[int]:
"""Generates up to ``max_new_tokens`` tokens after ``prompt``.
``next_logits`` maps the tokens so far to the next token's logits.
Generation stops early at ``eos_token_id``. Returns only the new tokens.
"""
ids = list(prompt)
for _ in range(max_new_tokens):
token = sample(next_logits(ids), **sampling)
ids.append(token)
if token == eos_token_id:
break
return ids[len(prompt) :]
Saved to minillm/sampling.py.
Top-p adapts to the model’s confidence. After “The capital of France is”, two tokens hold about three quarters of the probability; after “Once upon a time, there was a”, it spreads over many. Move \(p\) and compare how many tokens each prompt keeps (blue) and cuts (gray). The bars show the likeliest 40; the count covers the whole vocabulary:
prompts = ["The capital of France is", "Once upon a time, there was a"]
shown = 40
sorted_probs = []
for text in prompts:
probs = F.softmax(next_logits(tokenizer([text])["input_ids"][0])).to_numpy()
sorted_probs.append(np.sort(probs)[::-1])
ps = [0.5, 0.75, 0.9, 0.95, 0.99]
rank = np.arange(1, shown + 1)
steps = []
for p in ps:
group = []
for axis, probs in zip(["", "2"], sorted_probs):
before = np.cumsum(probs) - probs # the same test as `sample`
kept = before < p
count = int(kept.sum())
kept, top = kept[:shown], probs[:shown]
for mask, color, name in [
(kept, viz.BLUE, "kept"),
(~kept, viz.MUTED, "cut"),
]:
group.append(
go.Bar(
x=rank[mask],
y=top[mask],
marker_color=color,
name=name,
showlegend=False,
xaxis=f"x{axis}",
yaxis=f"y{axis}",
hovertemplate="rank %{x}: %{y:.3f}<extra>"
+ name
+ "</extra>",
)
)
group.append(
go.Scatter(
x=[shown],
y=[0.9],
mode="text",
text=[f"{count:,} kept"],
textposition="middle left",
showlegend=False,
hoverinfo="skip",
xaxis=f"x{axis}",
yaxis=f"y{axis}",
)
)
steps.append(group)
fig = make_subplots(
rows=2,
cols=1,
shared_xaxes=True,
vertical_spacing=0.12,
subplot_titles=[repr(text) for text in prompts],
)
fig.update_layout(
height=560,
bargap=0.15,
title="The likeliest 40 next tokens, sorted, at temperature 1",
)
fig.update_yaxes(range=[0, 1], title="probability")
fig.update_xaxes(title="rank", row=2, col=1)
viz.show(viz.with_slider(fig, steps, [str(p) for p in ps], "top_p ", active=2))
generate stops at the model’s end-of-sequence token, which a chat model
emits when its answer is complete. Try a few settings. random.set_seed
makes a sampled run repeatable:
for settings in [
dict(temperature=0),
dict(temperature=0.7, top_p=0.9),
dict(temperature=1.0, top_k=50),
dict(temperature=1.5),
]:
random.set_seed(0)
new = generate(next_logits, story, 24, config.eos_token_id, **settings)
print(f"{settings}\n {tokenizer.decode(new)!r}\n")
{'temperature': 0}
' young woman named Sarah who was struggling to find her place in the world. She had always been a bit of a rebel'
{'temperature': 0.7, 'top_p': 0.9}
' young girl named Lily who was determined to be a better version of herself. She had always been the "perfect" girl'
{'temperature': 1.0, 'top_k': 50}
' young musician who had spent their life playing and composing. He was known as "The Weaver" due to his exceptional skill'
{'temperature': 1.5}
' neighboring realm on your planet, mist. Even then; Unlocking beastellandsis deathshire istsubbralgenaki'
Higher temperatures read less predictably. At 1.5 without a filter, the text
turns to gibberish within a few words. Many chat applications use settings near
temperature=0.7 and top_p=0.9. The handbook’s
inference parameters
page covers these and the other settings servers accept, such as stop sequences
and repetition penalties.
The cost of each token
Generating the token after \(n\) tokens runs the model over all \(n\) of them. Time that step at several lengths, with random token IDs since only the length matters:
rng = np.random.default_rng(0)
lengths = [16, 64, 256, 512, 1024, 2048]
step_ms = []
for length in lengths:
ids = rng.integers(0, config.vocab_size, length).tolist()
next_logits(ids).to_numpy() # warm up this length
times = []
for _ in range(3):
start = time.perf_counter()
next_logits(ids).to_numpy()
times.append((time.perf_counter() - start) * 1000)
step_ms.append(min(times))
print(f"{length:5} tokens: {step_ms[-1]:6.1f} ms per new token")
fig = go.Figure(
go.Scatter(
x=lengths,
y=step_ms,
mode="lines+markers",
line_color=viz.ORANGE,
hovertemplate="%{x} tokens: %{y:.1f} ms<extra></extra>",
)
)
fig.update_layout(
title="Time to generate one token, recomputing the whole sequence",
xaxis_title="sequence length (tokens)",
yaxis_title="ms per token",
yaxis_rangemode="tozero",
height=380,
)
viz.show(fig)
16 tokens: 82.2 ms per new token
64 tokens: 104.0 ms per new token
256 tokens: 101.7 ms per new token
512 tokens: 186.9 ms per new token
1024 tokens: 570.1 ms per new token
2048 tokens: 1906.7 ms per new token
In the run shown here, on an Apple-silicon Mac with a Metal GPU, a step took about 80 to 105 ms up to 256 tokens, about 570 ms at 1,024 tokens, and about 1.9 s at 2,048. Treat these numbers as an example: yours depend on your hardware and vary from run to run. For short sequences the step costs about the same at any length, because fixed work, such as reading every weight and launching each kernel, dominates. Past a few hundred tokens the cost climbs faster than the length itself, because attention compares every position with every earlier one. Other devices give other numbers with the same shape.
The keys and values of earlier positions never change, so all the work of recomputing them is wasted. Section 1.8 caches them. The handbook’s LLM inference metrics page explains why this per-token time, the TPOT from Section 1.1, is the number users feel while a reply streams in.
Exercises
- Sample the same prompt 5 times at
temperature=1.0with and withouttop_p=0.9. How often does each run go off topic? - Implement min-p sampling: keep tokens whose probability is at least
min_ptimes the top token’s probability. Which MAX operations do you need? - Plot how many tokens top-p keeps at each step of a generation. When is the model confident?
Next
Section 1.8 adds a KV cache so each step processes one token instead of the whole sequence.