6.1 Qwen3: normalized queries and keys
Most open models are close relatives of the Llama architecture you built in
Chapter 1. To bring up a new one, you find
what changed and why. This section adds Qwen3, here the 0.6-billion-parameter
model Qwen/Qwen3-0.6B. It subclasses your Llama classes, overrides only what
differs, and verifies the result with the tools from
Chapter 4.
import logging
import warnings
import numpy as np
import plotly.graph_objects as go
import torch
from max.driver import CPU
from max.dtype import DType
from max.pipelines import AutoTokenizer
from max.tensor import Tensor
from max.tensor.tensor import TensorType
from transformers import AutoModelForCausalLM
from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.model import load_model
from minillm.sampling import generate
from minillm.testing import compare_outputs
from minillm.weights import load_checkpoint
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
QWEN3 = "Qwen/Qwen3-0.6B"
What changed
Compare the two configs:
smol, qwen = (
LlamaConfig.from_pretrained(MODEL_ID),
LlamaConfig.from_pretrained(QWEN3),
)
for field in [
"vocab_size",
"hidden_size",
"num_hidden_layers",
"num_attention_heads",
"num_key_value_heads",
"head_dim",
"intermediate_size",
"rope_theta",
"rms_norm_eps",
]:
print(f"{field:22} {getattr(smol, field):>10} {getattr(qwen, field):>10}")
vocab_size 49152 151936
hidden_size 576 1024
num_hidden_layers 30 28
num_attention_heads 9 16
num_key_value_heads 3 8
head_dim 64 128
intermediate_size 1536 3072
rope_theta 100000 1000000
rms_norm_eps 1e-05 1e-06
Four differences matter:
- A larger vocabulary, 151,936 tokens, which covers many languages and code. The embedding alone holds 156M parameters, about a quarter of the model.
- An explicit
head_dim: 16 heads of 128 dimensions make 2048, wider than the 1024-dimension hidden state.q_projprojects up, ando_projprojects back down. The config class from Section 1.3 readshead_dimwhen a model sets it. - A larger RoPE base, 1,000,000, so the rotations turn more slowly and positions stay distinguishable over longer contexts.
- QK-norm: an RMSNorm on every query and key head, before RoPE.
The first three are config values that your code already handles. QK-norm needs new code.
QK-norm
Attention scores are dot products of queries and keys. During training, nothing stops some heads’ queries and keys from growing until a few scores dominate every softmax, which can destabilize the training of large models. Normalizing the queries and keys, with a learned scale per dimension, bounds the scores. Qwen3 and Gemma 3 normalize each head separately; OLMo 2 normalizes the whole query and key projections.
For each head’s query \(q\) (and likewise each key), with \(d\) = head_dim and
a learned weight \(w\) of size \(d\):
\[\hat{q} = \frac{q}{\sqrt{\tfrac{1}{d}\sum_j q_j^2 + \epsilon}} \odot w\]
The division gives every query the same root mean square, so the length of
\(\hat q\) depends on \(w\) and on \(q\)’s direction, not on \(q\)’s size. A score
\(\hat q \cdot \hat k\) is then bounded however large \(q\) and \(k\) get. In code,
QK-norm is two RMSNorms of size head_dim inside attention, applied to
each head after the projection and before RoPE. The other classes only swap
in the new attention:
from __future__ import annotations
from max.graph import Dim
from max.nn import ModuleList, RMSNorm
from max.tensor import Tensor
from minillm.attention import Attention
from minillm.config import LlamaConfig
from minillm.layers import apply_rope, rope_cos_sin
from minillm.model import DecoderLayer, LlamaForCausalLM, LlamaModel
class Qwen3Attention(Attention):
"""Llama attention with an RMSNorm on each query and key head."""
def __init__(self, config: LlamaConfig) -> None:
super().__init__(config)
self.q_norm = RMSNorm(self.head_dim, eps=config.rms_norm_eps)
self.k_norm = RMSNorm(self.head_dim, eps=config.rms_norm_eps)
def project(
self, x: Tensor, start: int | Dim | Tensor
) -> tuple[Tensor, Tensor, Tensor]:
batch, seq, _ = x.shape
q = self.q_proj(x).reshape([batch, seq, self.num_heads, self.head_dim])
k = self.k_proj(x).reshape(
[batch, seq, self.num_kv_heads, self.head_dim]
)
v = self.v_proj(x).reshape(
[batch, seq, self.num_kv_heads, self.head_dim]
)
q, k = self.q_norm(q), self.k_norm(k)
cos, sin = rope_cos_sin(
self.head_dim, self.rope_theta, start, seq, x.mesh
)
return apply_rope(q, cos, sin), apply_rope(k, cos, sin), v
class Qwen3DecoderLayer(DecoderLayer):
def __init__(self, config: LlamaConfig) -> None:
super().__init__(config)
self.self_attn = Qwen3Attention(config)
class Qwen3Model(LlamaModel):
def __init__(self, config: LlamaConfig) -> None:
super().__init__(config)
self.layers = ModuleList(
[Qwen3DecoderLayer(config) for _ in range(config.num_hidden_layers)]
)
class Qwen3ForCausalLM(LlamaForCausalLM):
"""Qwen3: a Llama model whose attention normalizes queries and keys."""
def __init__(self, config: LlamaConfig) -> None:
super().__init__(config)
self.model = Qwen3Model(config)
Saved to minillm/qwen3.py.
The new parameters are named q_norm and k_norm, as in Qwen3’s checkpoint, so
load_model from Section 1.6
loads it unchanged. load_model builds the model inside F.lazy(), so the
Llama layers that the subclasses replace are never allocated.
model = load_model(qwen, load_checkpoint(QWEN3), DEVICE, Qwen3ForCausalLM)
compiled = model.compile(
TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)
n_params = sum(p.num_elements() for _, p in model.parameters)
print(
f"{n_params / 1e6:.0f}M values, including the untied copy of the embedding"
)
752M values, including the untied copy of the embedding
To see what QK-norm guarantees on real activations, run layer 0’s input norm and
query projection on a sentence. Then scale the queries up 10 times, as
projection weights that grew during training would. The figure compares the
median length of the query vectors in each head, before and after q_norm:
attn = model.model.layers[0].self_attn
ids = AutoTokenizer(QWEN3)(
["Normalizing queries and keys keeps attention scores in a stable range."]
)["input_ids"][0]
h = model.model.layers[0].input_layernorm(
model.model.embed_tokens(Tensor(np.array([ids]), device=DEVICE))
)
q = attn.q_proj(h).reshape([1, len(ids), attn.num_heads, attn.head_dim])
styles = {}
for scale, dash in [(1, "solid"), (10, "dash")]:
scaled = q * scale
styles[f"before q_norm, input x{scale}"] = (scaled, viz.ORANGE, dash)
styles[f"after q_norm, input x{scale}"] = (
attn.q_norm(scaled),
viz.BLUE,
dash,
)
fig = go.Figure()
for name, (t, color, dash) in styles.items():
median = np.median(
np.linalg.norm(t.to_numpy()[0], axis=-1), axis=0
) # per head
fig.add_trace(
go.Scatter(
x=np.arange(attn.num_heads),
y=median,
name=name,
mode="lines+markers",
line=dict(
color=color, dash=dash, width=4 if dash == "solid" else 2
),
hovertemplate="head %{x}: %{y:.1f}<extra>" + name + "</extra>",
)
)
fig.update_layout(
title="Median query vector length per head, Qwen3 layer 0",
height=440,
xaxis=dict(title="query head", dtick=1),
yaxis=dict(title="length (log scale)", type="log", dtick=1),
)
viz.show(fig)
Without the norm, query lengths grow with the queries: queries 10 times
larger are 10 times longer, and scores grow 100 times once the keys grow
too. After q_norm, the two blue lines lie on top of each other.
Each query’s length now depends only on its direction and the learned
per-dimension scale, which also gives each head its own typical size, so no
input can make a score blow up.
Verify it
Compare with Hugging Face’s PyTorch Qwen3 on a few prompts, using the metrics from Section 4.2:
tokenizer = AutoTokenizer(QWEN3)
reference = AutoModelForCausalLM.from_pretrained(
QWEN3, torch_dtype=torch.float32
).eval()
def softmax(z: np.ndarray) -> np.ndarray:
e = np.exp(z.astype(np.float64) - z.max(-1, keepdims=True))
return e / e.sum(-1, keepdims=True)
for text in [
"The capital of France is",
"def add(a, b):\n return",
"Le chat est assis sur",
]:
ids = tokenizer([text])["input_ids"][0]
ours = compiled(Tensor(np.array([ids]), device=DEVICE)).to_numpy()[0]
with torch.no_grad():
theirs = reference(torch.tensor([ids])).logits[0].numpy()
r = compare_outputs(ours, theirs, atol=1e-3, rtol=1e-3)
p, q = softmax(theirs), softmax(ours)
kl = float((p * (np.log(p + 1e-30) - np.log(q + 1e-30))).sum(-1).mean())
top1 = (ours.argmax(-1) == theirs.argmax(-1)).mean()
print(
f"{text[:24]!r:28} max err {r.max_abs_err:.1e} KL {kl:.1e} top-1 {top1:.0%}"
)
assert top1 >= 0.99 and kl < 1e-6
'The capital of France is' max err 1.4e-04 KL 1.0e-10 top-1 100%
'def add(a, b):\n retur' max err 1.1e-04 KL 1.9e-11 top-1 100%
'Le chat est assis sur' max err 8.5e-05 KL 7.8e-11 top-1 100%
def next_logits(ids: list[int]) -> Tensor:
return compiled(Tensor(np.array([ids]), device=DEVICE))[0, -1]
prompt = tokenizer(["The three primary colors are"])["input_ids"][0]
print(
tokenizer.decode(prompt + generate(next_logits, prompt, 24, temperature=0))
)
The three primary colors are red, blue, and yellow. The three secondary colors are red and blue, blue and yellow, and red and yellow
That is the whole bring-up: about 40 lines of model code for a new architecture, checked against the reference to rounding error. MAX’s own Qwen3 implementation serves the same model with a paged KV cache and continuous batching, the techniques that Chapter 5 builds:
max serve --model Qwen/Qwen3-0.6B --quantization-encoding float32 --no-prefer-module-v3
--no-prefer-module-v3 selects MAX’s text-generation implementation of
Qwen3. Without it, current nightly builds pick Qwen3’s embedding variant for
this checkpoint and fail to start.
Qwen3 is also a reasoning model: by default it thinks step by step inside
<think> tags before it answers, which costs tokens and improves answers to
hard questions. Its chat template has an enable_thinking switch that turns
this off. The handbook’s
choosing the right model
page discusses that trade-off.
Exercises
- Add a KV cache to Qwen3: subclass
CachedAttentionfrom Section 1.8 with the sameq_normandk_norm, and check it against the full model. - Remove QK-norm: copy
load_modelfrom Section 1.6, buildLlamaForCausalLMinstead ofQwen3ForCausalLM, and passstrict=Falsetoload_state_dictso that it skips theq_normandk_normweights. Rerun the comparison. What happens to the output? - Count the parameters in the embedding, the attention projections, and the MLPs. How do the proportions compare with SmolLM2’s?
Next
Section 6.2 adds sliding-window attention with Gemma 3, and Section 6.3 builds a mixture of experts.