6.1 Qwen3: normalized queries and keys

Open in Colab Open in Kaggle View on GitHub

Most open models are close relatives of the Llama architecture you built in Chapter 1. To bring up a new one, you find what changed and why. This section adds Qwen3, here the 0.6-billion-parameter model Qwen/Qwen3-0.6B. It subclasses your Llama classes, overrides only what differs, and verifies the result with the tools from Chapter 4.

import logging
import warnings

import numpy as np
import plotly.graph_objects as go
import torch
from max.driver import CPU
from max.dtype import DType
from max.pipelines import AutoTokenizer
from max.tensor import Tensor
from max.tensor.tensor import TensorType
from transformers import AutoModelForCausalLM

from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.model import load_model
from minillm.sampling import generate
from minillm.testing import compare_outputs
from minillm.weights import load_checkpoint

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU
QWEN3 = "Qwen/Qwen3-0.6B"

What changed

Compare the two configs:

smol, qwen = (
    LlamaConfig.from_pretrained(MODEL_ID),
    LlamaConfig.from_pretrained(QWEN3),
)
for field in [
    "vocab_size",
    "hidden_size",
    "num_hidden_layers",
    "num_attention_heads",
    "num_key_value_heads",
    "head_dim",
    "intermediate_size",
    "rope_theta",
    "rms_norm_eps",
]:
    print(f"{field:22} {getattr(smol, field):>10} {getattr(qwen, field):>10}")
vocab_size                  49152     151936
hidden_size                   576       1024
num_hidden_layers              30         28
num_attention_heads             9         16
num_key_value_heads             3          8
head_dim                       64        128
intermediate_size            1536       3072
rope_theta                 100000    1000000
rms_norm_eps                1e-05      1e-06

Four differences matter:

  • A larger vocabulary, 151,936 tokens, which covers many languages and code. The embedding alone holds 156M parameters, about a quarter of the model.
  • An explicit head_dim: 16 heads of 128 dimensions make 2048, wider than the 1024-dimension hidden state. q_proj projects up, and o_proj projects back down. The config class from Section 1.3 reads head_dim when a model sets it.
  • A larger RoPE base, 1,000,000, so the rotations turn more slowly and positions stay distinguishable over longer contexts.
  • QK-norm: an RMSNorm on every query and key head, before RoPE.

The first three are config values that your code already handles. QK-norm needs new code.

QK-norm

Attention scores are dot products of queries and keys. During training, nothing stops some heads’ queries and keys from growing until a few scores dominate every softmax, which can destabilize the training of large models. Normalizing the queries and keys, with a learned scale per dimension, bounds the scores. Qwen3 and Gemma 3 normalize each head separately; OLMo 2 normalizes the whole query and key projections.

For each head’s query \(q\) (and likewise each key), with \(d\) = head_dim and a learned weight \(w\) of size \(d\):

\[\hat{q} = \frac{q}{\sqrt{\tfrac{1}{d}\sum_j q_j^2 + \epsilon}} \odot w\]

The division gives every query the same root mean square, so the length of \(\hat q\) depends on \(w\) and on \(q\)’s direction, not on \(q\)’s size. A score \(\hat q \cdot \hat k\) is then bounded however large \(q\) and \(k\) get. In code, QK-norm is two RMSNorms of size head_dim inside attention, applied to each head after the projection and before RoPE. The other classes only swap in the new attention:

from __future__ import annotations

from max.graph import Dim
from max.nn import ModuleList, RMSNorm
from max.tensor import Tensor

from minillm.attention import Attention
from minillm.config import LlamaConfig
from minillm.layers import apply_rope, rope_cos_sin
from minillm.model import DecoderLayer, LlamaForCausalLM, LlamaModel


class Qwen3Attention(Attention):
    """Llama attention with an RMSNorm on each query and key head."""

    def __init__(self, config: LlamaConfig) -> None:
        super().__init__(config)
        self.q_norm = RMSNorm(self.head_dim, eps=config.rms_norm_eps)
        self.k_norm = RMSNorm(self.head_dim, eps=config.rms_norm_eps)

    def project(
        self, x: Tensor, start: int | Dim | Tensor
    ) -> tuple[Tensor, Tensor, Tensor]:
        batch, seq, _ = x.shape
        q = self.q_proj(x).reshape([batch, seq, self.num_heads, self.head_dim])
        k = self.k_proj(x).reshape(
            [batch, seq, self.num_kv_heads, self.head_dim]
        )
        v = self.v_proj(x).reshape(
            [batch, seq, self.num_kv_heads, self.head_dim]
        )
        q, k = self.q_norm(q), self.k_norm(k)
        cos, sin = rope_cos_sin(
            self.head_dim, self.rope_theta, start, seq, x.mesh
        )
        return apply_rope(q, cos, sin), apply_rope(k, cos, sin), v


class Qwen3DecoderLayer(DecoderLayer):
    def __init__(self, config: LlamaConfig) -> None:
        super().__init__(config)
        self.self_attn = Qwen3Attention(config)


class Qwen3Model(LlamaModel):
    def __init__(self, config: LlamaConfig) -> None:
        super().__init__(config)
        self.layers = ModuleList(
            [Qwen3DecoderLayer(config) for _ in range(config.num_hidden_layers)]
        )


class Qwen3ForCausalLM(LlamaForCausalLM):
    """Qwen3: a Llama model whose attention normalizes queries and keys."""

    def __init__(self, config: LlamaConfig) -> None:
        super().__init__(config)
        self.model = Qwen3Model(config)

Saved to minillm/qwen3.py.

The new parameters are named q_norm and k_norm, as in Qwen3’s checkpoint, so load_model from Section 1.6 loads it unchanged. load_model builds the model inside F.lazy(), so the Llama layers that the subclasses replace are never allocated.

model = load_model(qwen, load_checkpoint(QWEN3), DEVICE, Qwen3ForCausalLM)
compiled = model.compile(
    TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)
n_params = sum(p.num_elements() for _, p in model.parameters)
print(
    f"{n_params / 1e6:.0f}M values, including the untied copy of the embedding"
)
752M values, including the untied copy of the embedding

To see what QK-norm guarantees on real activations, run layer 0’s input norm and query projection on a sentence. Then scale the queries up 10 times, as projection weights that grew during training would. The figure compares the median length of the query vectors in each head, before and after q_norm:

attn = model.model.layers[0].self_attn
ids = AutoTokenizer(QWEN3)(
    ["Normalizing queries and keys keeps attention scores in a stable range."]
)["input_ids"][0]
h = model.model.layers[0].input_layernorm(
    model.model.embed_tokens(Tensor(np.array([ids]), device=DEVICE))
)
q = attn.q_proj(h).reshape([1, len(ids), attn.num_heads, attn.head_dim])
styles = {}
for scale, dash in [(1, "solid"), (10, "dash")]:
    scaled = q * scale
    styles[f"before q_norm, input x{scale}"] = (scaled, viz.ORANGE, dash)
    styles[f"after q_norm, input x{scale}"] = (
        attn.q_norm(scaled),
        viz.BLUE,
        dash,
    )

fig = go.Figure()
for name, (t, color, dash) in styles.items():
    median = np.median(
        np.linalg.norm(t.to_numpy()[0], axis=-1), axis=0
    )  # per head
    fig.add_trace(
        go.Scatter(
            x=np.arange(attn.num_heads),
            y=median,
            name=name,
            mode="lines+markers",
            line=dict(
                color=color, dash=dash, width=4 if dash == "solid" else 2
            ),
            hovertemplate="head %{x}: %{y:.1f}<extra>" + name + "</extra>",
        )
    )
fig.update_layout(
    title="Median query vector length per head, Qwen3 layer 0",
    height=440,
    xaxis=dict(title="query head", dtick=1),
    yaxis=dict(title="length (log scale)", type="log", dtick=1),
)
viz.show(fig)

Without the norm, query lengths grow with the queries: queries 10 times larger are 10 times longer, and scores grow 100 times once the keys grow too. After q_norm, the two blue lines lie on top of each other. Each query’s length now depends only on its direction and the learned per-dimension scale, which also gives each head its own typical size, so no input can make a score blow up.

Verify it

Compare with Hugging Face’s PyTorch Qwen3 on a few prompts, using the metrics from Section 4.2:

tokenizer = AutoTokenizer(QWEN3)
reference = AutoModelForCausalLM.from_pretrained(
    QWEN3, torch_dtype=torch.float32
).eval()


def softmax(z: np.ndarray) -> np.ndarray:
    e = np.exp(z.astype(np.float64) - z.max(-1, keepdims=True))
    return e / e.sum(-1, keepdims=True)


for text in [
    "The capital of France is",
    "def add(a, b):\n    return",
    "Le chat est assis sur",
]:
    ids = tokenizer([text])["input_ids"][0]
    ours = compiled(Tensor(np.array([ids]), device=DEVICE)).to_numpy()[0]
    with torch.no_grad():
        theirs = reference(torch.tensor([ids])).logits[0].numpy()
    r = compare_outputs(ours, theirs, atol=1e-3, rtol=1e-3)
    p, q = softmax(theirs), softmax(ours)
    kl = float((p * (np.log(p + 1e-30) - np.log(q + 1e-30))).sum(-1).mean())
    top1 = (ours.argmax(-1) == theirs.argmax(-1)).mean()
    print(
        f"{text[:24]!r:28} max err {r.max_abs_err:.1e}  KL {kl:.1e}  top-1 {top1:.0%}"
    )
    assert top1 >= 0.99 and kl < 1e-6
'The capital of France is'   max err 1.4e-04  KL 1.0e-10  top-1 100%
'def add(a, b):\n    retur'  max err 1.1e-04  KL 1.9e-11  top-1 100%
'Le chat est assis sur'      max err 8.5e-05  KL 7.8e-11  top-1 100%
def next_logits(ids: list[int]) -> Tensor:
    return compiled(Tensor(np.array([ids]), device=DEVICE))[0, -1]


prompt = tokenizer(["The three primary colors are"])["input_ids"][0]
print(
    tokenizer.decode(prompt + generate(next_logits, prompt, 24, temperature=0))
)
The three primary colors are red, blue, and yellow. The three secondary colors are red and blue, blue and yellow, and red and yellow

That is the whole bring-up: about 40 lines of model code for a new architecture, checked against the reference to rounding error. MAX’s own Qwen3 implementation serves the same model with a paged KV cache and continuous batching, the techniques that Chapter 5 builds:

max serve --model Qwen/Qwen3-0.6B --quantization-encoding float32 --no-prefer-module-v3

--no-prefer-module-v3 selects MAX’s text-generation implementation of Qwen3. Without it, current nightly builds pick Qwen3’s embedding variant for this checkpoint and fail to start.

Qwen3 is also a reasoning model: by default it thinks step by step inside <think> tags before it answers, which costs tokens and improves answers to hard questions. Its chat template has an enable_thinking switch that turns this off. The handbook’s choosing the right model page discusses that trade-off.

Exercises

  1. Add a KV cache to Qwen3: subclass CachedAttention from Section 1.8 with the same q_norm and k_norm, and check it against the full model.
  2. Remove QK-norm: copy load_model from Section 1.6, build LlamaForCausalLM instead of Qwen3ForCausalLM, and pass strict=False to load_state_dict so that it skips the q_norm and k_norm weights. Rerun the comparison. What happens to the output?
  3. Count the parameters in the embedding, the attention projections, and the MLPs. How do the proportions compare with SmolLM2’s?

Next

Section 6.2 adds sliding-window attention with Gemma 3, and Section 6.3 builds a mixture of experts.