6.2 Gemma 3: sliding-window attention

Open in Colab Open in Kaggle View on GitHub

Attention’s cost grows with the square of the context, and the KV cache grows linearly with it. Gemma 3 reduces both with sliding-window attention: most of its layers look only at the most recent 512 tokens, and every sixth layer looks at all of them. This section brings up Gemma 3 270M, which differs from Llama in more places than Qwen3 did in Section 6.1, and verifies it on a prompt long enough for the window to matter.

Gemma’s weights are gated: accept the license on the model page and make a Hugging Face token available (on Colab, as the HF_TOKEN secret).

import logging
import math
import warnings
from dataclasses import dataclass

import numpy as np
import plotly.graph_objects as go
import torch
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.pipelines import AutoTokenizer
from max.tensor import Tensor
from max.tensor.tensor import TensorType
from plotly.subplots import make_subplots
from transformers import AutoModelForCausalLM

from minillm import viz
from minillm.device import pick_device
from minillm.model import load_model
from minillm.testing import compare_outputs
from minillm.weights import load_checkpoint

logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device()  # a GPU if MAX finds one, otherwise the CPU
GEMMA = "google/gemma-3-270m-it"

What changed

Gemma 3’s config.json has fields that Llama’s doesn’t. A config subclass adds them, and from_pretrained reads every dataclass field it finds:

from __future__ import annotations

import math
from dataclasses import dataclass, replace

from max import functional as F
from max.driver import Device
from max.dtype import DType
from max.graph import Dim
from max.nn import Embedding, GemmaRMSNorm, Linear, Module, ModuleList
from max.nn.kernels.attention import flash_attention
from max.tensor import Tensor
from max.tensor.sharding import DeviceMesh

from minillm.attention import Attention, causal_mask
from minillm.config import LlamaConfig
from minillm.layers import MLP, apply_rope, rope_cos_sin


@dataclass(frozen=True)
class Gemma3Config(LlamaConfig):
    sliding_window: int = 512
    layer_types: tuple[str, ...] = ()
    rope_local_base_freq: float = 10000.0
    query_pre_attn_scalar: float = 256.0

    def __post_init__(self) -> None:
        super().__post_init__()
        object.__setattr__(self, "layer_types", tuple(self.layer_types))

Saved to minillm/gemma3.py.

config = Gemma3Config.from_pretrained(GEMMA)
print(config)
print(
    "global layers:",
    [i for i, t in enumerate(config.layer_types) if t == "full_attention"],
)
Gemma3Config(vocab_size=262144, hidden_size=640, intermediate_size=2048, num_hidden_layers=18, num_attention_heads=4, num_key_value_heads=1, rope_theta=1000000.0, rms_norm_eps=1e-06, max_position_embeddings=32768, tie_word_embeddings=True, eos_token_id=1, head_dim=256, sliding_window=512, layer_types=('sliding_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'full_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'full_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'full_attention'), rope_local_base_freq=10000.0, query_pre_attn_scalar=256)
global layers: [5, 11, 17]

Compared with your Llama, Gemma 3 changes six things:

PartLlamaGemma 3
Attention spanevery layer sees all past tokens5 of 6 layers see only the last 512
RoPE baseone, 100,000 for SmolLM210,000 for windowed layers, 1,000,000 for global ones
Norms per layer2, before attention and before the MLP4, before and after each
RMSNormx / rms(x) * wx / rms(x) * (1 + w) (max.nn.GemmaRMSNorm)
MLP activationSiLU (SwiGLU)tanh-approximated GELU (GeGLU)
Embeddingas storedmultiplied by \(\sqrt{\text{hidden size}}\)

It also has QK-norm, as in Qwen3, and scales attention scores by one over the square root of query_pre_attn_scalar instead of the head dimension (both are 256 in this model). Its vocabulary has 262,144 tokens, so the embedding holds 168M of the model’s 270M parameters.

Sliding-window attention

A windowed layer adds one more condition to the causal mask: query i may see key j only if \(i - j < 512\). F.band_part builds that band. Each layer reads its own RoPE base and window from the config.

The mask uses a large finite number instead of \(-\infty\), for the reason Section 5.1 gave: a query more than 512 tokens in has its whole first block of keys hidden, and a blockwise attention kernel computes \(e^{-\infty + \infty}\) = NaN for such a row.

def sliding_window_mask(
    q_len: int | Dim,
    kv_len: int | Dim,
    window: int,
    device: Device | DeviceMesh,
) -> Tensor:
    """Like ``causal_mask``, but each query also can't see keys ``window`` or
    more positions behind it."""
    # A finite value, not -inf: a window hides a query's whole first block of
    # keys, and a blockwise attention kernel would turn that into NaN.
    blocked = F.broadcast_to(
        Tensor(-1e30, dtype=DType.float32, device=device), [q_len, kv_len]
    )
    return F.band_part(
        blocked, num_lower=window - 1, num_upper=kv_len - q_len, exclude=True
    )


class Gemma3Attention(Attention):
    def __init__(self, config: Gemma3Config, layer: int) -> None:
        super().__init__(config)
        self.sliding = config.layer_types[layer] == "sliding_attention"
        self.window = config.sliding_window
        self.rope_theta = (
            config.rope_local_base_freq if self.sliding else config.rope_theta
        )
        self.scale = config.query_pre_attn_scalar**-0.5
        self.q_norm = GemmaRMSNorm(self.head_dim, eps=config.rms_norm_eps)
        self.k_norm = GemmaRMSNorm(self.head_dim, eps=config.rms_norm_eps)

    def forward(self, x: Tensor) -> Tensor:
        batch, seq, _ = x.shape
        q = self.q_norm(
            self.q_proj(x).reshape([batch, seq, self.num_heads, self.head_dim])
        )
        k = self.k_norm(
            self.k_proj(x).reshape(
                [batch, seq, self.num_kv_heads, self.head_dim]
            )
        )
        v = self.v_proj(x).reshape(
            [batch, seq, self.num_kv_heads, self.head_dim]
        )
        cos, sin = rope_cos_sin(self.head_dim, self.rope_theta, 0, seq, x.mesh)
        q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
        if self.sliding:
            mask = sliding_window_mask(seq, seq, self.window, x.mesh)
        else:
            mask = causal_mask(seq, seq, x.mesh)
        mask = F.broadcast_to(mask, [batch, seq, seq])
        out = flash_attention(q, k, v, self.scale, mask).rebind_mapping(
            q.mapping
        )
        return self.o_proj(
            out.reshape([batch, seq, self.num_heads * self.head_dim])
        )

Saved to minillm/gemma3.py.

Compare the two masks on 16 tokens, with a window small enough to see. The slider changes the window; Gemma 3’s is 512:

seq = 16
full = (causal_mask(seq, seq, DEVICE).to_numpy() == 0).astype(int)
steps = []
for window in [2, 4, 8, 16]:
    windowed = (
        sliding_window_mask(seq, seq, window, DEVICE).to_numpy() == 0
    ).astype(int)
    steps.append(
        [
            go.Heatmap(
                z=m,
                zmin=0,
                zmax=1,
                showscale=False,
                xgap=2,
                ygap=2,
                xaxis=f"x{a}",
                yaxis=f"y{a}",
                colorscale=[
                    [0, viz.GRID],
                    [0.5, viz.GRID],
                    [0.5, viz.BLUE],
                    [1, viz.BLUE],
                ],
                hovertemplate="query %{y}, key %{x}: "
                + "%{z}"
                + "<extra>"
                + title
                + "</extra>",
            )
            for m, a, title in [
                (full, "", "global layer"),
                (windowed, "2", f"window {window}"),
            ]
        ]
    )
fig = make_subplots(
    rows=1,
    cols=2,
    horizontal_spacing=0.08,
    subplot_titles=[
        "global layer: causal",
        "sliding layer: causal and windowed",
    ],
)
fig.update_xaxes(title_text="key position", showgrid=False)
fig.update_yaxes(
    title_text="query position", showgrid=False, autorange="reversed"
)
fig.update_layout(title="Which keys each query sees (1 = visible)", height=440)
viz.show(
    viz.with_slider(fig, steps, ["2", "4", "8", "16"], "window ", active=1)
)

Once the sequence is longer than the window, a sliding layer’s cost per query stops growing, and so does the KV cache it needs: only the last window keys. With five sliding layers for every global one, most of Gemma 3’s KV cache stops growing at 512 tokens.

The rest of the model

GeGLU is the SwiGLU MLP with a different activation. The decoder layer has the four norms, and the model scales its embedding:

class GeGLU(MLP):
    def forward(self, x: Tensor) -> Tensor:
        return self.down_proj(
            F.gelu(self.gate_proj(x), approximate="tanh") * self.up_proj(x)
        )


class Gemma3DecoderLayer(Module):
    def __init__(self, config: Gemma3Config, layer: int) -> None:
        eps = config.rms_norm_eps
        self.input_layernorm = GemmaRMSNorm(config.hidden_size, eps=eps)
        self.self_attn = Gemma3Attention(config, layer)
        self.post_attention_layernorm = GemmaRMSNorm(
            config.hidden_size, eps=eps
        )
        self.pre_feedforward_layernorm = GemmaRMSNorm(
            config.hidden_size, eps=eps
        )
        self.mlp = GeGLU(config)
        self.post_feedforward_layernorm = GemmaRMSNorm(
            config.hidden_size, eps=eps
        )

    def forward(self, x: Tensor) -> Tensor:
        x = x + self.post_attention_layernorm(
            self.self_attn(self.input_layernorm(x))
        )
        return x + self.post_feedforward_layernorm(
            self.mlp(self.pre_feedforward_layernorm(x))
        )


class Gemma3Model(Module):
    def __init__(self, config: Gemma3Config) -> None:
        self.embed_scale = math.sqrt(config.hidden_size)
        self.embed_tokens = Embedding(config.vocab_size, dim=config.hidden_size)
        self.layers = ModuleList(
            [
                Gemma3DecoderLayer(config, i)
                for i in range(config.num_hidden_layers)
            ]
        )
        self.norm = GemmaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)

    def forward(self, input_ids: Tensor) -> Tensor:
        x = self.embed_tokens(input_ids) * self.embed_scale
        for layer in self.layers:
            x = layer(x)
        return self.norm(x)


class Gemma3ForCausalLM(Module):
    def __init__(self, config: Gemma3Config) -> None:
        self.model = Gemma3Model(config)
        self.lm_head = Linear(config.hidden_size, config.vocab_size, bias=False)

    def forward(self, input_ids: Tensor) -> Tensor:
        return self.lm_head(self.model(input_ids))


def without_sliding_window(config: Gemma3Config) -> Gemma3Config:
    """Returns ``config`` with windows too wide to hide any token; each layer
    keeps its own RoPE base."""
    return replace(config, sliding_window=config.max_position_embeddings)

Saved to minillm/gemma3.py.

weights = load_checkpoint(GEMMA)
model = load_model(config, weights, DEVICE, Gemma3ForCausalLM)
compiled = model.compile(
    TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)

Verify it

A short prompt never reaches the window, so verify on two prompts: a short one, and one of about 700 tokens, where windowed layers must ignore the oldest tokens.

tokenizer = AutoTokenizer(GEMMA)
reference = AutoModelForCausalLM.from_pretrained(
    GEMMA, torch_dtype=torch.float32, attn_implementation="eager"
).eval()
prompts = {
    "short": "The capital of France is",
    "long (700 tokens)": "The quick brown fox jumps over the lazy dog. " * 70
    + "The capital of France is",
}


def softmax(z: np.ndarray) -> np.ndarray:
    e = np.exp(z.astype(np.float64) - z.max(-1, keepdims=True))
    return e / e.sum(-1, keepdims=True)


def verify(run) -> dict[str, tuple[float, float]]:  # noqa: ANN001
    # Prints and returns each prompt's KL divergence and top-1 agreement.
    results = {}
    for name, text in prompts.items():
        ids = tokenizer([text])["input_ids"][0]
        ours = run(Tensor(np.array([ids]), device=DEVICE)).to_numpy()[0]
        with torch.no_grad():
            theirs = reference(torch.tensor([ids])).logits[0].numpy()
        r = compare_outputs(ours, theirs)
        p, q = softmax(theirs), softmax(ours)
        kl = float((p * (np.log(p + 1e-30) - np.log(q + 1e-30))).sum(-1).mean())
        top1 = (ours.argmax(-1) == theirs.argmax(-1)).mean()
        print(
            f"{name:18} {len(ids):4} tokens  max err {r.max_abs_err:.1e}  KL {kl:.1e}  top-1 {top1:.0%}"
        )
        results[name] = (kl, top1)
    return results


matched = verify(compiled)
assert all(kl < 1e-6 and top1 >= 0.99 for kl, top1 in matched.values())
short                 6 tokens  max err 1.6e-04  KL 4.8e-11  top-1 100%
long (700 tokens)   706 tokens  max err 4.1e-04  KL 6.5e-12  top-1 100%

Both match. To see that the window does real work, build the same model with windows too wide to hide anything, keep each layer’s RoPE base, and verify again:

no_window = load_model(
    without_sliding_window(config), weights, DEVICE, Gemma3ForCausalLM
)
unwindowed = verify(
    no_window.compile(
        TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
    )
)
assert unwindowed["short"][0] < 1e-6  # no query is far enough back to notice
assert (
    unwindowed["long (700 tokens)"][0] > 100 * matched["long (700 tokens)"][0]
)
short                 6 tokens  max err 1.6e-04  KL 4.8e-11  top-1 100%
long (700 tokens)   706 tokens  max err 4.9e+01  KL 2.8e-01  top-1 93%

On the short prompt the two models are identical, because no query is more than 512 tokens from any key. On the long prompt the model without the window diverges from the reference. The windowed layers do ignore the oldest tokens, and the bring-up must get the window right to match.

A windowed layer’s attention cost and KV cache stop growing at 512 tokens. With 15 of Gemma 3’s 18 layers windowed, a 32,000-token context needs about a fifth of the cache that a fully global model would need. The handbook’s GPU memory calculation page shows how cache size limits context and batch size.

Exercises

  1. Compute the KV cache size of this model for 8K and 32K tokens of context, with and without sliding windows.
  2. Plot attention weights, as in Section 1.5, for a windowed and a global layer on the long prompt. What does the window look like?
  3. Check the embedding scale: remove embed_scale and verify again. Which metric shows it?

Next

Section 6.3 builds a mixture-of-experts layer and spreads its experts across devices.