6.2 Gemma 3: sliding-window attention
Attention’s cost grows with the square of the context, and the KV cache grows linearly with it. Gemma 3 reduces both with sliding-window attention: most of its layers look only at the most recent 512 tokens, and every sixth layer looks at all of them. This section brings up Gemma 3 270M, which differs from Llama in more places than Qwen3 did in Section 6.1, and verifies it on a prompt long enough for the window to matter.
Gemma’s weights are gated: accept the license on the
model page and make a
Hugging Face token available (on Colab, as the HF_TOKEN secret).
import logging
import math
import warnings
from dataclasses import dataclass
import numpy as np
import plotly.graph_objects as go
import torch
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.pipelines import AutoTokenizer
from max.tensor import Tensor
from max.tensor.tensor import TensorType
from plotly.subplots import make_subplots
from transformers import AutoModelForCausalLM
from minillm import viz
from minillm.device import pick_device
from minillm.model import load_model
from minillm.testing import compare_outputs
from minillm.weights import load_checkpoint
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
GEMMA = "google/gemma-3-270m-it"
What changed
Gemma 3’s config.json has fields that Llama’s doesn’t. A config subclass
adds them, and from_pretrained reads every dataclass field it finds:
from __future__ import annotations
import math
from dataclasses import dataclass, replace
from max import functional as F
from max.driver import Device
from max.dtype import DType
from max.graph import Dim
from max.nn import Embedding, GemmaRMSNorm, Linear, Module, ModuleList
from max.nn.kernels.attention import flash_attention
from max.tensor import Tensor
from max.tensor.sharding import DeviceMesh
from minillm.attention import Attention, causal_mask
from minillm.config import LlamaConfig
from minillm.layers import MLP, apply_rope, rope_cos_sin
@dataclass(frozen=True)
class Gemma3Config(LlamaConfig):
sliding_window: int = 512
layer_types: tuple[str, ...] = ()
rope_local_base_freq: float = 10000.0
query_pre_attn_scalar: float = 256.0
def __post_init__(self) -> None:
super().__post_init__()
object.__setattr__(self, "layer_types", tuple(self.layer_types))
Saved to minillm/gemma3.py.
config = Gemma3Config.from_pretrained(GEMMA)
print(config)
print(
"global layers:",
[i for i, t in enumerate(config.layer_types) if t == "full_attention"],
)
Gemma3Config(vocab_size=262144, hidden_size=640, intermediate_size=2048, num_hidden_layers=18, num_attention_heads=4, num_key_value_heads=1, rope_theta=1000000.0, rms_norm_eps=1e-06, max_position_embeddings=32768, tie_word_embeddings=True, eos_token_id=1, head_dim=256, sliding_window=512, layer_types=('sliding_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'full_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'full_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'sliding_attention', 'full_attention'), rope_local_base_freq=10000.0, query_pre_attn_scalar=256)
global layers: [5, 11, 17]
Compared with your Llama, Gemma 3 changes six things:
| Part | Llama | Gemma 3 |
|---|---|---|
| Attention span | every layer sees all past tokens | 5 of 6 layers see only the last 512 |
| RoPE base | one, 100,000 for SmolLM2 | 10,000 for windowed layers, 1,000,000 for global ones |
| Norms per layer | 2, before attention and before the MLP | 4, before and after each |
| RMSNorm | x / rms(x) * w | x / rms(x) * (1 + w) (max.nn.GemmaRMSNorm) |
| MLP activation | SiLU (SwiGLU) | tanh-approximated GELU (GeGLU) |
| Embedding | as stored | multiplied by \(\sqrt{\text{hidden size}}\) |
It also has QK-norm, as in Qwen3, and scales attention scores by one over
the square root of query_pre_attn_scalar instead of the head dimension
(both are 256 in this model). Its vocabulary has 262,144 tokens, so the
embedding holds 168M of the model’s 270M parameters.
Sliding-window attention
A windowed layer adds one more condition to the causal mask: query i may
see key j only if \(i - j < 512\). F.band_part builds that band. Each
layer reads its own RoPE base and window from the config.
The mask uses a large finite number instead of \(-\infty\), for the reason Section 5.1 gave: a query more than 512 tokens in has its whole first block of keys hidden, and a blockwise attention kernel computes \(e^{-\infty + \infty}\) = NaN for such a row.
def sliding_window_mask(
q_len: int | Dim,
kv_len: int | Dim,
window: int,
device: Device | DeviceMesh,
) -> Tensor:
"""Like ``causal_mask``, but each query also can't see keys ``window`` or
more positions behind it."""
# A finite value, not -inf: a window hides a query's whole first block of
# keys, and a blockwise attention kernel would turn that into NaN.
blocked = F.broadcast_to(
Tensor(-1e30, dtype=DType.float32, device=device), [q_len, kv_len]
)
return F.band_part(
blocked, num_lower=window - 1, num_upper=kv_len - q_len, exclude=True
)
class Gemma3Attention(Attention):
def __init__(self, config: Gemma3Config, layer: int) -> None:
super().__init__(config)
self.sliding = config.layer_types[layer] == "sliding_attention"
self.window = config.sliding_window
self.rope_theta = (
config.rope_local_base_freq if self.sliding else config.rope_theta
)
self.scale = config.query_pre_attn_scalar**-0.5
self.q_norm = GemmaRMSNorm(self.head_dim, eps=config.rms_norm_eps)
self.k_norm = GemmaRMSNorm(self.head_dim, eps=config.rms_norm_eps)
def forward(self, x: Tensor) -> Tensor:
batch, seq, _ = x.shape
q = self.q_norm(
self.q_proj(x).reshape([batch, seq, self.num_heads, self.head_dim])
)
k = self.k_norm(
self.k_proj(x).reshape(
[batch, seq, self.num_kv_heads, self.head_dim]
)
)
v = self.v_proj(x).reshape(
[batch, seq, self.num_kv_heads, self.head_dim]
)
cos, sin = rope_cos_sin(self.head_dim, self.rope_theta, 0, seq, x.mesh)
q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
if self.sliding:
mask = sliding_window_mask(seq, seq, self.window, x.mesh)
else:
mask = causal_mask(seq, seq, x.mesh)
mask = F.broadcast_to(mask, [batch, seq, seq])
out = flash_attention(q, k, v, self.scale, mask).rebind_mapping(
q.mapping
)
return self.o_proj(
out.reshape([batch, seq, self.num_heads * self.head_dim])
)
Saved to minillm/gemma3.py.
Compare the two masks on 16 tokens, with a window small enough to see. The slider changes the window; Gemma 3’s is 512:
seq = 16
full = (causal_mask(seq, seq, DEVICE).to_numpy() == 0).astype(int)
steps = []
for window in [2, 4, 8, 16]:
windowed = (
sliding_window_mask(seq, seq, window, DEVICE).to_numpy() == 0
).astype(int)
steps.append(
[
go.Heatmap(
z=m,
zmin=0,
zmax=1,
showscale=False,
xgap=2,
ygap=2,
xaxis=f"x{a}",
yaxis=f"y{a}",
colorscale=[
[0, viz.GRID],
[0.5, viz.GRID],
[0.5, viz.BLUE],
[1, viz.BLUE],
],
hovertemplate="query %{y}, key %{x}: "
+ "%{z}"
+ "<extra>"
+ title
+ "</extra>",
)
for m, a, title in [
(full, "", "global layer"),
(windowed, "2", f"window {window}"),
]
]
)
fig = make_subplots(
rows=1,
cols=2,
horizontal_spacing=0.08,
subplot_titles=[
"global layer: causal",
"sliding layer: causal and windowed",
],
)
fig.update_xaxes(title_text="key position", showgrid=False)
fig.update_yaxes(
title_text="query position", showgrid=False, autorange="reversed"
)
fig.update_layout(title="Which keys each query sees (1 = visible)", height=440)
viz.show(
viz.with_slider(fig, steps, ["2", "4", "8", "16"], "window ", active=1)
)
Once the sequence is longer than the window, a sliding layer’s cost per
query stops growing, and so does the KV cache it needs: only the last
window keys. With five sliding layers for every global one, most of
Gemma 3’s KV cache stops growing at 512 tokens.
The rest of the model
GeGLU is the SwiGLU MLP with a different activation. The decoder layer has the four norms, and the model scales its embedding:
class GeGLU(MLP):
def forward(self, x: Tensor) -> Tensor:
return self.down_proj(
F.gelu(self.gate_proj(x), approximate="tanh") * self.up_proj(x)
)
class Gemma3DecoderLayer(Module):
def __init__(self, config: Gemma3Config, layer: int) -> None:
eps = config.rms_norm_eps
self.input_layernorm = GemmaRMSNorm(config.hidden_size, eps=eps)
self.self_attn = Gemma3Attention(config, layer)
self.post_attention_layernorm = GemmaRMSNorm(
config.hidden_size, eps=eps
)
self.pre_feedforward_layernorm = GemmaRMSNorm(
config.hidden_size, eps=eps
)
self.mlp = GeGLU(config)
self.post_feedforward_layernorm = GemmaRMSNorm(
config.hidden_size, eps=eps
)
def forward(self, x: Tensor) -> Tensor:
x = x + self.post_attention_layernorm(
self.self_attn(self.input_layernorm(x))
)
return x + self.post_feedforward_layernorm(
self.mlp(self.pre_feedforward_layernorm(x))
)
class Gemma3Model(Module):
def __init__(self, config: Gemma3Config) -> None:
self.embed_scale = math.sqrt(config.hidden_size)
self.embed_tokens = Embedding(config.vocab_size, dim=config.hidden_size)
self.layers = ModuleList(
[
Gemma3DecoderLayer(config, i)
for i in range(config.num_hidden_layers)
]
)
self.norm = GemmaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
def forward(self, input_ids: Tensor) -> Tensor:
x = self.embed_tokens(input_ids) * self.embed_scale
for layer in self.layers:
x = layer(x)
return self.norm(x)
class Gemma3ForCausalLM(Module):
def __init__(self, config: Gemma3Config) -> None:
self.model = Gemma3Model(config)
self.lm_head = Linear(config.hidden_size, config.vocab_size, bias=False)
def forward(self, input_ids: Tensor) -> Tensor:
return self.lm_head(self.model(input_ids))
def without_sliding_window(config: Gemma3Config) -> Gemma3Config:
"""Returns ``config`` with windows too wide to hide any token; each layer
keeps its own RoPE base."""
return replace(config, sliding_window=config.max_position_embeddings)
Saved to minillm/gemma3.py.
weights = load_checkpoint(GEMMA)
model = load_model(config, weights, DEVICE, Gemma3ForCausalLM)
compiled = model.compile(
TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)
Verify it
A short prompt never reaches the window, so verify on two prompts: a short one, and one of about 700 tokens, where windowed layers must ignore the oldest tokens.
tokenizer = AutoTokenizer(GEMMA)
reference = AutoModelForCausalLM.from_pretrained(
GEMMA, torch_dtype=torch.float32, attn_implementation="eager"
).eval()
prompts = {
"short": "The capital of France is",
"long (700 tokens)": "The quick brown fox jumps over the lazy dog. " * 70
+ "The capital of France is",
}
def softmax(z: np.ndarray) -> np.ndarray:
e = np.exp(z.astype(np.float64) - z.max(-1, keepdims=True))
return e / e.sum(-1, keepdims=True)
def verify(run) -> dict[str, tuple[float, float]]: # noqa: ANN001
# Prints and returns each prompt's KL divergence and top-1 agreement.
results = {}
for name, text in prompts.items():
ids = tokenizer([text])["input_ids"][0]
ours = run(Tensor(np.array([ids]), device=DEVICE)).to_numpy()[0]
with torch.no_grad():
theirs = reference(torch.tensor([ids])).logits[0].numpy()
r = compare_outputs(ours, theirs)
p, q = softmax(theirs), softmax(ours)
kl = float((p * (np.log(p + 1e-30) - np.log(q + 1e-30))).sum(-1).mean())
top1 = (ours.argmax(-1) == theirs.argmax(-1)).mean()
print(
f"{name:18} {len(ids):4} tokens max err {r.max_abs_err:.1e} KL {kl:.1e} top-1 {top1:.0%}"
)
results[name] = (kl, top1)
return results
matched = verify(compiled)
assert all(kl < 1e-6 and top1 >= 0.99 for kl, top1 in matched.values())
short 6 tokens max err 1.6e-04 KL 4.8e-11 top-1 100%
long (700 tokens) 706 tokens max err 4.1e-04 KL 6.5e-12 top-1 100%
Both match. To see that the window does real work, build the same model with windows too wide to hide anything, keep each layer’s RoPE base, and verify again:
no_window = load_model(
without_sliding_window(config), weights, DEVICE, Gemma3ForCausalLM
)
unwindowed = verify(
no_window.compile(
TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
)
)
assert unwindowed["short"][0] < 1e-6 # no query is far enough back to notice
assert (
unwindowed["long (700 tokens)"][0] > 100 * matched["long (700 tokens)"][0]
)
short 6 tokens max err 1.6e-04 KL 4.8e-11 top-1 100%
long (700 tokens) 706 tokens max err 4.9e+01 KL 2.8e-01 top-1 93%
On the short prompt the two models are identical, because no query is more than 512 tokens from any key. On the long prompt the model without the window diverges from the reference. The windowed layers do ignore the oldest tokens, and the bring-up must get the window right to match.
A windowed layer’s attention cost and KV cache stop growing at 512 tokens. With 15 of Gemma 3’s 18 layers windowed, a 32,000-token context needs about a fifth of the cache that a fully global model would need. The handbook’s GPU memory calculation page shows how cache size limits context and batch size.
Exercises
- Compute the KV cache size of this model for 8K and 32K tokens of context, with and without sliding windows.
- Plot attention weights, as in Section 1.5, for a windowed and a global layer on the long prompt. What does the window look like?
- Check the embedding scale: remove
embed_scaleand verify again. Which metric shows it?
Next
Section 6.3 builds a mixture-of-experts layer and spreads its experts across devices.