6. Bigger models

Your SmolLM2 is a plain Llama. Newer models change a few parts, each for a reason. This chapter extends your model to them, and everything still runs on a laptop.

SectionYou learn
6.1 Qwen3: normalized queries and keysBringing up Qwen3-0.6B by subclassing your Llama, and what QK-norm does to query lengths.
6.2 Gemma 3: sliding-window attentionBringing up Gemma 3 270M: sliding-window attention, GeGLU, and extra normalization.
6.3 Mixture of expertsA Qwen3-MoE layer checked against Hugging Face, routing, and expert parallelism on a simulated mesh.

Related handbook pages: choosing a model, quantization, and parallelism strategies.