6. Bigger models
Your SmolLM2 is a plain Llama. Newer models change a few parts, each for a reason. This chapter extends your model to them, and everything still runs on a laptop.
| Section | You learn |
|---|---|
| 6.1 Qwen3: normalized queries and keys | Bringing up Qwen3-0.6B by subclassing your Llama, and what QK-norm does to query lengths. |
| 6.2 Gemma 3: sliding-window attention | Bringing up Gemma 3 270M: sliding-window attention, GeGLU, and extra normalization. |
| 6.3 Mixture of experts | A Qwen3-MoE layer checked against Hugging Face, routing, and expert parallelism on a simulated mesh. |
Related handbook pages: choosing a model, quantization, and parallelism strategies.