Build and serve an LLM with MAX
This book takes you from zero to serving your own language model with MAX, Modular’s framework for AI inference. You start by running a real model in three lines of Python. Then you rebuild that model from scratch, check every part against MAX’s own implementation, and go on to the techniques that production inference engines use: distributing a model across devices, writing your own Mojo kernels, testing and benchmarking, building a serving engine, bringing up newer models, and serving your own architecture.
Why this book
Most explanations of LLM inference stop at diagrams, and most inference engines are too large to read. This book sits between the two. You write every piece yourself, small enough to read in one sitting, and you run it on a real model so that each claim is a number you measured: the logits match the reference, the KV cache makes decoding flat, batching raises throughput.
You do it with MAX’s own APIs, the same ones its production models use, so what you learn carries over directly:
max.pipelines:AutoPipeline,AutoModel, andAutoTokenizerfor running models from Python.max.tensorandmax.functional: tensors and operations that run eagerly, like NumPy, and compile into fast graphs.max.nn: modules and layers for building models.- Mojo custom ops: kernels you write yourself and call from Python.
max.graph: the graph API thatcompilebuilds on, for full control.max serve,max generate, andmax benchmark: the command-line tools for serving, testing, and measuring models.
Every section is a notebook with interactive figures you can hover, zoom, and slide through, and every notebook checks its own results.
The model
The book uses SmolLM2-135M-Instruct. It is a real chat model, small enough to run on a laptop CPU, and it uses the Llama architecture that most open models share: RMSNorm, rotary position embeddings, grouped-query attention, and a SwiGLU MLP. What you build for it applies to Llama, Qwen, Mistral, and their relatives, and Chapter 6 extends it to Qwen3, Gemma 3, and mixture-of-experts models.
Get started
Check your hardware first
Before you install anything, check that MAX supports your machine and GPU in the MAX system requirements. MAX runs on Linux and on macOS with Apple silicon. These notebooks need an NVIDIA GPU from the Ampere generation or newer; on an older one, such as a T4 or P100, they run on the CPU instead.
You can run the notebooks in Colab or Kaggle with nothing installed, or on your own machine.
In Colab or Kaggle
Every section page starts with Open in Colab and Open in Kaggle buttons. The first cell of every notebook installs what the notebook needs, so run the cells in order from the top:
# The setup cell, shortened: it installs MAX and this book's package
# with uv, but only if they're missing.
if importlib.util.find_spec("max") is None:
uv_install("max[all]") # from https://whl.modular.com/nightly/simple/
if importlib.util.find_spec("minillm") is None:
uv_install("git+https://github.com/modular/max-llm-book@main")
max[all] is the MAX package with everything the book uses: the Python
APIs, the max and mojo commands, serving, and the Jupyter %%mojo
magic. minillm is the package you build in the book, installed so that
later sections can import what earlier ones wrote.
Choose the runtime before you run the first cell:
- In Colab, the CPU runtime works for every notebook. For a GPU, choose an L4 or A100 runtime. MAX’s matrix multiplication needs an NVIDIA GPU from the Ampere generation or later, so on a T4 runtime the setup cell hides the GPU and the notebook runs on the CPU.
- In Kaggle, turn on Internet in the notebook settings so that the setup cell can download packages. Kaggle’s GPUs, the T4 and the P100, are older than Ampere, so the notebooks run on the CPU there.
On your own machine
Install with pixi or uv, on a machine that meets the system requirements.
With pixi, which installs Python, MAX, and every other dependency into a project environment:
-
Install pixi:
curl -fsSL https://pixi.sh/install.sh | sh -
Clone the book:
git clone https://github.com/modular/max-llm-book cd max-llm-book -
Open the notebooks in JupyterLab:
pixi run notebook
With uv and Python 3.10 or later:
-
Create and activate an environment:
uv venv --python 3.12 source .venv/bin/activate -
Install MAX from the nightly package index:
uv pip install --prerelease allow --index-strategy unsafe-best-match \ --extra-index-url https://whl.modular.com/nightly/simple/ "max[all]" -
Clone the book and install its package, with JupyterLab:
git clone https://github.com/modular/max-llm-book cd max-llm-book uv pip install -e . jupyterlab -
Open the notebooks:
jupyter lab notebooks
Check the installation, and whether MAX found a GPU:
max --version
python -c "from max.driver import accelerator_count; print('GPUs:', accelerator_count())"
The book uses the MAX nightly build, because some of the APIs it teaches are newer than the last stable release. Every notebook runs on a CPU and uses a GPU automatically when MAX finds a supported one.
About the numbers
Timings, speedups, and other measurements on these pages come from one run on an Apple-silicon Mac with a Metal GPU. Treat them as examples. Your numbers depend on your hardware and vary from run to run; the notebooks check results, such as matching logits, that hold on every machine.
Chapters
| Chapter | What you do |
|---|---|
| 1. Foundations | Run and serve SmolLM2, then rebuild it from scratch: tensors, weights, layers, attention, the full model checked against MAX’s, sampling, and a KV cache. |
| 2. Distributed tensors | Shard tensors and the model across a device mesh, simulated on one machine, and check that the result doesn’t change. |
| 3. Custom ops in Mojo | Write a kernel in Mojo, call it from your model, and see the graph API that compile builds on. |
| 4. Testing and benchmarking | Test layers against references, verify logits, and benchmark and profile models and kernels. |
| 5. A serving engine | Build the core of an inference engine: continuous batching, a paged KV cache, prefix caching, and speculative decoding. |
| 6. Bigger models | Extend your model to Qwen3, Gemma 3, and mixture-of-experts. |
| 7. Production serving | Serve your own architecture with max serve, benchmark it, use structured output and tools, and call state-of-the-art open models on Modular’s cloud. |
Read the chapters in order the first time: each section builds on code from
earlier ones, which the minillm package collects for you.
The handbook
The Modular handbook explains how LLM inference works in production: metrics, batching, KV cache management, quantization, and GPU kernels. This book is its hands-on companion. Each section links the handbook pages for the ideas it implements, so you can read the concept and then build it.
Prerequisites
You should be comfortable with Python and NumPy-style array code. Knowing
what a neural network layer is helps; the book explains everything specific
to transformers as it goes. Section 6.2 uses Gemma 3,
whose weights require you to accept Google’s license on Hugging Face and log
in with hf auth login.