4. Testing and benchmarking
A model has to be correct before it is worth making fast. This chapter applies the tools MAX uses to test and measure its own models to yours.
| Section | You learn |
|---|---|
| 4.1 Testing layers against references | Testing a decoder layer on many shapes against Hugging Face’s PyTorch Llama, with the pattern MAX’s layer benchmarks use. |
| 4.2 Verifying a model’s logits | Comparing a whole model’s logits with a reference, using the metrics MAX’s verification tools use. |
| 4.3 Benchmarking kernels and layers | A benchmark harness, and why decode is limited by memory and prefill by arithmetic. |
| 4.4 Profiling | Where the time goes, from a whole generation run down to the kernels and the parts of your model. |
Related handbook pages: LLM inference metrics and performance benchmarks.