4. Testing and benchmarking

A model has to be correct before it is worth making fast. This chapter applies the tools MAX uses to test and measure its own models to yours.

SectionYou learn
4.1 Testing layers against referencesTesting a decoder layer on many shapes against Hugging Face’s PyTorch Llama, with the pattern MAX’s layer benchmarks use.
4.2 Verifying a model’s logitsComparing a whole model’s logits with a reference, using the metrics MAX’s verification tools use.
4.3 Benchmarking kernels and layersA benchmark harness, and why decode is limited by memory and prefill by arithmetic.
4.4 ProfilingWhere the time goes, from a whole generation run down to the kernels and the parts of your model.

Related handbook pages: LLM inference metrics and performance benchmarks.