2. Distributed tensors
Large models don’t fit on one device, and even small ones serve faster when several devices share the work. In MAX, a tensor that spans several devices behaves like any other tensor: you describe how it is laid out, and MAX runs each operation on every device and inserts the communication it needs.
Distributed code is hard to develop when every test needs a multi-GPU machine. MAX can simulate a mesh of devices on one CPU, with the same code and the same results, so this whole chapter runs on a laptop or in Colab.
| Section | You learn |
|---|---|
| 2.1 Distributed tensors | Device meshes, placements, sharded matmuls, and the collectives they imply. |
| 2.2 A tensor-parallel model | Splitting SmolLM2 across three devices and checking that its logits don’t change. |
The handbook’s parallelism strategies and distributed inference pages explain when each strategy pays off in production.