3.1 Your first custom op
MAX’s operations, from F.silu to flash attention, run kernels written in Mojo.
You can write your own the same way, compile it from a notebook cell, and call
it from Python like any other MAX operation. This section writes a fused SwiGLU
kernel, the silu(gate) * up inside SmolLM2’s MLP from
Section 1.4, tests it,
runs it inside your full model, and measures whether it helps.
import logging
import time
import warnings
from pathlib import Path
import numpy as np
import plotly.graph_objects as go
from max import functional as F
from max.driver import CPU
from max.dtype import DType
from max.pipelines import AutoTokenizer
from max.tensor import Tensor, default_device, default_dtype
from max.tensor.tensor import TensorType
from plotly.subplots import make_subplots
from minillm import viz
from minillm.device import pick_device
from minillm.config import MODEL_ID, LlamaConfig
from minillm.layers import MLP
from minillm.model import load_model
from minillm.weights import load_checkpoint
logging.getLogger("max").setLevel(logging.ERROR)
warnings.filterwarnings("ignore", message="The eager interpreter failed")
DEVICE = pick_device() # a GPU if MAX finds one, otherwise the CPU
Write the kernel
import mojo.notebook adds a %%mojo cell magic to Jupyter. A cell that
starts with %%mojo package -o kernels.mojoc compiles its Mojo code into a
package that MAX can load.
import mojo.notebook
The simplest kind of custom op is elementwise: each output element depends
only on the input elements at the same position. You write a struct with one
static elementwise method that maps input vectors to an output vector, and
MAX handles everything else: splitting the work across threads, loading and
storing memory, and choosing CPU or GPU code.
@extensibility.register("swiglu")names the op; Python calls it by this name.ElementwiseBinaryOpdeclares the shape of the method: two inputs, one output, all of one dtype.SIMD[dtype, width]is a vector ofwidthvalues. MAX picks the width that fills the hardware’s vector registers, so one call processes several elements at once.comptime assertruns at compile time and rejects integer dtypes, for whichexpis undefined.
SiLU is \(x \cdot \sigma(x) = x / (1 + e^{-x})\):
%%mojo package -o kernels.mojoc
import extensibility
from extensibility import ElementwiseBinaryOp
from std.math import exp
@extensibility.register("swiglu")
struct SwiGLU(ElementwiseBinaryOp):
@staticmethod
def elementwise[
dtype: DType, width: SIMDLength
](gate: SIMD[dtype, width], up: SIMD[dtype, width]) -> SIMD[dtype, width]:
comptime assert dtype.is_floating_point(), "SwiGLU needs floats"
return gate / (1 + exp(-gate)) * up
Call it from Python
F.custom calls a custom op by name on eager tensors. It needs the device to
run on, the inputs, the type of each output (here, the same as gate’s), and
the package to load the kernel from:
KERNELS = Path("kernels.mojoc").resolve()
def swiglu(gate: Tensor, up: Tensor) -> Tensor:
return F.custom(
"swiglu",
device=gate.device,
values=[gate, up],
out_types=[gate.type],
custom_extensions=KERNELS,
)[0]
rng = np.random.default_rng(0)
g = rng.standard_normal((4, 8)).astype(np.float32)
u = rng.standard_normal((4, 8)).astype(np.float32)
result = swiglu(Tensor(g, device=DEVICE), Tensor(u, device=DEVICE))
print(result.to_numpy()[0, :4])
print(F.silu(Tensor(g, device=DEVICE)).to_numpy()[0, :4] * u[0, :4])
[-0.01063813 -0.03336792 0.09002377 0.01961606]
[-0.01063813 -0.03336792 0.09002377 0.01961606]
Test it
A kernel needs the same tests as any other code, against a reference you
trust. Check shapes that don’t divide evenly into vectors, a single element,
and inputs extreme enough to overflow exp:
def reference(g: np.ndarray, u: np.ndarray) -> np.ndarray:
return g / (1 + np.exp(-g.astype(np.float64))) * u
cases = {
"1 element": (1,),
"odd length": (1, 7, 1001),
"model size": (2, 32, 1536),
}
for name, shape in cases.items():
g = rng.standard_normal(shape).astype(np.float32) * 4
u = rng.standard_normal(shape).astype(np.float32)
out = swiglu(Tensor(g, device=DEVICE), Tensor(u, device=DEVICE)).to_numpy()
print(f"{name:12} max error {np.abs(out - reference(g, u)).max():.1e}")
assert np.allclose(out, reference(g, u), rtol=1e-4, atol=1e-5)
extremes = np.array([-100.0, -20.0, 0.0, 20.0, 100.0], np.float32)
out = swiglu(
Tensor(extremes, device=DEVICE),
Tensor(np.ones(5, np.float32), device=DEVICE),
)
print("extremes:", out.to_numpy())
assert np.isfinite(out.to_numpy()).all()
1 element max error 1.0e-08
odd length max error 1.8e-06
model size max error 3.0e-06
extremes: [-0.0000000e+00 -4.1223068e-08 0.0000000e+00 2.0000000e+01
1.0000000e+02]
Sweep the input to see the whole function and its error. With up set to
ones, the op is SiLU itself:
sweep = np.linspace(-12, 12, 481, dtype=np.float32)
ones = np.ones_like(sweep)
ours = swiglu(
Tensor(sweep, device=DEVICE), Tensor(ones, device=DEVICE)
).to_numpy()
error = np.abs(ours - reference(sweep, ones))
fig = make_subplots(
rows=2,
cols=1,
shared_xaxes=True,
vertical_spacing=0.08,
row_heights=[0.6, 0.4],
)
fig.add_trace(
go.Scatter(
x=sweep,
y=ours,
name="your kernel",
line_color=viz.BLUE,
hovertemplate="silu(%{x:.2f}) = %{y:.4f}<extra></extra>",
),
row=1,
col=1,
)
fig.add_trace(
go.Scatter(
x=sweep,
y=np.maximum(error, 1e-12),
name="error against float64 NumPy",
line_color=viz.ORANGE,
hovertemplate="%{x:.2f}: error %{y:.1e}<extra></extra>",
),
row=2,
col=1,
)
fig.update_yaxes(title_text="output", row=1, col=1)
fig.update_yaxes(title_text="absolute error", type="log", dtick=1, row=2, col=1)
fig.update_xaxes(title_text="gate", row=2, col=1)
fig.update_layout(title="The swiglu op with up = 1", height=520)
viz.show(fig)
The error stays at the scale of float32 rounding, about \(10^{-7}\) of the
output or less, across the whole range.
At \(-100\), \(e^{100}\) overflows float32 to infinity, and \(-100 / \infty\) is
\(-0\), the correct limit. A different but equivalent formula, such as
gate * exp(gate) / (1 + exp(gate)), would compute \(\infty / \infty\) and
return NaN at \(+100\). Tests with extreme inputs catch this kind of bug.
Use it in your model
FusedMLP is the MLP from
Section 1.4 with the
custom op in place of F.silu(...) *:
class FusedMLP(MLP):
def forward(self, x: Tensor) -> Tensor:
return self.down_proj(swiglu(self.gate_proj(x), self.up_proj(x)))
config = LlamaConfig.from_pretrained()
weights = load_checkpoint()
model = load_model(config, weights, DEVICE)
for layer in model.model.layers:
with F.lazy(), default_device(DEVICE), default_dtype(DType.float32):
fused = FusedMLP(config)
fused.load_state_dict(dict(layer.mlp.parameters))
layer.mlp = fused
Compile the model as before, with one addition: custom_extensions tells
the compiler where to find the kernel. Then compare the logits with the
model from Section 1.6:
token_type = TensorType(DType.int64, ("batch", "seq_len"), device=DEVICE)
fused_model = model.compile(token_type, custom_extensions=[KERNELS])
plain_model = load_model(config, weights, DEVICE).compile(token_type)
tokenizer = AutoTokenizer(MODEL_ID)
ids = Tensor(
np.array([tokenizer(["The capital of France is"])["input_ids"][0]]),
device=DEVICE,
)
fused_logits = fused_model(ids).to_numpy()
plain_logits = plain_model(ids).to_numpy()
print(f"max difference {np.abs(fused_logits - plain_logits).max():.1e}")
print(
"next token:", repr(tokenizer.decode([int(fused_logits[0, -1].argmax())]))
)
assert (
np.abs(fused_logits - plain_logits).max() < 1e-2
and fused_logits[0, -1].argmax() == plain_logits[0, -1].argmax()
)
max difference 4.9e-05
next token: ' Paris'
Is it faster?
Time a compiled MLP block with and without the custom op, on a batch of 512 positions:
with default_device(DEVICE), default_dtype(DType.float32):
plain_mlp = MLP(config)
fused_mlp = FusedMLP(config)
fused_mlp.load_state_dict(dict(plain_mlp.parameters))
block_type = TensorType(
DType.float32, ("batch", "seq_len", config.hidden_size), device=DEVICE
)
runs = {
"MAX ops": plain_mlp.compile(block_type),
"custom op": fused_mlp.compile(block_type, custom_extensions=[KERNELS]),
}
x = Tensor(
rng.standard_normal((1, 512, config.hidden_size)).astype(np.float32),
device=DEVICE,
)
times = {}
for name, run in runs.items():
run(x).to_numpy()
times[name] = []
for _ in range(30):
start = time.perf_counter()
run(x).to_numpy()
times[name].append((time.perf_counter() - start) * 1000)
print(f"{name:12} median {np.median(times[name]):.2f} ms")
fig = go.Figure(
[
go.Box(
y=ms,
name=name,
marker_color=color,
boxpoints="all",
jitter=0.4,
pointpos=0,
hovertemplate="%{y:.2f} ms<extra>" + name + "</extra>",
)
for (name, ms), color in zip(times.items(), [viz.BLUE, viz.ORANGE])
]
)
fig.update_layout(
title="A compiled MLP block on 512 positions, 30 runs each",
height=400,
yaxis=dict(title="ms", rangemode="tozero"),
showlegend=False,
)
viz.show(fig)
MAX ops median 2.84 ms
custom op median 2.18 ms
In the runs behind this page, on an Apple-silicon Mac with a Metal GPU, the two
medians land within about 10 to 20% of each other, and which one is faster
changes from run to run. Treat the output above as one example: your hardware
will give different numbers, so measure on the hardware you serve on. There is
little to gain because the graph compiler already fuses elementwise operations
into their neighbors: the kernel trace in
Section 4.4 shows silu and the
multiply applied in the epilogue of a matmul, so the plain version never writes
the activation to memory either. Most of the time goes to the three matrix
multiplications.
Write a custom op when MAX doesn’t have the operation you need, or when a better algorithm needs control that a chain of separate operations can’t express. Flash attention (Section 1.5) is the classic case: it never materializes the attention matrix, which no combination of separate operations can avoid. Elementwise ops written as above stay fusible, so the compiler can still merge them with their neighbors, and the same kernel runs on a GPU with no changes.
Kernels that need control over threads and memory, such as a matmul or
attention, implement an execute method instead, written with Mojo’s GPU
programming model. MAX’s guide to
building custom ops shows
the pattern, and the
Mojo GPU puzzles teach GPU
programming in Mojo from first principles. The handbook’s
kernel optimization tools
page compares Mojo with CUDA and Triton, and
GPU architecture fundamentals
explains what a kernel has to work with.
Exercises
- Write a
gelu_tanhcustom op as anElementwiseUnaryOp(unary elementwise ops take one input) and test it againstF.geluwithapproximate="tanh". - Add a
float16case to the tests. How large is the error, and why? - Fuse more: write an op that takes
gate,up, and a scale, and returnssilu(gate) * up * scale. Does the timing change?
Next
Section 3.2 looks underneath
compile: the graph MAX builds from your module, and the graph API you can use
to build one directly.