category: infrastructure · status: draft · updated: 2026-11-02

MTP speculative decoding: the detachable draft head

Modern Qwen bases ship with a stowaway: a tiny extra transformer head trained, during pretraining, to predict not the next token but the one after it (and often two after). It exists to make pretraining loss cheaper per byte. But at inference time it is a free draft model that shares the trunk's hidden states, tokenizer, and KV cache — and that turns it into the single biggest speed lever we have on Apple Silicon. The catch: it's calibrated to the pretrained trunk. Every time we finetune the trunk, we must re-validate it.

The mechanism

Speculative decoding is a two-model game:

A cheap drafter proposes the next k tokens, one at a time.
The big verifier scores all k in a single batched forward pass, and accepts the longest prefix that matches what it would have generated anyway.
Rejected suffix is discarded; the verifier's own token is kept. Output distribution is mathematically identical to slow decoding — it's lossless, not an approximation.

Expected tokens per step is (1 − α^(k+1)) / (1 − α), where α is per-token acceptance. At α = 0.75 with k = 3 drafted tokens that's ~2.7 tokens per forward pass — nearly 3x if drafting is cheap and verification batches well. (Empirically k = 2–3 is the sweet spot; beyond that acceptance sags and wasted drafts eat the gain.)

Normally the drafter is a second model you keep in RAM and keep in sync. Multi-token prediction (MTP) folds the drafter into the checkpoint: one or two extra layers that read the trunk's final hidden state and predict token t+2 / t+3. The idea goes back to Gloeckle et al. 2024; DeepSeek-V3 made the MTP module explicitly detachable — train with it, ship with or without it — and Qwen3-Next shipped MTP layers in an 80B-A3B open model in Sep 2025. The Qwen3.6/3.7/3.8 lineage kept them, so our Qwen3.8-27B and 35B-A3B bases arrive with trained MTP heads in the weights.

Runtime reality on Apple Silicon (checked 2026-11)

The head is only worth what the runtime does with it:

oMLX — native MTP speculative decoding on MLX. Community runs report ~2.4–2.9x on structured, code-like generation — exactly our trajectory format.
MTPLX (github.com/youssofal/MTPLX) — a dedicated Apple Silicon MTP engine claiming 2.2–3x tokens/s over standard MLX decoding.
llama.cpp --draft-mtp — excellent on CUDA (70–90% acceptance reported on Qwen3.6/3.7), weak on Metal for us: 4–9% net uplift. This matches public reports: LM Studio tracker issue #1969 (May 2026) found MTP variants of Qwen slower than non-MTP on Apple Silicon, and a Hugging Face discussion on Qwen3.5-9B measured no throughput gain on Metal even at 100% draft acceptance — the verify batch doesn't amortize over Metal kernel overhead. The engine, not the head, is the bottleneck.
Ollama — bundles its own lagging llama.cpp fork; MTP support depends on when it syncs, and it exposes no draft configuration. Not a fast path today; re-check each release.

Reference datapoint on older silicon: a 27B dense Qwen on M1 Max went 12.0 → 19.1 tok/s at 78% acceptance (~1.6x — dense models are bandwidth-bound, so the ceiling is lower than on A3B MoE). Our 35B-A3B only pays ~3B active parameters per token, so draft+verify is far cheaper per token: 2x+ on oMLX/MTPLX is realistic, which is how the 150+ tok/s goal on M5 Max is reached at all.

Why we freeze the head during SFT and re-attach at inference

The MTP head is only a good drafter while its predicted t+2 distribution matches the trunk's t+1 distribution. SFT and RLVR move the trunk. That creates a fork in the road:

Joint training would need the MTP deep-supervision objective (predict t+2 from trunk state at t). Plain SFT loss doesn't teach that; applied naively it corrupts the head's calibration.
Freeze and re-attach is simpler and recoverable: we detach the head (don't load mtp.* tensors into the trainer at all), full-parameter SFT the trunk, and re-attach the untouched MTP tensors at GGUF/MLX export. The head keeps its architecture and vocabulary; only its statistical alignment to the trunk has drifted.

Drift is measurable, so we measure it. After every finetune:

Decode a fixed 200-prompt suite (fleet trajectories, not chat) with speculation on and off.
Log acceptance rate and tokens/step as a standing row in RESULTS.md. Healthy is α ≥ 0.6 after SFT; below that, speculation costs more than it earns.
If α dropped, run head-only calibration: LoRA on just the MTP head, trained with the MTP objective over our verified-trajectory corpus. It's one layer — this trains on the M5 Max itself in minutes, not a GPU rental.

One extra trap: llama.cpp issue #25618 documents draft-mtp output diverging from non-speculative decode when the target is quantized. Our harness therefore carries a bitwise spot-check: same prompt, spec on/off, same sampler settings, outputs must match.

Worked example

Weekly refresh, cycle 7: the SFT'd 35B-A3B rolls out against the vulhub farm. Baseline decode on M5 Max: 78 tok/s. Measured MTP acceptance after SFT: 0.71 at k = 3 → ~2.4 tokens/step → 165 tok/s on oMLX. A 30K-token bounty trajectory finishes in ~3 minutes instead of ~6.5. Overnight fan-out of 400 rollouts: ~33 GPU-hours of decode compressed to ~16 — which is the difference between the rollout server keeping up with the environment fleet or not. If cycle 8's SFT drops acceptance to 0.45, the same test catches it Monday morning and head-calibration runs before the new build ships.

Why it matters here