category: training · status: draft · updated: 2026-11-02

SFT vs DPO vs GRPO vs RLVR

You have a base model — Qwen3.8-27B or 35B-A3B, straight off the shelf. It can write code and chat, but it doesn't reliably pop a shell, triage a SOC alert, or know when to stop probing and start writing the report. Post-training is everything you do after pretraining to close that gap. There are four regimes that matter, and they differ on one axis: where the signal comes from.

• SFT (supervised fine-tuning) — signal from examples. "Here is a full solution; imitate it."
• DPO (direct preference optimization) — signal from contrasts. "This response is better than that one; move probability from loser to winner."
• GRPO (group relative policy optimization) — signal from trial and error. "Try the task 16 times; whatever the group scores, push toward the winners."
• RLVR (RL with verifiable rewards) — not a fourth algorithm but a reward source: the score comes from a program that checks the work (did the PoC crash the pre-patch build?), not from a human or a learned reward model. GRPO over our fleet's differential oracles is RLVR.

Mechanism, one paragraph each

SFT is next-token prediction on curated transcripts. Take a verified trajectory from our fleet — the full interleaving of prompt, tool calls, environment output, self-correction, final flag — and train the model to predict each token of that trajectory given everything before it. Loss: cross-entropy. It's flashcards. Cheap, stable, and it teaches format, tool syntax, and workflow shape. What it can't do is teach the model anything the transcripts don't already demonstrate.

DPO (Rafailov et al., arXiv:2305.18290) trains on pairs: a chosen and a rejected response to the same prompt. The loss directly increases the log-ratio of chosen over rejected relative to a frozen reference model — no reward model, no RL loop, no rollouts. It costs about the same as an SFT step. It's for preferences you can't verify with a program: terser reports, safer phrasing, not giving up after one failed exploit attempt.

GRPO (from DeepSeekMath, arXiv:2402.03300) samples a group of responses per prompt (say 16), scores each, and computes each response's advantage as (its score − group mean) / group std. No value model to train — the group itself is the baseline. The policy then gets nudged toward above-average rollouts via a PPO-style clipped objective. This is on-policy: the model is learning about its own current attempts, which is why it generalizes where SFT memorizes.

RLVR (popularized by Tulu 3, arXiv:2411.15124) is what makes GRPO usable for us. Instead of a reward model you can't trust on cyber tasks, the reward is an executable check: the ARVO-style differential oracle (PoC crashes the pre-patch container, not the post-patch one), an EVM assert, a flag file check, a DuckDB query recall target on the blue side. "We grade execution, not claims."

Compute reality for 27B and 35B-A3B on rented GPUs

The two bases have opposite cost profiles. Training memory scales with total parameters; rollout cost scales with active parameters.

• 27B dense: full-param SFT needs weights + gradients + AdamW states ≈ 16 bytes/param ≈ 430 GB plus activations — 8–16 rented H100s under ZeRO-3/FSDP, a few $k for a 3-epoch pass over ~50k trajectories (our CyberFactory rebuild notes). LoRA on it runs overnight on one 8-card node.
• 35B-A3B (MoE): only 3B params fire per token, so rollouts are dirt cheap — this is the dream shape for GRPO, where 90% of the bill is generating attempts under vLLM (Novee runs 500–1,000 concurrent rollouts this way). But training touches all 35B params (every expert gets gradients and optimizer states), so the trainer looks like a 35B model: ~560 GB before activations. The trap is remembering that "A3B" saves inference money, not training money.
• DPO costs an SFT-shaped run on pair data. Negligible next to GRPO.

Why GSPO exists (MoE stability)

GRPO's importance ratio is computed per token and clipped per token. In a MoE, tokens route to different experts; a few aggressively clipped tokens can yank the router, and Qwen's team found this destabilizes expert routing during RL — sometimes needing router-replay hacks. GSPO (Group Sequence Policy Optimization, arXiv:2507.18071) computes the importance ratio over the whole sequence (geometric mean of token ratios) and clips at the response level, which they argue is better-posed and demonstrably more stable for MoE — Qwen3 was RL-trained with it. Rule of thumb for us: GRPO is fine on the dense 27B; reach for GSPO when training the A3B.

Our pipeline: SFT → rejection sampling → GRPO/GSPO → DPO garnish

The factory's merge gate says RL = SFT warm-start, and the order is not negotiable, because each stage manufactures the preconditions of the next:

1. SFT warm-start on verified trajectories distilled from stronger teachers (the CyberFactory pattern: a skill-conditioned GLM 5.2 teacher, its successful rollouts kept by the differential oracle). The base model learns our agent format, tool vocabulary, and the shape of a competent attempt. Starting GRPO from a base model wastes millions of rollouts learning "output valid tool calls" — SFT buys that for pennies.
2. Rejection sampling (see [[rejection-sampling-star]]): the warm-started model itself now attempts fleet tasks k times; verified winners become fresh SFT data. Nearly free — inference cost only — and it aligns the data distribution with our model on our environments.
3. GRPO/GSPO on-policy against the fleet's win-condition checkers (RLVR). Now the model is pushed past its demonstrated behavior, learning from failures as well as successes — the phase that converts memorized workflows into generalized skill.
4. DPO garnish: a thin final layer of preference pairs from the same tasks (clean-vs-sloppy verified solutions, persistent-vs-quitter trajectories) to fix what no oracle measures. Garnish, not meal — pair-only training moves style more than capability.

Worked example. One vulhub SQLi environment. Stage 1: teacher solves it in a verified 40-step trajectory; the model SFTs on it and 49,999 siblings. Stage 2: the model tries 16 variants of a procedural reskin of that task; 5 pass the oracle; dedupe, keep 3, retrain. Stage 3: GRPO with group=16 on fresh procedural variants weekly — reward 1 only when the injected query exfiltrates the marker from the pre-patch container and not the patched one. Stage 4: among two passing rollouts, prefer the one that dropped the two useless nikto scans. Model ships as GGUF/MLX to the M5 Max; RESULTS.md gets its row.

Primary sources: DPO · GRPO / DeepSeekMath · GSPO · Tulu 3 / RLVR · DeepSeek-R1 · [[cyberfactory-openaegis]]

See also: [[rejection-sampling-star]] · [[trajectories-vs-writeups]] · [[cyberfactory-openaegis]] · [[landscape-2026]]