An oracle (or verifier) is the function that answers "did the agent actually win?" during training. In RLVR — reinforcement learning from verified rewards — it is the entire reward signal: the model gets gradient from the oracle and nothing else. So the oracle's quality is the training run's quality. A good oracle is deterministic, cheap, per-run, and aligned with real capability. A bad oracle is a checkers target: the policy learns to satisfy the checker, not to do the work. That failure mode is reward hacking, and every design choice below exists to block a specific way of faking a win.
The cheapest way to see the difference: "an LLM judge reads the agent's report and scores it 1–10" is a bad oracle in cyber, because prose is forgeable. Novee says it plainly: "we don't grade claims, we grade execution" (NoveeGym-RL) — an LLM-judged writeup "rewards persuasive prose over working exploits," and the policy will learn to write better prose, not better exploits.
1. CyberFactory's crash-pre-not-post differential oracle. A PoC (proof of concept) counts as a win only if it **crashes the pre-patch build and does not crash the post-patch build, both under the sanitizer (CyberFactory, arXiv:2608.23181; the protocol is CyberGym's; details in [[cyberfactory-openaegis]]). Worked example: the agent submits a script for CVE-XXXX-YYYYY; the harness runs it against two pinned Docker images — vulnerable commit and fix commit. Crash in both → rejected (that's an environmental crash, e.g. abort() or a bad command line — not the bug). Crash in neither → rejected. Crash only in the vulnerable build → reward 1, and the trajectory is kept. Blocks:** reward hacking via trivial crashes ("make it crash any way possible") and via unverifiable claims. The win condition is behavior of a real binary, which is simultaneously the thing we actually want the model to learn. One oracle, two builds, zero judgment calls.
2. Bugcrowd/ExploitBench's per-run randomized challenge-response. Deterministic oracles run the binary; but what about memorized answers? ExploitBench (arXiv:2605.14153, Lee & Brumley; exploitbench.ai) grades V8 exploitation across a 5-tier ladder (coverage → crash → primitives → arbitrary R/W → code execution) using per-run randomized challenges: to prove arbitrary write, the exploit must write a value the harness chose at random this run to an address chosen this run; code execution is proven via signal-handler catches of the run's random token. A canned exploit from a writeup fails every run, because the question changes. Blocks: memorization and replay — the anti-contamination principle of [[contamination-and-decontamination]] pushed into the reward function itself. Also blocks "almost-right" exploits: partial control can't answer a challenge it wasn't told in advance.
3. Novee's re-execute-the-exploit grading. For web targets, "the validator takes the agent's exploit, runs it against the [live] app, and checks whether the attack actually fired" — a deterministic validator with anti-cheating checks, implemented per vulnerability class as plug-ins, because "clean, cheap, un-gameable verification can be non-trivial" (NoveeGym-RL; dossier in [[landscape-2026]]). Worked example: an XSS task — the submitted payload gets driven into the live app via a real browser session, and the reward is 1 only if the payload executes in the DOM, not if the report says it did. Blocks: judge-gaming (persuasive writeups), screenshot/claim spoofing, and exploits that only work against a stale snapshot.
4. On-chain asserts for EVM levels (our fleet). For web3 environments, the oracle is the chain itself: the level deploys a contract with an invariant assert (e.g., "vault balance ≥ deposits unless owner withdraws"); the agent's transaction sequence is simulated/submitted, and the win is the assert firing — state transition, not narrative. Any EVM node is a deterministic re-executor for free. Blocks: claimed-success stories about "draining the pool," and grader drift — the assert is code in the level, reviewed once, not an opinion re-evaluated per run.
5. pySigma compile + score for detections (the blue side). A detection task's output is a Sigma rule. Gate one: it must compile and pass validators in pySigma (sigma check; SigmaHQ QA pipeline) — catches malformed YAML and junk logic for cents. Gate two — the real one — is execution scoring: run the rule against red-team telemetry (Atomic Red Team executions from our purple loop) plus benign baseline logs; reward = fires on the attack, silent on the noise. Blocks: detection slop — fluent, plausible rules that match nothing (the LLM equivalent of a SOC analyst who writes great queries for attacks that never happened), and over-broad rules that alert on everything (reward = precision ∧ recall, not recall alone).
| Oracle design | Hacking route it blocks |
|---|---|
| Differential (pre-crash ∧ post-clean) | trivial/environmental crashes, unverifiable claims |
| Per-run randomized challenge-response | memorized writeup answers, replay |
| Re-execute vs live app | prose-gaming a judge, stale-snapshot exploits |
| On-chain asserts | narrative wins, grader opinion |
| Compile + fire-and-stay-quiet scoring | plausible-but-dead rules, alert-everything rules |
The meta-rule: never let the reward depend on anything the policy can control with words. Crash logs, execution traces, blockchain state, and rule-firing counts are outside the policy's pen. LLM judges inside training loops are not — keep them for dataset QA ([[dataset-lineage]]) and CyberQA-style synthesis, not for the reward.
Our factory is verified trajectories: trajectories = fleet ∧ teacher — nothing enters SFT/RL unless it passed an oracle, and weekly refresh demands the oracle be cheap enough to run millions of times. The five patterns above are the design space we copy from CyberFactory, Bugcrowd, and Novee — and extend to classes they don't cover (EVM asserts, Sigma scoring, purple-loop telemetry hunts).
[[contamination-and-decontamination]] · [[dataset-lineage]] · [[cyberfactory-openaegis]] · [[bugcrowd-rl-environments]] · [[landscape-2026]]