category: fleet-family · status: shipped (v0.1; vulhub slots pending docker farm) · updated: 2026-11-02

Held-out Eval — private true-north scoring

Runnable code: fleet/held-out-eval/ (README.md + ROTATION.md there are the operator docs; this page is the vault view). Part of stream D, feeds [[RESULTS]] rows. Never used for training.

What it is

20 scenarios per ISO week composed from the OTHER fleet families — procedural-generator instances at fixed-but-rotating seeds, purple-loop hunts, vulhub environments with per-run canaries, EVM challenges — with per-run randomized flags/ports/paths, scored pass@k on fresh instances (attempt j re-materializes the scenario at nonce j−1), per-family breakdown, and cost tracking (wall time always; tokens measured or flagged-estimated). Emits a RESULTS.md-compatible row.

Current composition: procedural + purple-loop real (10 slots), evm stand-in (seeded stack-VM, labeled standin: true, 5 slots), vulhub spec-reserved (requires: docker, scored no_attempt, 5 slots) — 15/20 runnable today.

Why it matters

CTFusion (ICML 2026) showed static benchmarks are contamination-inflated and live evaluation is robust; our weekly procedural rotation buys the same property without depending on live events. This eval is the truth anchor for every [[RESULTS]] row: public harnesses can drift or leak; this one rekeys, reseeded, every week.

Mechanics

• Seeds: BASE + slot*1009 + week_index*7919, week monotone → never repeat;

nonces hash-fresh instances per attempt.

• Public manifest: specs only, checksummed; score.py refuses tampered ones.
• Answers are NOT stored anywhere for solvable families — the grader

re-derives them from (seed, nonce); canaries (vulhub) live only in gitignored private/ derived from private/eval.key (0600).

• Verification is execution-graded: flag/canary compare via

hmac.compare_digest; hunts graded against private ground truth by hunt_env.grade_verdicts — the agent never sees labels.

Lineage — copy

• CTFusion: rotate, never score a fixture twice, contamination as the

default assumption, first-correct-flag forwarding-style caching only for infrastructure, never scoring.

• Simbian CDB: deterministic seeded instance simulation; entity/time

obfuscation so memorized strings cannot cross instances; grading on submitted evidence, never self-report.

Lineage — avoid

• CTFusion's dependence on live external events (cadence must never slip —

rotate procedurally).

• Answer material in manifests or logs; training seeds overlapping the eval

seed space; publishing per-instance results (aggregates only).

Status & next

• [x] v0.1 shipped 2026-11-02: builder + scorer + rotation policy + smoke

green (baseline plumbing run: pass@2 0.93 over 15 runnable).

• [ ] Swap evm stand-in for fleet/evm-asserts when the family lands.
• [ ] Activate vulhub slots when the docker farm ships (canary verification

already implemented).

• [ ] Delegate procedural slots to the procedural-generator family.
• [ ] Quarterly memorization audit of attempt logs (per ROTATION.md).

See also: [[purple-loop]] · [[landscape-2026]] · [[RESULTS]]