Status: active · Owner: stream F (eval harness) · Applies to: fleet/held-out-eval/
CTFusion (KAIST, ICML 2026 — openreview, code) showed the academic seal on what we already believed: static CTF benchmarks are contamination-inflated — published challenges leak into pretraining corpora and agents can recover known solutions via web search; evaluation on live/freshly-generated events is robust. Their pipeline streams live CTFs; ours can't always wait for a live event, so we get the same property procedurally: every week the manifest's seeds rotate, and every attempt re-materializes a fresh instance from a new nonce. A model cannot memorize its way to pass@k because no two scored instances are alike.
Simbian CDB adds the same lesson from the blue side: their campaign simulator time-shifts and entity-obfuscates the raw recordings precisely so string memorization from public datasets (Mordor) cannot solve episodes. Our purple-loop family inherits that; this eval inherits it at the composition layer.
| what | value |
|---|---|
| cadence | weekly (ISO week), rebuilt every Monday before any eval run |
| scenario count | 20 per week (5 per family, interleaved slots) |
| families | procedural · purple-loop · vulhub · evm (see builder FAMILY_ORDER) |
| seed formula | GLOBAL_BASE + slot*1009 + week_index*7919 (+ nonce hash for attempts) |
| week_index | iso_year*53 + iso_week (monotone) |
| nonce | attempt j uses nonce j-1 → fresh instance per attempt (pass@k on fresh instances) |
| window | seeds never repeat: week_index grows monotonically; slot strides are prime |
| manifest | eval-manifest-<week>.json public (specs only, checksummed) · answers/canaries never in it |
Same week + same nonce ⇒ byte-identical instances (reproducibility for audits); different week or nonce ⇒ different flags, ports, paths, hunt injections.
Run through this EVERY time a new family is added or a rotation lands:
(slot, week, nonce) at grading time, not stored. Canary values live only in private/ (gitignored, 0600) derived from private/eval.key.
secret per nonce), ports (vulhub 10000–64000), paths (/.canary-<8hex>), hunt injection points/entities/GUIDs/hashes.
--k): pass@k must measuregeneralization; replaying the same instance is explicitly not scored.
score.py before any run — refuse toscore a tampered manifest.
(none do today; vulhub will be local docker).
flag reuse across nonces, sub-second solves of multi-stage hunts, suspiciously identical reasoning traces across instances.
different seeds (e.g. purple-loop training runs use the fleet's own seeds, not the eval week formulas). If a family's instance generator is shared, the eval's (slot, week, nonce) space is excluded from training sweep ranges.
RESULTS.md rows and per-family pass@k;no instance material, no flags, no transcripts on the public site.
against public CTF corpora (picoCTF-style affine/XOR/base64/CRT puzzles are deliberately generic shapes, not copies of specific challenges).
set standin: true in the manifest and be swapped, not silently kept, when the real family ships.
cd fleet/held-out-eval
python3 builder.py # current ISO week manifest + private canaries
python3 score.py --manifest eval-manifest-$(date +%G-W%V).json \
--agent module:/abs/path/to/your/agent.py:attempt --k 3
# append the results_md_row string to ../../RESULTS.md
Rotation of the pool composition (which vulhub apps, which procedural types) follows the seeds automatically; a human reviews the manifest diff at the quarterly sync (stream F ticket).