fleet/held-out-eval/)The private true-north eval: 20 scenarios per week composed from the OTHER fleet families — procedural-generator instances at fixed-but-rotating seeds, vulhub environments with per-run canaries, purple-loop hunts, EVM challenges — never used in training, per-run randomized flags/ports/paths, scored pass@k on fresh instances. If the public harnesses say one thing and this says another, this one wins.
| file | role |
|---|---|
builder.py | compose 20 scenarios from the fleet families; emit public + private manifests |
score.py | run an agent (builtin baseline or module:), verify, pass@k + per-family + cost, RESULTS.md row |
ROTATION.md | weekly rotation policy + canary/anti-memorization checklist (CTFusion rationale) |
smoke.sh | build manifest → score with baseline → assert sanity |
private/ | eval.key + per-week canary manifests — gitignored, 0600, never committed |
builder.py --week 2026-W44
slot 0..19 -> family = [procedural, purple-loop, vulhub, evm][slot % 4]
seed = BASE + slot*1009 + week_index*7919 (fixed per week, rotates weekly)
┌──────────────────────────────────────────────────────────────┐
│ procedural REAL affine / xor / base64-chain / CRT │
│ purple-loop REAL attack_gen -> telemetry_sim -> hunt │
│ evm STAND-IN seeded stack-VM puzzle (until family) │
│ vulhub PENDING app + random port + /.canary-<hex> │
│ (spec reserved; no_attempt until docker farm) │
└──────────────────────────────────────────────────────────────┘
-> eval-manifest-2026-W44.json public specs, checksummed, NO answers
-> private/eval-manifest-*-private canaries only (vulhub family)
score.py --manifest ... --agent ... --k 3
attempt j -> materialize at nonce j-1 (FRESH instance: new flag/port/path)
agent sees: scenario spec (+ events.jsonl for hunts) — nothing private
verify (grader-side, execution-graded):
answer_compare procedural/evm — expected re-derived from (seed, nonce)
canary_compare vulhub — vs private manifest
purple_loop_score hunts — verdicts graded vs private ground truth
-> pass@k, pass@1, per-family breakdown, wall time, tokens
-> JSON with results_md_row compatible with ../../RESULTS.md
Same week, same nonce → byte-identical instance (auditability). Different nonce → same challenge shape, new randomization: new flag secret, new hunt injection/entities/GUIDs, new port/canary path. pass@k therefore cannot be gamed by memorizing a specific instance — only by actually solving the family. (CTFusion showed static fixtures are contamination-inflated; Simbian's simulator does the same trick per-campaign. Details in ROTATION.md.)
--agent module:/abs/path.py:attempt where
def attempt(scenario: dict, workdir: Path, helpers: dict) -> dict:
# scenario: the public manifest entry (family, kind, spec, verify-bar)
# helpers: {"events_path": str|None, "fleet": <fleet dir>}
return {"submitted": "cf{...}", # flag / answer
"usage": {"prompt_tokens": 0, "completion_tokens": 0}, # optional
"log": "optional transcript (token estimate)"}
# purple-loop: submitted = list of verdict dicts (technique_id, entity,
# time_window); hunt the stream at helpers["events_path"]
# using fleet/purple-loop/hunt_env (ThreatHuntEnv without gt)
# cannot try? return {"no_attempt": "reason"} — excluded from denominator
builtin:baseline demonstrates the contract: solves procedural + evm stand-ins algorithmically, hunts with the purple-loop reference heuristics, declares no_attempt on vulhub. It exists to prove the plumbing, not to set the bar.
Wall time per attempt always; tokens from agent-reported usage when available, else a chars/4 estimate over the returned log (flagged tokens_estimated in the JSON — never silently presented as measured).
(requires: docker, scored no_attempt, excluded from the runnable denominator and reported as pending) until the vulhub farm family ships; evm slots run a seeded stack-VM stand-in labeled standin: true — it is NOT the EVM and will be swapped for fleet/evm-asserts the day it lands. pass@k today covers procedural + purple-loop + stand-in evm (15/20 slots).
CRT). They exercise rotation/verification mechanics, not deep skill; depth comes when the procedural-generator family ships and this adapter delegates.
validation, not a competitive number. Report model runs with module:.
inherited from the purple-loop v0.1 grader; revisit after human validation.
private/ is gitignored and 0600, but acareless copy of the directory leaks canaries. Key rotation = regenerate eval.key (invalidates old canary manifests; answer-derived families are unaffected).
module: agents —run untrusted agents under your own container/VM discipline.
weekly; treat contamination as the default assumption; cache/replay logic only for infrastructure isolation, never for scoring.
entity/time obfuscation so nothing string-memorizable crosses instances; execution-graded verification (their flag-timestamp checking; our verdict grading + canary comparison), never self-reported success.
rotate procedurally instead, so the cadence never slips.
seeds into the eval seed space; publishing per-instance results.
See also: [[purple-loop]] (feeds the hunt family) · [[landscape-2026]] · ../../RESULTS.md (rows this emits) · ROTATION.md