Procedural Generator — the anti-contamination core

Randomized, seeded challenge instances across five classes, generated and verified locally, stdlib-only. This is the fleet's answer to benchmark contamination: never train twice on the same bytes.

Upstream lineage (verified 2026-11-02):

Improving LLM Agents with Reinforcement Learning on Cryptographic CTF Challenges (arXiv 2506.02048, Muzsai/Imolai/Lukács): pick a scheme subtype, randomize the scheme's parameters (cipher keys, substitution dictionaries), encrypt a random flag{...}, wrap it in a narrative → "near infinite variations" across 50 subtypes / 8 archetypes. The paper RL-trained (GRPO) a tool-using agent from the EnIGMA agent lineage (arXiv 2409.16165) and validated on unseen challenges and external benchmarks.

(NYU CTF Bench, HackTheBox-derived sets, …) are contaminated — models have seen the solutions in pretraining, inflating scores. Their fix: continuously generated "Live CTFs" instead of static sets.

We take the union of those two arguments and generalize beyond crypto.

The contamination argument, concretely

  1. Static benchmarks leak. Every picoCTF/CTFtime writeup, every HTB walkthrough

is in pretraining corpora. A model can score on memorized challenges without any capability — CTFusion measured static CTF benchmarks being unreliable for agent assessment.

  1. Randomized parameters defeat memorization. Random-Crypto's 50 subtypes with

per-instance keys/plaintexts mean the procedure (decode → analyze → invert) is the only thing that generalizes. That is exactly the behavior we want to reinforce.

  1. Contamination is not just a pretraining problem — it is a fine-tuning

problem. Training on a fixed instance set lets the policy overfit surface features (fixed ports, paths, flag prefixes). Randomizing everything the agent can anchor on (flags, values, ports, paths, positions, timings) forces the reward to flow through the actual exploit/solve procedure.

  1. Verifiability scales. Each instance carries a machine-checkable win

condition, so infinitely renewable tasks still yield exact RLVR rewards — Random-Crypto's core insight, now in five domains instead of one.

Weekly refresh doctrine

window (e.g. ISO-week-based: seed = week_number * 1_000_000 + k), emits task.json sets for that week's episodes, and archives the previous week. Old instances are never re-fed to training on the same seeds.

task.json (agent-visible) + solution.json (private). Trajectories record the verifier verdict, not the solution. Agents get no solution text at any point in the loop.

(and disjoint variant choices at the same difficulty) so eval instances are guaranteed-unseen even against our own training pipeline.

instance (asserted by the smoke test), so any episode is replayable and auditable; freshness comes from choosing new seeds, not from noise.

Classes (≥5)

ClassWhat varies per instanceWin condition
cryptovariant (Caesar, Atbash, base64, single/repeating XOR, add-shift, Vigenère-with-crib, encoding chains, RSA-small-e), keys, plaintexts, chain order/depthsubmit exact flag{...}
webbase path, endpoints, admin user, canary file, token dir, decoys; SQLi (string-concat SQLite), path traversal (root-confined one level too high), simulated command injectionsubmit exact flag (multi-vuln instances = dash-joined fragments)
forensicshost, day, users, IPs, technique, injection position/timing, noise volume, decoy pattern (d5)name technique (aliases ok) + timestamp inside attack window
reconservice/version set, ports, vulnerable↔decoy mixexact set of exploitable CVEs (aliases ok)
privescsudoers entries, host, GTFOBins choice / wildcard / LD_PRELOAD / writable-script misconfig, decoy rules, shufflesname the abusable binary + technique vector

Difficulty dial 1–5 per class (see generator.py mapping tables): more vulns chained, more decoys, subtler patterns, tighter tolerances.

Safety by construction: the web service is a throwaway in-memory app — the SQLi hits only a sandbox SQLite DB, traversal is confined to a service-created temp root, and command injection is simulated (no subprocess ever touches attacker input). Nothing generated here touches a real target or a real secret.

Layout


fleet/procedural-generator/
├── generator.py    # class builders + instance assembly (stdlib only)
├── verifier.py     # per-class answer checking with normalization/aliases
├── smoke_test.py   # 3 instances per class, solved + verified + leak-checked
└── README.md

Usage


# one instance
python3 generator.py --class crypto --seed 1234 --difficulty 4 --out /tmp/inst
# -> /tmp/inst/instance-crypto-1234-d4/{task.json,solution.json,challenge.txt}

# verify an agent's answer (exit 0 = correct)
echo '{"flag": "flag{...}"}' | python3 verifier.py --instance /tmp/inst/instance-crypto-1234-d4 --answer -

# full smoke: 15 instances, all solved by construction, verified, leak-checked
python3 smoke_test.py

Library use from the fleet:


from generator import generate_instance
from verifier import verify
inst = generate_instance("web", seed=77, difficulty=5, out_root="/tmp/week43")
task = json.load(open(inst + "/task.json"))   # give the agent this only
verdict = verify(inst, agent_answer)          # {"correct": bool, "score": float, ...}

What the smoke test proves

  1. Solvability by construction — every instance is solved from its own

solution.json (crypto inverted, web exploited over real HTTP on an ephemeral port, forensics/recon/privesc answered structurally).

  1. Verifier soundness — true answers pass, mutated answers fail.
  2. No leakage — flags/keys/answers never appear in task.json (or

public artifacts where they'd be answers).

  1. Determinism — same (class, seed, difficulty) reproduces identical

task.json.

  1. Web services compile and survive their exploit traffic.

See also