fleet/purple-loop/)The red-trains-blue loop nobody has shipped: a red agent generates randomized ATT&CK attacks, the attacks are rendered into Sysmon-style telemetry (or run in a real Windows lab), and a blue agent hunts them — both trajectories are kept as verified training data. This is the training-loop closure our [[landscape-2026]] dossier identified as unclaimed white space (Simbian benchmarked blue but trains nothing; CyberGym-E2E is eval-only).
┌──────────────────────────────────────────────────────────┐
│ PURPLE LOOP (per run) │
│ │
seed + ───────►│ attack_gen.py RED │
weekly refresh │ ── pick N atomics (T1003, T1059, T1105, T1560, ...) │
│ ── randomize: file names, paths, payload hashes, │
│ hosts/users/domains, GUIDs, ports, timing │
│ ── emit attack_plan.json (kill-chain ordered) │
│ │ │
│ ▼ │
│ telemetry_sim.py BRIDGE │
│ ── Sysmon-style JSON: EID 1/3/5/7/10/11/13/22 │
│ (process trees, image loads, file writes, registry, │
│ network, DNS — real, decodable command lines) │
│ ── benign noise: synthetic fleet, or bootstrap from an │
│ exported real stream (e.g. BOTSv3, CC0) │
│ ── randomized injection point + per-step dwell jitter │
│ ── ground_truth.json ◄── PRIVATE, never shown to blue │
│ │ (labels: technique, guid, │
│ │ timestamps, narrative steps)│
│ ▼ │
│ ┌─────────────────┐ ┌───────────────────────────┐ │
│ │ hunt_env.py │ BLUE│ sigma_gen.py DETECTOR │ │
│ │ DuckDB(SQLite) │ │ Sigma YAML in │ │
│ │ 50-query budget │ │ → SQL over events │ │
│ │ 10-row results │ │ → EQL equivalent │ │
│ │ verdicts = tech │ │ → P/R/F1 vs labels │ │
│ │ + entity + win │ │ (per technique) │ │
│ └────────┬────────┘ └────────────┬──────────────┘ │
│ │ grader: coverage (Simbian-style narrative │
│ │ steps), precision, per-technique/tactic │
│ │ recall, attribution gap │
│ ▼ │
│ trajectories/: red plan + blue transcript + detector │
│ scores — BOTH SIDES' trajectories kept for SFT/RLVR │
└──────────────────────────────────────────────────────────┘
improvement direction: red variants that survive the blue grader raise the
bar; blue hunts/detectors that catch more variants raise theirs. Filtered
both ways → verified trajectories for post-training (see ../../RESULTS.md).
| file | role |
|---|---|
attack_gen.py | red: pick atomics, emit randomized per-run variants (plan.json) |
telemetry_sim.py | bridge: synthesize Sysmon-style events.jsonl + private ground_truth.json |
hunt_env.py | blue: SQL threat-hunt environment + grader + reference agent |
sigma_gen.py | detector: Sigma YAML → SQL/EQL → precision/recall vs labeled stream |
rules/ | sample Sigma rules (fire on the embedded catalog) |
smoke.sh | end-to-end: attack → telemetry → hunt → sigma |
cd fleet/purple-loop
./smoke.sh # full loop, stdlib-only (SQLite path)
python3 attack_gen.py --seed 20261102 --num-atomics 6 --out runs/demo/plan.json
python3 telemetry_sim.py --plan runs/demo/plan.json --noise-events 6000 --out-dir runs/demo
python3 hunt_env.py --events runs/demo/events.jsonl --ground-truth runs/demo/ground_truth.json \
--agent demo --out runs/demo/hunt_results.json
python3 sigma_gen.py --events runs/demo/events.jsonl --ground-truth runs/demo/ground_truth.json \
--rules rules --out runs/demo/sigma_results.json
Optional deps (all documented, all degrade gracefully):
pip install duckdb — DuckDB engine (default when importable; else sqlite3).pip install pyyaml — full YAML for --art-dir imports and arbitrary Sigmarules (built-in mini-YAML covers the documented subset otherwise).
pip install pysigma — Sigma schema validation in sigma_gen.py(conversion is ours either way, so output is identical without it).
Episode = one campaign. reset() → briefing + schema_help + actions.
{"action": "run_sql", "sql": "SELECT ..."} // costs 1 of 50 queries
{"action": "submit_verdicts", "verdicts": [ // free (Simbian-style)
{"technique_id": "T1003", "entity": "rundll32.exe",
"time_window": {"start": "2026-11-02 08:00:00", "end": "2026-11-02 08:05:00"},
"note": "comsvcs minidump"}]}
{"action": "give_up"}
Observation for run_sql: {"query_result": {"ok", "columns", "rows" (max 10), "row_count" (always full), ...}}. A verdict is a true positive iff its technique matches an as-yet-uncovered narrative step, its time window overlaps the step's events, and its entity equals (case-insensitive) a step entity — full path, basename, host, user, or process GUID.
Grader output: coverage_score (fraction of narrative steps — the primary metric), precision, recall per technique and per ATT&CK tactic, evidence_seen_malicious / attribution_gap (evidence seen in query results minus verdicts landed — Simbian's observed failure mode).
Plug an LLM in with --agent module:/abs/path/agent.py:hunt where hunt(env, info) loops env.step(...), or replay a fixed policy with --agent script:actions.jsonl.
Field names are Sysmon EventData names (identical to the events columns). logsource category → EventID filter: process_creation→1, network_connection→3, image_load→7, process_access→10, file_create→11, registry_set/registry_add→13, dns_query→22.
| Sigma construct | SQL emitted (both engines) | EQL emitted | |
|---|---|---|---|
plain value x | LOWER(col) LIKE LOWER('x') ESCAPE '/' | field == 'x' | |
wildcards */? | LIKE with %/_ (metachars escaped) | field like~ 'x' | |
| `field\ | contains` | LIKE '%x%' | field like~ 'x' |
| `field\ | startswith` | LIKE 'x%' | field like~ 'x*' |
| `field\ | endswith` | LIKE '%x' | field like~ '*x' |
| list for one field | OR (AND with `\ | all`) | or (and with all) |
| several fields in one selection | AND | and | |
value: null | col IS NULL | field == null | |
| `field\ | re` | Python-side regex post-filter (portable) | field regex~ '...' |
sel1 and not filter1 | (...) AND NOT (...) | ... and not ... | |
1 of sel*, N of them, all of them | SUM(CASE) >= N / chained AND | or/and + comment | |
condition list [a, b] | disjunction of conditions | disjunction | |
| keyword-only selection, `\ | base64offset`, flow mappings, … | rejected loudly (UnsupportedSigma) | — |
Scoring: per rule — matched events, TP/FP/FN, precision, recall, F1. Recall denominator = malicious events of the rule's attack.tXXXX tag(s) if tagged, else all malicious events (so a rule is judged on what it claims to cover).
attack_technique / display_name /atomic_tests[] with name, description, supported_platforms, input_arguments{name:{description,type,default}}, executor{name, command, elevation_required, cleanup_command}, dependencies — mirrored by our embedded catalog; --art-dir imports the real thing from a local checkout.
5/7/8/10/11/12/13/14/15/17/18/19-21/22/23/25/26 field names as emitted here (ProcessGuid, Hashes SHA1=,MD5=,SHA256=,IMPHASH=, TargetObject with HKLM/ HKU abbreviations, …).
tags) and pySigma (pip install pysigma, LGPL-2.1, SigmaRule object model, backends/pipelines) — used as documented above.
(wineventlog, Sysmon, stream:*, CloudTrail, …). We consume it only as an exported JSONL noise bootstrap — export recipe below.
50-query budget, 10-row results with full counts reported, free flag submission, coverage over narrative steps, attribution-gap finding, time-shift + entity-obfuscation anti-memorization. Their engine is SQLite (505-column flat logs table); ours is DuckDB-with-SQLite-fallback — an intentional divergence, contract identical.
BOTSv3 ships pre-indexed for Splunk, so JSONL must be exported first: install the dataset under $SPLUNK_HOME/etc/apps, then e.g.
index=botsv3 sourcetype="xmlwineventlog:microsoft-windows-sysmon/operational"
| table _time, EventID, Computer, UtcTime, ProcessGuid, Image, CommandLine, ...
| outputtext delim=comma
(or splunk search ... -output json). Feed the result to telemetry_sim.py --bootstrap export.jsonl. Without it the synthetic fleet is used; both paths are labeled in ground_truth.json:bootstrap.
./smoke.sh runs the whole loop on three seeds and asserts: events were generated; malicious events exist and were labeled; the reference hunter lands coverage > 0 with precision = 1.0 on at least one seed (and < 1.0 coverage on at least one other — variants that survive hunting are the point); every sample Sigma rule compiles; at least one rule scores F1 > 0. Exit 0 = green.
decodable (e.g. -enc payloads are real UTF-16LE base64), but no Windows host ran them. Event-field fidelity ⊂ real Sysmon (8 event IDs, no CallTrace detail, no HKCR renames). A real-lab path (Invoke-AtomicRedTeam → WEF/Winlogbeat → this schema) is the planned upgrade; the grader contract does not change.
--art-dirimports real atomics, but imported tests get generic process-creation-only traces until per-technique telemetry templates are written.
Run-key/LSASS lookalikes by construction — asserted). BOTSv3 bootstrap helps realism but its Sysmon share is a 2018 CTF's, not an enterprise's.
window matching rewards exactly-attributed hunts; no partial credit for coarse windows (deliberate, cf. Simbian's microsecond-precision stance).
PowerShell/4688, no network sensors). Lateral movement renders as one host.
base64offset,correlation rules and flow YAML are rejected, not approximated. Rules from the SigmaHQ corpus (DRL 1.1) are NOT vendored here for license reasons.
schema_help); our reference queries use LOWER() everywhere. The SQL guard is a referee, not a security boundary.
module:/script:runners); model calls belong to the training harness (streams F/D), keeping this component offline-reproducible.
counts; free verdict submission + give_up; coverage scored over narrative steps; per-tactic recall table; attribution-gap diagnostic; seeded time-shift/entity-substitution/GUID re-randomization; determinism given a seed.
(ours: open, reproducible from a seed, ground truth stays with the episode); timestamp-only grading (ours: technique + entity + window = hunting, not copying); zero training on their own benchmark (ours: both trajectories kept as training data — that gap is the reason this family exists).
survives a reseed; evaluate on fresh instances, not static fixtures.
(ART YAML schema, Sigma schema, CC0 noise data) — with licenses respected (no SigmaHQ rule content vendored).
See also: [[held-out-eval]] (this family feeds the private eval) · [[landscape-2026]] · [[cyberfactory-openaegis]]