category: competitor-dossier · type: academic paper; claimed release = VAPORWARE · threat: none (empty repo); inspiration: maximum · reuse: rebuild pipeline from paper spec; ARVO/OSS-Fuzz sources · status: deep-dived · updated: 2026-11-02

CyberFactory / OpenAegis — great paper, phantom release

⚠️ Verified 2026-11-02: the "full code + data + model release" does not exist. The GitHub repo is empty and no OpenAegis weights or datasets are on HuggingFace. The methodology is real and excellent; rebuild it from the paper as spec.

1. Problem & approach

Gap: frontier open-weight models (GLM-5, Kimi K3, DeepSeek-V4) show strong cyber capability but publish no reproducible training recipe; open efforts (CTF datasets, CyberPal, Primus, Cyber-zero) are isolated-task, non-agentic, unscalable; naive rollout scaling fails because agents lack domain priors. CyberFactory (arXiv:2608.23181, v3 Sep 1 2026; Beihang + IQuest + SMU; first author Jian Yang, lead Chengran Yang) unifies data construction, trajectory synthesis, and SFT across three tasks: PoC generation, vulnerability patching, CyberQA.

Instance pipeline (PoC), three sources of increasing difficulty:

• ARVO — ready-made pre/post-patch Docker image pairs with ground-truth PoCs (Mei et al. 2026).
• OSS-Fuzz — only the introducing commit is known; the fix commit is located via ARVO's binary-search method.
• Wild CVEs (hardest) — keep CWE-typed CVEs, locate fix commits from affected-version ranges, build pre/post images, then filter out any instance an inference model can already solve (crash info/CWE hints used only for this check, then discarded). Task descriptions: LLM classifies fix-commit-message quality; weak ones get rewritten from vuln evidence + fix commit.

Patch generation reuses the CVE-Factory recipe over CVEfixes/Vulrepair corpora. CyberQA is answer-first: trusted answers from execution-derived results (crash locations, test outcomes), structure-derived facts (changed functions, call graphs), or authoritative reports; an LLM judge checks traceability, answer leakage, and uniqueness; failures regenerate once or get dropped.

Skill guidance: not a taxonomy — one reusable "vulnerability-analysis skill" (a written procedure): inspect target + build constraints → explore candidate inputs with analysis/testing techniques → validate evidence → revise on failure. It conditions the teacher (GLM 5.2) during synthesis only. Success = the differential oracle: a PoC must crash the pre-patch build and not crash the post-patch build under the sanitizer (CyberGym protocol). Verified trajectories are kept.

Context compaction, precisely: rollouts approach the 256K-token limit. When usage hits τ=0.9 (90%), the accumulated trajectory is rewritten into a continuation state preserving verified evidence, failed attempts, open hypotheses, artifacts, build status, pending actions — discarding redundant logs/search output — then the agent resumes with the same task/tools/oracle. Applied at both synthesis and inference. Ablation: 0.8 over-compacts; 0.95 leaves too little room for the compaction call itself.

2. The released artifacts — the central problem

• GitHub CSJianYang/CyberFactory: an empty repository. No code, no data, no README, no license. Its sole content is issue #1 (Sep 11, 2026): Hugging Face's NielsRogge asking the authors to verify the Papers-with-Code eval rows — still unanswered.
• Hugging Face: nothing. Papers page shows 0 models, 0 datasets, 0 Spaces linked; model search "openaegis" returns zero; the authors' org Multilingual-Multimodal-NLP (22 models) and CSJianYang's profile contain no Aegis.
• No dataset scale reported — trajectory counts, token totals, instance counts per source are all absent (serious omission for a "scaling" paper).
• Only license in force: arXiv's non-exclusive license on the PDF.
• Training recipe (§4.2): init Qwen3.5-397B-A17B; full-parameter SFT, 3 epochs, packed to 131,072 tokens (longer discarded), micro-batch 1 / global batch 32, Adam (β1 0.9, β2 0.95, ε 1e-8), wd 0.1, clip 1.0, cosine LR 2e-5→5e-7 with 10% warmup, BF16, 256 GPUs (TP=8, CP=2, EP=32, ETP=8).

3. Results

CyberGym PoC reproduction, 1-hour budget, Pass@1, identical scaffold (v3):

Model	Pass@1
OpenAegis (Qwen3.5-397B-A17B + their SFT)	58.1
Kimi K2.7 (1T-A32B)	51.7
GLM 5.2 (744B-A40B)	43.3
Qwen 3.5-397B-A17B (base)	29.6

Ablations: compaction is the biggest lever (58.1 vs 52.1 full-history vs 45.6 truncation; on >40-tool-call tasks +8.5 Pass@1, −11.7 pts context exhaustion). Skill on teacher: GLM 5.2 43.3% (1×60min) → 46.5% (5×15min) — ~4× cheaper per successful trajectory, not compute-matched (paper admits). Internalization well-evidenced: exploration/validation calls ~0 → 1.32/1.05 per trajectory without the skill at inference; shell calls 70.1%→89.9%, ASAN compiles 155→1,795, single-submission rate 37.9%→48.2%.

Red flag: the arXiv abstract page still says 52.4% / +22.8 while the v3 HTML body says 58.1% / +28.5 — the abstract was never updated after v1 numbers changed. Headline results moved between versions with no changelog.

4. Successes

• Methodology is sound and imitable: differential oracle + verified-trajectory retention = clean, objective supervision; skill-as-teacher-prior (then distilled away via SFT) is an elegant pattern.
• Compaction ablation is honest and useful (τ=0.9 sweet spot, exhaustion-rate metric).
• Trajectory-analysis section (Tables 4–6) unusually deep — proves the workflow transfers, not just scores.
• Real-vuln grounding via ARVO/OSS-Fuzz/wild CVEs beats synthetic CTF data.

5. Mistakes / limitations

• The release doesn't exist. Paper claims "we train and release OpenAegis" and "we release synthesized instances" — repo empty, HF empty (verified 2026-11-02).
• No dataset-scale reporting; no license; no code — the "unified open-source framework" is currently PDF-only.
• Numbers inconsistency between abstract and body across versions.
• Single benchmark (CyberGym), single teacher (GLM 5.2 — single-teacher bias), skill-vs-no-skill comparison not compute-matched.
• Narrow domain: ARVO/OSS-Fuzz sanitizer oracle ⇒ C/C++ memory-safety bugs; web vulns, logic bugs, crypto, non-English unaddressed (paper admits patch/CyberQA lack rigorous eval).
• No eval-contamination analysis between training CVEs and CyberGym tasks (CyberGym's 1,507 CVEs overlap the same public ecosystem — a rebuilder must dedupe).
• No community criticism exists yet — alphaXiv has zero comments. Nobody has noticed the empty repo.

6. Rebuild notes (us: M5 Max + rented GPUs)

• Cannot reuse their data — it isn't published. Rebuild from primary sources: ARVO (public, ~6,100 reproducible vulns), OSS-Fuzz, NVD/CVEfixes, plus CVE-Factory for the patch-task recipe. Write our own skill doc (~1–2 pages) — cheapest, highest-leverage component.
• Their code isn't a scaffold either (empty). Use the paper as spec; OpenHands or a minimal ReAct+Docker loop as scaffold.
• Swap the env layer: their sources are memory-safety C/C++; our fleet generalizes the differential-oracle pattern to our own vuln classes — pre/post-patch pairs + programmatic oracle per class. That's the differentiator.
• Compute for ~50k trajectories: generation dominates, not training. Skill-guided teacher at 5×15-min attempts, ~30–50% verify rate → 50k kept needs ≈150–300k rollouts ≈ 40–75 container-hours × 1,000 — run 50–100 parallel sandboxes on a 64–128 vCPU rental (~$1.5–3k) for 2–4 weeks, plus teacher API cost (the real bill; $10–30k frontier API, or self-host a mid-size open model).
• Training: skip 397B full-param on 256 GPUs. Rent 8×H100 for a 14–32B base at 32–64K packed length (131K packing overkill once compaction caps trajectory length) — a few $k for 3 epochs over 50k trajectories; or LoRA a 7–14B on the M5 Max via MLX at short context as a smoke test.

Verdict

1. Copy: differential pre/post-patch oracle as the only success signal; skill-as-teacher-prior distilled by SFT; τ=0.9 structured compaction; easy-instance filtering before synthesis.
2. Avoid: their release discipline (empty repo, unreported data scale, abstract/body drift); single-teacher synthesis; sanitizer-only oracles; no train/eval CVE dedup.
3. Differentiate: actually ship data+code+model with a license; multi-teacher synthesis; extend beyond C/C++ memory bugs via our lab fleet's environments; report per-source yields and token counts.

Primary sources: arXiv abs · v3 HTML · GitHub repo (empty) · HF papers page · ARVO · CVE-Factory

See also: [[bugcrowd-rl-environments]] · [[htb-ai-range]] · [[landscape-2026]]