category: competitor-dossier · type: government capability report (market signal, not a competitor) · threat: n/a — validates our base-model and teacher strategy · reuse: teacher selection; rollout-server budget and economics design · status: deep-dived · updated: 2026-11-02
UK AISI, Jul 2026: open-weight cyber capability — GLM-5.2 is the open frontier
1. The report
"How Far Behind the Frontier are Leading Open Weight Models on Cyber?" — UK AI Security Institute, July 17, 2026. The question underneath our whole plan: are open weights good enough to build the cyber copilot on, or do we need frontier APIs? AISI's answer: good enough, with a 4–7 month lag.
2. Methodology
Two eval classes, models run off-the-shelf with no special elicitation or optimization (which they flag as a bias toward underestimating open models):
- Narrow cyber tasks: a 70-task subset across four difficulty levels (technical non-expert → expert), spanning vulnerability research, reverse engineering, web exploitation, and cryptography; 5 attempts, 2.5M-token cap each.
- Cyber ranges: expert-built simulated networks for autonomous long-horizon attack — flagship "The Last Ones": a 32-step corporate attack across ~20 hosts, estimated ~20 hours of expert human work, run under a 100M-token budget.
- Models: GLM-5.2 (June 2026), DeepSeek V4-Pro, with Kimi K3 queued for when weights dropped (late July 2026); frontier comparators included Opus 4.5/4.6, GPT-5.3-Codex, Sonnet 4.5. AISI also folds in internal 2025 testing of earlier open models for the trend line.
3. Findings
| Comparison | Result |
|---|
| GLM-5.2 | Most cyber-capable open model tested. Narrow tasks: matches Opus 4.6 and GPT-5.3-Codex (~4-month gap). Ranges: reaches Opus 4.5 level (~7-month gap) |
| DeepSeek V4-Pro | Matches Opus 4.5 on narrow tasks (~5-month gap); falls below Sonnet 4.5 on The Last Ones |
| Trend | Open-weight lag shrank from 6–10 months (2025 internal testing) to 4–7 months |
| Safeguards | V4-Pro occasionally refused reverse-engineering tasks; repeat attempts bypassed it |
| Cost per 100M-token range run | Opus ~$85 · GLM-5.2 ~$46 · V4-Pro ~$1.19 |
4. What it means for base-model choice
We train Qwen-class small models as the copilot; the report doesn't change the student, it settles the teacher: GLM-5.2 is the strongest open base at acceptable cost — and it is exactly the teacher CyberFactory used for trajectory synthesis ([[cyberfactory-openaegis]]), so two independent lines now converge on it. Concretely: (1) open-base capability is already frontier-adjacent, so distilling an open teacher loses little vs. paying for Opus-class APIs; (2) V4-Pro's ~$1.19/100M-token economics make it the bulk-rollout workhorse where raw persistence matters more than peak skill; GLM-5.2 at ~$46/100M is the quality tier for hard instances — a two-tier teacher pool; (3) the 4–7-month lag is closing (6–10 → 4–7 within a year), so a teacher chosen today depreciates slowly; re-benchmark each open release (XBOW's end-to-end model evals are the best template for doing this ourselves on validated-findings criteria, not MCQ).
5. Limitations (read before over-trusting the numbers)
- No elicitation — deliberately generic prompting/scaffolds; agentic skill is highly scaffold-sensitive, so the true open-model ceiling is likely higher than reported.
- Ranges are weak evidence: few scenarios, and trajectories can't separate cyber-skill limits from agentic/planning limits — a model can fail The Last Ones for planning reasons that a better harness (or skill docs, cf. CyberFactory's teacher skill guidance) would fix.
- Contamination isn't addressed in the report. Narrow-task evals of open models trained on public CTF/write-up-adjacent corpora are partially memorization; the range results are the trustworthy half. For teacher selection, weight novel-procedure performance over public-benchmark scores.
- Capability ≠ deployability: AISI's framing — once weights are open, deployment-time controls are permanently gone. For a local copilot that's the feature, not the bug; but it means the refusal/safety layer is entirely ours to build and test (see Primus's measured Garak regressions after security CPT — [[primus-trend-micro]]).
- Cyber-only scope; no transfer conclusions to other capabilities.
6. Implications for our teacher / rollout-server design
- Teacher tiering: GLM-5.2 (quality) + V4-Pro (volume) behind one router; upgrade path = "whatever AISI's next report crowns," re-ranked on our own XBOW-style validated-findings eval.
- Budget your rollouts like AISI budgets evals: per-attempt caps (~2.5M tokens narrow, 5 attempts) as default trajectory budgets; long-horizon jobs get a 100M-token-class ceiling — which at V4-Pro pricing is ~$1.19 of API, i.e., the constraint is wall-clock and context management, not money. That makes context compaction (CyberFactory's τ=0.9 structured rewrite) and checkpoint/resume the load-bearing rollout-server features, plus retry-on-refusal (AISI bypassed V4-Pro refusals by simply re-asking — build that in, with logging).
- Self-host optionality: a 100M-token run for dollars means renting GPUs for a self-hosted open teacher is defensible the moment API pricing or ToS shifts; keep the teacher interface model-agnostic.
- Eval hygiene for us: adopt their two-tier shape (narrow + long-horizon range) as our internal acceptance test for each teacher/student swap, but with per-run procedural randomization to patch the contamination hole they leave open.
Verdict
- Adopt: GLM-5.2 as primary teacher (corroborated by CyberFactory's independent choice); V4-Pro as cheap bulk-rollout engine; narrow+range two-tier eval design; 5-attempt/2.5M-token micro-budgets.
- Caveat: treat open-model narrow-benchmark scores as contamination-suspect; trust novel long-horizon procedures; re-crown the teacher on our own agentic eval, not public leaderboards.
- Act on: rollout server = compaction + checkpointing + retry-on-refusal + model-agnostic router; safety layer is ours to own because open weights ship without one.
Primary source: AISI blog (Jul 17, 2026) · context: [[landscape-2026]] §6 (first sighting) · AISI's wider eval tooling lineage includes ControlArena/BashArena ([[bugcrowd-rl-environments]] for the oracle-rigor parallel)
See also: [[cyberfactory-openaegis]] · [[landscape-2026]] · [[primus-trend-micro]] · [[xbow]]