category: competitor-dossier · type: open benchmarks (Purple Llama / CC BY 4.0) · threat: none (eval suites, not products); risk: they define the default "blue eval" and it is static · reuse: malware-analysis + threat-intel data as knowledge items in our blue evals · status: deep-dived · updated: 2026-11-02

CyberSecEval → CyberSOCEval — Meta and CrowdStrike's blue benchmarks, and what they don't measure

1. The lineage

Meta's CyberSecEval suite (all open in meta-llama/PurpleLlama, scores reported in every Llama model card) grew version by version: v1 (arXiv:2402.04644) — insecure coding practices + cyberattack helpfulness; v2 (arXiv:2404.13161) — prompt injection + code-interpreter abuse; v3 (arXiv:2408.03584) — autonomous offensive cyber ops, scaled vulnerability ID, and spear-phishing capability/awareness; v4 (arXiv:2410.07665, Oct 2024) — prompt injection/abuse, code-interpreter abuse, and vulnerability exploitation (models degrade badly on complex exploitation). It is the de-facto industry checklist for "is this model cyber-safe/cyber-capable" — cited across vendor model cards and papers — which matters because whatever it measures becomes what gets optimized.

2. CyberSOCEval (arXiv 2509.20166)

At Fal.Con 2025 (Sep 15, 2025), CrowdStrike and Meta jointly launched CyberSOCEval — the defensive turn: the first open benchmark suite for SOC work, added to CyberSecEval 4. The paper (v1 Sep 24, v2 Nov 10, 2025; Deason et al., Meta + CrowdStrike, senior authors Joshua Saxe & Sven Krasser; Meta publication page) ships exactly two benchmarks:

• Malware Analysis — 609 QA pairs over JSON sandbox-detonation telemetry of real Windows malware run in CrowdStrike Falcon Sandbox: EDR/AV killers, ransomware, RAT-Remcos, infostealers, API-unhooking evasion. Up to 10 options, multiple correct; questions Llama-3.2-90B-generated then expert-edited.
• Threat Intelligence Reasoning — 588 QA pairs from 45 real reports (APT campaign analyses, malware reports, vuln advisories), served as page images (multimodal) or text; up to 6 options, multiple correct; some questions require reading charts.

Scoring: strict all-and-only accuracy plus Jaccard partial credit. Data: CrowdStrike/CyberSOCEval_data (raw CrowdStrike report PDFs + Hybrid-Analysis JSONs; 25 stars, 4 forks, 2 commits — near-zero community uptake); code lives in Purple Llama; paper CC BY 4.0.

Results (OpenAI/Google/Anthropic/Meta/DeepSeek frontier models, default params): malware analysis ~23–34% accuracy (v2) against a ~0.6–4.3% random baseline; TI reasoning ~43–53%. Two findings worth internalizing: (1) reasoning/test-time-scaling models get no step-change here unlike coding/math — they haven't been trained on cybersecurity reasoning (paper); (2) text input beats image input for every vision model. Nothing saturates.

Note on scope claims: launch coverage variously promised "incident response, malware analysis, threat comprehension" (ITBrief); phishing triage and SOC-regulations/REG-style tasks are not in the shipped suite — phishing lives back in CyberSecEval v3, and compliance/regulatory reasoning is simply absent. "Incident response and triage" is listed in the paper as future work. Announced scope > shipped scope.

3. CrowdStrike's Aug 2026 expansion

CrowdStrike's Aug 6, 2026 post (Hines & Midler) is a position piece, not a release: AI security evals should "begin with the defender's operational reality" — post-compromise detection in telemetry, alert triage, investigation, detection engineering, threat hunting, response — plus evaluating AI's ability to emulate adversary tradecraft (e.g., the ALTERED SPIDER profile), grounded in CrowdStrike's trillions of daily events and tailored per customer vertical (finance ≠ healthcare). It openly names the field's diseases: agenda capture, benchmark saturation, data contamination — an incumbent indicting the static-MCQ category it just entered. The promised "principle to practice" demo at Fal.Con 2026 (Sept) materialized as platform AI (SafeMind with NVIDIA, AI-SOC outcomes) — as of Nov 2026, no new public agentic blue benchmark has shipped from CrowdStrike.

4. Gaps (why this doesn't settle the blue side)

• Static and MCQ-heavy: everything is multiple-choice with multiple correct answers over frozen artifacts. No tool use, no queries, no actions — the paper itself concedes MCQ "doesn't perfectly proxy" free-form SOC work.
• No agentic telemetry loop: a SOC analyst's actual job is iteratively querying logs/EDR and revising hypotheses; CyberSOCEval never touches a live system. (Simbian's Cyber Defense Benchmark — agent hunts 26 campaigns in 75k–135k-log telemetry on a 50-query budget; 0/32 models pass — exists precisely because CyberSOCEval doesn't do this.)
• Windows-only malware, file-execution detonations only; TI biased to text over charts; context-window caps force filtering.
• Self-bias: Llama models generated the questions.
• No phishing triage, no REG/compliance reasoning, no IR/triage despite launch-era framing.

5. How our blue evals differ (and must keep differing)

Ours are agentic and closed-loop: the defender agent queries procedurally generated telemetry, and the attack that produced it was itself generated by our red agent (purple loop — nobody has this). Scoring is execution-based (did the agent surface the true scope of the intrusion), not option-matching; environments are context-morphed per run (anti-memorization à la Simbian) and refreshed weekly (the CTFusion contamination lesson); and the eval doubles as an RLVR training environment, which an MCQ suite can never be. CyberSOCEval's artifacts are still useful as a knowledge floor: the 609 detonation QAs and 45-report TI set are exactly the "reading telemetry/reports" skill items to fold into our blue curriculum alongside Primus CTI corpora ([[primus-trend-micro]]).

Verdict

• Copy: real-data grounding (Falcon Sandbox detonations, real APT reports); multimodal TI presentation; publishing both strict and partial-credit metrics; the v3 spear-phishing awareness framing; expert-edited synthetic questions.
• Avoid: MCQ as the substrate for SOC claims; frozen artifacts with no tool loop; Llama-generated questions grading Llama models; announcing scope broader than you ship.
• Differentiate: agentic telemetry-loop blue evals, purple red-generates/blue-hunts design, per-run randomization, RLVR-trainable environments — the Aug 2026 CrowdStrike post is effectively their admission that this is where the field must go; they haven't shipped it, and we can.

Primary sources: arXiv abs · v2 HTML · data repo · PurpleLlama · Aug 2026 post · ITBrief

See also: [[primus-trend-micro]] · [[htb-ai-range]] · [[landscape-2026]] · [[bugcrowd-rl-environments]]