category: data · status: draft · updated: 2026-11-02

Dataset Lineage

There are 1,205+ cyber datasets on Hugging Face. Sounds like abundance. In practice it's a photo-copier room: most of those datasets are photocopies of photocopies of a handful of originals. Lineage is the family tree of a dataset — which root corpus it was sliced, filtered, translated, or distilled from — and lineage clustering is how you compute that tree instead of trusting dataset cards.

Get this wrong and you "train on 30 datasets" that are secretly one dataset with 40% wrong labels, repeated 30 times.

Where the roots come from

Sweep the HF cyber category and nearly everything falls into a few root families:

• Commit-diff mined vulnerability data. BigVul (MSR 2020) mined 3,754 vulnerabilities from GitHub commits reached via CVE records; DiverseVul re-mined the same idea at scale (18,945 vulnerable functions from 7,514 commits). Dozens of downstream datasets are re-slices of these functions — different CWE filters, different splits, same commits.
• ATT&CK dumps. MITRE ATT&CC technique tables dumped as instruction/QA pairs, remixed across many dataset IDs.
• GPT-distill templates. Datasets made by prompting a frontier model — "write a penetration-testing question with an answer" — thousands of times. They share phrasings, structure, and errors like siblings share a face. CyberPal-style and most "cyber chat" SFT sets live here.
• CTF writeup scrapes. Blogs and repo writeups converted to instruction data — which is also our contamination problem wearing a different hat ([[contamination-and-decontamination]]).

The kicker: label noise propagates down the tree. Commit-diff mining labels a function "vulnerable" because the commit fixed a CVE — but commits touch clean code too. SecLLMHolmes ("LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities," arXiv:2312.12575) manually audited a DiverseVul sample and found roughly 44% of "vulnerable"-labeled functions were not actually vulnerable. PrimeVul (arXiv:2403.18624) re-audited the whole lineage — BigVul and DiverseVul included — and showed the noise plus duplication plus flattering metrics had the field training and scoring on sand. (Our LICENSE_MANIFEST.md carries this as the ~40% rule: relabel before SFT.) Every derivative dataset inherits those wrong labels. Training on ten derivatives doesn't average the noise away — it amplifies it, because the same mislabeled function gets ten votes.

How lineage clustering works

Same machinery as decontamination, pointed inward at our own corpus:

1,205 datasets ──► MinHash/LSH ──► candidate pairs ──► embeddings ──► cluster ──► pick root
                  (shingle        (near-identical      (semantic      (connected     per cluster,
                   signatures;     rows across          neighbors —     components     license +
                   cheap, O(N))     datasets)            catches         / community     noise audit
                                                       paraphrase,       detection)      once)
                                                       translation)

1. MinHash + LSH over row shingles. Hash every row of every dataset; locally-sensitive hashing makes near-duplicate rows collide. This cross-links datasets that share rows even when they claim different provenance — the empirical family tree, as opposed to the README's story.
2. Embedding neighbors. Rows that paraphrase, translate, or re-format each other don't collide in MinHash but sit close in embedding space. This catches the GPT-distill siblings and lightly-edited derivatives.
3. Clustering. Union the two similarity graphs into clusters; each cluster is one lineage with (usually) one true root.
4. Per-cluster verdicts, made once. License class (a CLEAN root can have NC remixes — taint flows down), relabel-or-drop, dedupe to canonical rows.

Why it collapses effort

Concretely, for Stream A's triage:

• 1,205 datasets → dozens of real roots. You don't deep-dive 1,205 datasets; you audit ~40 lineages and inherit the verdicts. A dataset's tier is its root's tier until proven otherwise.
• Fix labels once, at the root. Relabeling is the expensive step (teacher-model or execution time). Relabel BigVul-lineage rows once, and every derivative is fixed for free. Relabel per-leaf and you pay 12 times to fix the same wrong functions.
• Kills phantom evals. If dataset A (train) and dataset B (eval) are BigVul derivatives, your eval is train-set leakage with extra steps. Lineage clustering catches what "different name, different card" hides.
• License hygiene. Our PERSONAL/BANNED rules apply to whole lineages: one PortSwigger-derived root taints every remix (LICENSE_MANIFEST).

Worked example

Stream A pulls three plausible SFT sets: a "vuln-detect-100k", a "secure-code-instruct", and a "CVE-functions-cleaned". MinHash says 78% of rows in the first two collide pairwise; embeddings tie the third in with near-duplicate paraphrases. Cluster of 3 → one lineage → root identified as DiverseVul. Manifest check: CLEAN*. Noise check: audit a 200-row sample against fix commits; ~40% wrong labels confirmed. Decision: dedupe to canonical rows, execution-relabel what we can (does the function actually change sanitizer behavior in pre/post builds — see [[differential-oracles-and-win-conditions]]), drop the rest. Three dataset cards, one audit, one bill.

That's the whole point: lineage turns a data lake into a family tree, and you only have to vet the grandparents.

Why it matters here

Our moat is environments + verified trajectories, not scraped corpora — but L0/L1 SFT still starts from the HF sweep. Lineage collapse is what makes Stream A a 40-audit job instead of a 1,205-audit job, and it's the first line of defense against both label noise and accidental train/eval incest.

See also

[[contamination-and-decontamination]] · [[differential-oracles-and-win-conditions]] · [[landscape-2026]] · [[cyberfactory-openaegis]]