Continued pre-training (CPT) means taking a base model that already finished general pre-training and training it more on your domain's raw text — exploit-db entries, Sigma rules, kernel source, CTF writeups, CTI reports. It injects vocabulary, formats, and background knowledge that instruction tuning can't, because there are no instructions — just text. The catch is as old as transfer learning itself: specialize too hard and the model forgets its general abilities. Every decision in this explainer is about buying domain skill without paying the forgetting tax.
CPT works, at modest scale. Primus (EMNLP 2025) is the cleanest public datapoint: continued pre-training of Llama-3.1-8B on ~2.8B tokens of curated security text lifted the aggregate cyber score +15.9%; on a 70B base, 7.6B tokens of filtered cyber corpus gave +11.2%. Meanwhile general capability barely moved — MMLU drifted from 67.56% to 66.29%, about a point. So billions of tokens of domain CPT is roughly a +10–15% domain gain for ~1 point of general loss when the corpus is curated and the model is big enough to absorb it.
Forgetting is real and replay is the brake. Train on a narrow distribution for billions of tokens and gradient updates overwrite general knowledge — outputs go domain-flavored, reasoning benchmarks sag. The standard mitigation is replay: mix a fraction of general + code data into the CPT corpus. We budget 20–30% general/code replay as our starting band (industry practice clusters there; it's a starting point to fit, not gospel). Below ~20% you start leaking MMLU; above ~30% you're mostly not doing CPT anymore.
LoRA learns less and forgets less. Biderman et al., TMLR 2024 ran exactly our regime — LoRA vs full fine-tuning on ~10–20B tokens of code/math CPT — and found a stable tradeoff: LoRA underperforms full FT on target-domain learning, but retains notably more of the original model's capabilities; full FT learns more and damages more. Rank and target-module choices only partially close the learning gap. LoRA also has an operational bonus for us: it's cheap, resumable, and merges back into the base with zero decode-time cost — so the 200 t/s M5 Max decode floor from CyberTiel-class MTP builds survives.
The mix is predictable without the big run. Data mixing laws (Ye et al., ICLR 2025) show that loss as a function of mixture proportions follows fittable functional forms: nest a step-scaling law, a model-size law, and a mixing law, fit all three on small proxy runs, and you can predict — and then optimize — the large run's mixture before spending on it. Apple's follow-up confirms the extrapolation holds. This is how we pick our cyber:general:code ratio empirically instead of by folklore.
| Full-param CPT | LoRA-CPT | |
|---|---|---|
| Domain gain per token | higher | lower (learning gap) |
| Forgetting | worse | better (retention) |
| Cost on 27B | big multi-GPU rental, full checkpoint each time | small rental or even M5 Max MLX, adapter per experiment |
| Reversibility | no (new weights) | yes (drop adapter) |
| Decode speed after merge | same | same |
LoRA-CPT, 3–5B tokens, only on base-model jumps, always gated. Concretely:
Suppose the next Qwen base lands and we green-light a CPT event for our 27B build. Corpus target: 4B tokens = 3B cyber (exploit/advisory/Sigma-auditd-Sysmon-telemetry/tool-source mix from Stream A's tier-1 datasets, license-screened) + 1B replay (≈600M general web + 400M code). First, five proxy runs at ~200M tokens each sweeping replay 15/20/25/30/35%; fit the mixing law, find predicted loss-optimal at 24% replay with MMLU constraint satisfied at 26% — take 26%. Then one LoRA-CPT run on rented 8×H100, gate on MMLU (−1.4 observed, passes), MT-Bench (−0.05, passes), and harness scores (ExploitBench Tier-1/2 rates up). Merge the adapter, tag the checkpoint, and the weekly SFT/RL loop resumes on the new floor. Total CPT cost: a few thousand dollars and two weeks — because it's LoRA-sized and proxy-optimized, not a 256-GPU re-pretrain like the CyberFactory paper's 397B build ([[cyberfactory-openaegis]]).
[[purple-loop]] · [[capability-ladders]] · [[cyberfactory-openaegis]] · [[landscape-2026]] · [[index]]