vulhub-farm — orchestrated vulhub environments with mechanical win conditions

Family of the cyber-factory fleet (../../index.md, ../README.md) that turns the public vulhub corpus (334+ real vulnerable compose environments, repo moved from phith0n/vulhub) into RL-ready exploit tasks with per-run randomized, anti-memorization win conditions. Agents talk the shared JSON-line protocol in ../protocol.md.

Layout

file	role
sync.py	clone/update vendor/vulhub (depth-1, master)
catalog.py	enumerate envs, parse each docker-compose.yml with a stdlib YAML-subset parser (services, images, host ports, env)
tasks.json	30 curated task specs (env id, vuln class, goal, verifier, timeouts, cleanup)
verifiers.py	pluggable win conditions (see below)
runner.py	lifecycle orchestrator + agent subprocess + trajectory snapshot
data/trajectories/<run_id>/	run.json, agent logs, compose ps/logs telemetry (gitignored)

Smoke test

cd fleet/vulhub-farm
python3 -I catalog.py --list            # needs vendor sync for full listing
python3 -I runner.py --list             # curated tasks; works without docker
python3 -I runner.py --smoke fastjson/1.2.47-rce

--smoke validates the task spec + verifier wiring + (if the vulhub checkout exists) parses the real compose file. If docker is present it additionally performs the full lifecycle (up → canary precheck → built-in noop agent → verifier → down → trajectory); without docker it reports SKIPPED for the lifecycle part and exits 0.

Full run

python3 -I sync.py                                              # first time only
python3 -I runner.py --run thinkphp/5-rce \
    --agent "python3 -I /path/to/agent.py" \
    --agent-timeout 600 \
    --callback-host host.docker.internal

Exit codes: 0 win, 1 loss/infra-failure during scoring, 2 usage error. Leftover projects after a crash: python3 -I runner.py --prune.

Verifiers (verifiers.py)

type	win condition	used for
file_read_canary	harness plants a random secret at service:path in the container FS via docker compose exec; agent must exfiltrate it (challenge–response)	RCE / deserialization / file-read envs
command_exec_canary	mode:http: agent is given a per-run nonce and must make it appear at a URL (e.g. echo into webroot via RCE); verifier GETs the path. mode:answer: nonce must come back in the result	upload-bypass envs where uploaded PHP must execute
http_marker	GET a fixed path that only becomes servable post-exploit (PUT/write vulns); content must equal the nonce	tomcat PUT etc.
http_callback	harness serves a per-run listener; the target server must fetch {callback_url}	SSRF + Log4Shell OOB
db_canary	harness plants a random secret row in the env's MySQL/Postgres (credentials resolved from the parsed compose env); agent must read it through the injection	SQLi envs
all	conjunction of the above	multi-step proofs

Anti-cheat notes

Per the ExploitBench pattern (deterministic oracles + per-run randomized challenge–response — see ../../competitors/bugcrowd-rl-environments.md):

1. Randomized per run: FS canaries, DB canary rows, nonces, marker

contents and callback ports/paths are secrets-generated each run — a memorized public write-up scores zero.

1. Baseline prechecks: http_marker/command_exec_canary GET the marker

path before the agent runs; if the marker already resolves, the run aborts env-dirty instead of scoring (defeats pre-seeded answers).

1. No oracle leakage: the task message contains the goal, endpoints, nonce

and callback URL — never the secret. Verifier internals are never serialized to the agent.

1. Isolation: each run is its own compose project (vhfr-<env>-<runid>,

isolated bridge network) and is always torn down with docker compose down -v --remove-orphans (volumes wiped → no cross-run DB state). DB containers are never published to the host except where the upstream compose does (e.g. mysql:3306) — runs are serialized per host (see TODO).

1. Protocol enforcement: agent exit code, timeout kill and the presence of

a final result line all gate the win (fleet/protocol.md); violations are recorded in the trajectory for trajectory-hygiene review.

1. Audit: every run writes run.json with the materialized goal, task

message, verifier verdict, timings and compose logs — reward hacking is reviewable after the fact.

Known limitations

• Two stretch tasks (mysql/CVE-2012-2122, spring/CVE-2022-22978) have

file-read canaries that may exceed what the app surface allows; refine after playtest (flagged in tasks.json notes).

• vulhub compose files bind fixed host ports (some bind :80) → one

environment at a time per host, and privileged ports need root on Linux.

• http_callback needs --callback-host that resolves from containers:

host.docker.internal on Docker Desktop; 172.17.0.1 (or the bridge GW) on Linux.

• Heavy images (weblogic, gogs, gitlab-class) pull multi-GB on first boot.
• The YAML parser is a compose-corpus subset, not general YAML; malformed

upstream files raise ComposeParseError and are skipped by --list.

TODO — scale-out

• [ ] Port-partitioned parallel workers: per-host vulhub checkouts + host-port

ranges (rewrite ports: through a generated override, or front every env with socat/traefik on a dynamic host port), then N concurrent runner.py processes with distinct compose projects.

• [ ] Remote workers over DOCKER_HOST=ssh://…; fleet scheduler ticket in

queue/ with per-host labels (arch, disk for big images).

• [ ] Image pre-pull daemon (docker compose pull sweep) to keep boot under

120s; cache volume for build: envs.

• [ ] Trajectory compaction: data/trajectories/ → parquet/sqlite +

retention policy; dedupe by (env, exploit-family).

• [ ] Curriculum: order tasks into capability ladders (recon → parameter

control → code exec → persistence), ExploitBench-style tiers.

• [ ] Per-env exploit-family tags for red-trains-blue trajectory filtering.