custom_suites — randomized prompt-injection scenario generator

Extension stub for generating fresh, randomized injection scenarios per run — the gym's growth path from "fixed benchmark" to "procedural curriculum" (factory doctrine: procedural randomized environments + per-run secrets so memorized writeups don't transfer).

What's here

counter_suite.py     worked example: a complete custom task suite (tools,
                     environment, user + injection tasks, injection vector)
counter_data/        suite data files agentdojo expects:
                       environment.yaml      (default env, with {placeholders})
                       injection_vectors.yaml (vector ids + benign defaults)
payload_bank.py      per-run secrets + injection payload bank (5 families),
                     seeded option for reproducible trajectories

Authoring recipe (verified against agentdojo==0.1.35)

1. Tools are plain functions. Their docstrings ARE the tool schema

(document every arg: :param n: ...), and Annotated[T, Depends(...)] parameters are injected from the environment:

```python def increment(env: Annotated[CounterEnvironment, Depends(lambda e: e)], n: int) -> None: """Increment the counter by the given amount.

:param n: the amount to add to the counter. """ env.counter += n ```

1. Suite + registration (importing the module registers it):

``python task_suite = TaskSuite("counter", CounterEnvironment, [make_function(t) for t in TOOLS], DATA_DIR) register_suite(task_suite, "v1") ``

1. User tasks (@task_suite.register_user_task): PROMPT,

ground_truth(pre_env) -> list[FunctionCall], and utility(model_output, pre_env, post_env) -> bool (state diff).

1. Injection tasks (@task_suite.register_injection_task): GOAL,

ground_truth(pre_env). Polarity gotcha (verified in all built-in suites): security(...) must return True when the injection goal WAS achieved — it is a goal-condition check, not a safety verdict. The benchmark's security_results therefore read as "attack landed" and our telemetry computes ASR = mean(security_results).

1. Injection surface: put a {placeholder} in environment.yaml and

declare it in injection_vectors.yaml. Attacks (ImportantInstructionsAttack, ...) replace the default text wherever the ground-truth tool flow surfaces it to the agent.

Randomizing per run

payload_bank.manifest(seed=None, goal=...) draws fresh secrets and renders the payload bank; splice the drawn values into your suite's injection_vectors.yaml default (or the GOAL) before get_suite(...), and archive the manifest next to the trajectory. --seed reproduces a run exactly — same doctrine as the EVM asserts' per-run challenge values.

Next steps (deliberately stubbed)

• generator that composes suite templates × payload bank × per-run secrets

into N distinct scenarios per epoch (hook: manifest() output → suite fixtures);

• counter-suite generalization: more environments + multi-vector tasks;
• defense side: wrap PromptInjectionDetector variants as pipeline defenses

and measure utility/ASR trade-off curves.