AI Red-Teaming Systems
AI red-teaming architecture: attack generators, target harnesses, severity scoring, CI regression, disclosure and triage, failure modes, and when a suite is not enough.
- Read time
- 16 min
- Level
- Advanced
- Updated
- 2026-08-25
- Components
- Attack generator · Target harness · Severity classifier
An AI red-teaming system is a repeatable pipeline that generates attacks against a model or agent, runs them in a harness that includes the real tools and policies, scores severity, and feeds confirmed failures into a CI regression suite and a disclosure/triage queue. It is how jailbreaks, prompt injection, data exfil, and unsafe tool use get found on purpose — not a one-off brainstorm before launch.
Key takeaways
- 01
Red team the assembled system (prompt, tools, retrieval, guardrails), not the base model in a vacuum; most production failures are integration failures.
- 02
Attacks need a taxonomy and a generator; a shared doc of ten prompts is a workshop, not a system.
- 03
Severity is business impact (exfil, unauthorized write, fraud, safety) plus exploitability, not “the model said something rude.”
- 04
Every confirmed hit becomes a frozen regression case; a fix that is not in CI will regress on the next prompt edit.
- 05
Disclosure and access control on the suite itself matter: attack corpora are dual-use and often contain production-like payloads.
Anatomy of a red-teaming system
Five parts. An attack generator produces cases from a taxonomy: jailbreak/roleplay, direct instruction override, indirect injection in retrieved or page content, data-exfil probes, tool-abuse (unauthorized send, SQL, payment), availability (runaway loops), and policy-edge social engineering. A target harness wraps the production-shaped stack — same tools, same retrieval, same guardrails, credentials replaced by fakes. A severity classifier (rules plus a calibrated judge, plus human review on high impact) labels harm. A regression suite stores confirmed attacks as fixtures. A triage queue assigns owners, SLAs, and disclosure state.
This is complementary to a how-to on running a red team. The system is what remains after the workshop: generators on a schedule, CI on every prompt and tool change, and a place confirmed exploits go to live forever.
Data flow: generate, execute, score, file
A run takes a target build id (prompt hash, model, tool allow-list, retriever version). The generator emits attacks, including mutations of prior hits (paraphrase, encoding, split payloads, HTML comments, retrieved-doc injection). The harness executes with tracing on, tools pointed at sandboxes, and network deny-by-default. Scoring checks for concrete outcomes: did the fake secret leave, did the write tool fire, did the SQL contain a drop, did the policy classifier miss. Human reviewers see the trace for high-severity candidates. Confirmed cases are filed with severity, repro steps, and a minimal fixture.
Indirect injection deserves its own path: plant the attack in a document, ticket, or web page the retriever or computer-use agent will load, then run a benign user query. If you only send jailbreaks as the user message, you will miss the failure mode that actually ships.
- Pin the target build; attacks without a build id are anecdotes
- Mutate prior hits; attackers paraphrase, they do not invent from scratch
- Execute in a harness with fake secrets and sandboxed tools
- Score outcomes, not vibes (exfil, write, policy miss)
- File confirmed hits as CI fixtures with severity and owner
Severity, CI, and the fix loop
Severity follows impact: unauthorized external send or payment, PII/secret exfil, instruction to self-harm or crime, silent ACL bypass, then availability and brand-tone issues. A sassy refusal is not Sev-1. Exploitability (single shot vs multi-turn, requires insider document vs public user message) modifies priority. The classifier suggests; humans own Sev-1/2.
CI runs the frozen suite on every change to prompts, tools, retrieval, or guardrails. Hard-fail on Sev-1/2 repros; report rates on the long tail. Fixes belong in policy, tool allow-lists, and trust boundaries first — dual-channel prompts, treating retrieved text as data, confirmation on writes — and in model choice second. A prompt bandage with no fixture is how the same jailbreak returns next month.
Failure modes
Testing the chat model without tools and then being surprised in production. Scoring “did the judge think this was bad” instead of “did the tool fire.” Overfitting the suite to last quarter’s jailbreak memes while missing indirect injection. A generator that never mutates confirmed hits. Guardrails in the harness that are stricter than production, so the suite is green and the live agent is not. No budget cap, so red-team jobs are a token bill with no new unique hits.
Process failures: attack datasets in an open bucket, no owner on Sev-1, and “we red-teamed at launch” with no CI. Treat the corpus as sensitive. Rotate any real secrets that ever entered a payload, even in a lab.
- Harness that does not match production tools and policies
- Judge-only scoring without outcome checks
- Stale taxonomy; no indirect-injection path
- Green CI because the harness is stricter than prod
- Attack corpus leaked or unowned Sev-1s
Measuring the red team itself
Measure coverage (taxonomy cells hit per run), unique confirmed findings, time-to-file, time-to-fix, repro rate of old fixtures, and false-positive rate of the severity classifier against human labels. A generator that produces 10,000 near-duplicates of one jailbreak is not covering the map. Track findings by surface: user message, retrieved doc, tool result, uploaded file, GUI content for computer-use agents.
Calibrate the attack judge the same way you calibrate any LLM judge: binary rubrics, frozen version, agreement with humans. Do not let the red-team judge be an unversioned cousin of the product judge; they will drift in opposite directions and both look successful.
When a red-teaming system is not the next step
If you have no golden evals, no tool allow-list, and no trace store, build those first — otherwise you cannot tell a real hit from a flaky demo. If the product cannot yet sandbox writes, red teaming will only confirm that writes are unconstrained. A one-week manual exercise is still the right start for a small copilot with no tools; the system described here is for agents that call tools, retrieve untrusted text, or act in a UI.
Do not treat a vendor model card as a substitute. Your prompt, your tools, and your corpus change the attack surface. Do not publish exploit payloads as marketing. File, fix, regress, and disclose internally (and externally when you have a policy for it). The value is the loop, not the wall of shame.