AI Red Teaming Guide: Planning, Running, and Learning From Exercises

Evaluation tells you how a system performs on the cases you thought of. Penetration testing tells you whether defined controls hold. Red teaming tells you what happens when creative, persistent adversaries try to make the system cause harm in ways nobody anticipated. It is where the surprising failures surface: the multi-turn manipulation that unlocks a tool, the innocuous document that rewrites an agent's goal, the customer phrasing that produces discriminatory output. This guide covers how to plan, run, and learn from red team exercises, drawing on FISTA Solutions' AI agents practice. The concept is in what is ai red teaming and the scoped complement in ai penetration testing. Exercises must be authorized and conducted against systems the organization owns or has permission to test.

How does red teaming fit with other testing?

Method	Asks	Cadence
Evaluation	Does the system meet thresholds on known cases?	Every change, in CI
Penetration testing	Do defined controls hold against known attack classes?	Before launch; after material change
Red teaming	What can creative adversaries make it do that we did not anticipate?	Before launch for exposed systems; periodically
Automated adversarial suites	Do past findings stay fixed?	Every change

Evaluation practice is in the ai evaluation checklist.

How do you set objectives and rules?

Define the harms in scope: data exposure, unauthorized actions, harmful or discriminatory content, policy bypass, fraud enablement, cost amplification, reputational outputs. Define what is out of scope and the rules of engagement: environments, data, side effects, provider terms, and stop conditions. Name the decision owner for findings. An exercise without objectives produces anecdotes. Governance context is in the agentic AI governance whitepaper.

Who should be on the team?

Security specialists with hands-on LLM attack experience; domain experts who recognize harmful outputs in context, such as a clinician for a healthcare assistant or a lender for a credit assistant; safety and policy people who define unacceptable behavior; and people who think like misusing users. Automated tooling generates attack variants at scale, and the humans direct it toward what matters. Independence from the builders is essential. Security staffing is in hire ai security engineers.

What should campaigns cover?

• Injection: instructions through user input and through content the system reads.
• Jailbreaks: bypassing safety behavior through framing, role play, encoding, and persistence.
• Agent manipulation: steering tool use, escalating privileges, bypassing gates, altering goals.
• Extraction: pulling retrieved documents, memory, system prompts, and secrets into outputs.
• Harmful content: discriminatory, defamatory, unsafe, or misleading outputs in domain context.
• Multi-turn social engineering: building context over turns to unlock behavior.
• Abuse: using the system for fraud, spam, or cost amplification.

Attack taxonomies are in ai agent security risks and what is jailbreaking in ai.

How do you run the exercise?

Brief the team on the system, its controls, and the objectives; allocate time across campaigns with freedom to follow leads; record every attempt with inputs, outputs, and context; hold daily syncs to share techniques; escalate critical findings immediately rather than waiting for the report; and close with a debrief that includes the builders. Exercises typically run days to weeks depending on scope and tier.

How are findings scored and reported?

By realistic impact and ease: the harm produced, who is affected, how likely a real adversary is to find the path, whether existing controls contain it, and reproducibility. The report contains objectives, methodology, findings with reproduction, impact ratings, affected components, recommended controls, and the automated cases to add. Findings map to the risk register. Register practice is in ai risk register.

How do findings become controls and tests?

Each finding gets a design-level remediation: structural separation of instructions from content, permission enforcement outside the model, output and action validation, tool scoping and gates, redaction, and safety behavior tuned and evaluated. Each reproducible attack becomes an automated case in the adversarial suite so it cannot regress. Prompt-only fixes are recorded as temporary. Defenses are in the prompt injection defense checklist and validation in llm output validation.

What cadence is appropriate?

Before launch for systems with customer exposure, sensitive data, or actions; after material changes including provider model updates, which can reopen closed findings; periodically by risk tier; and continuously through automated suites. Organizations with many systems maintain an internal capability and bring external teams in periodically. Change control is in how to manage ai vendors.

What mistakes weaken red teaming?

No objectives, so the exercise wanders; teams without domain expertise, so harmful outputs are not recognized; testing against sandboxes without production controls; findings rated by model misbehavior rather than harm; remediation by prompt patch; no conversion to automated tests; and a single exercise treated as permanent assurance.

What does a sound exercise look like?

A healthcare organization red teams a patient-facing scheduling and information assistant before launch. Objectives: no clinical advice, no cross-patient data, no unauthorized appointment changes. A team of a security specialist, a nurse, a privacy officer, and automated tooling runs campaigns for a week. Findings: a multi-turn path to clinical guidance rated high, a cross-patient retrieval gap rated critical, and several low items. Remediation fixes permission sync, adds output validation for clinical content, and encodes both paths in CI. A retest confirms, and the exercise repeats after the next provider model update. The safety context is in the AI safety in healthcare operations whitepaper.

How FISTA Solutions runs AI red teaming

FISTA Solutions plans red team exercises with client objectives and rules, assembles mixed security and domain teams with automated tooling, runs campaigns against systems in mirrored environments, scores findings by realistic impact, remediates through design, and converts findings into continuous adversarial suites. The AI agents practice delivers secured systems, AI enablement provides the testing platform, and forward deployed engineers embed with client security and safety teams. The record behind the approach is 150+ projects with 99.9% uptime.

To find the failures your evaluation did not anticipate, message FISTA on WhatsApp, or read what is ai red teaming for the concept behind the practice.