awesomechecklists
llms.txtGitHub ↗
/

16 — AI Red Teaming and Guardrails Checklist#

Verify that AI-enabled products have risk-based adversarial coverage and enforceable defenses against harmful outputs, unauthorized actions, data disclosure, and abuse. Red teaming seeks to discover failures; passing a finite test suite does not prove a model or product cannot be bypassed.

Scope: AI-specific. This file is one part of the 16_AI_Red_Teaming_and_Guardrails.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

  • Project / product:
  • Release / version / environment:
  • Scope and applicable jurisdictions / markets:
  • AI capabilities, model/provider, tools, data sources, and protected assets:
  • Authorized red-team scope / test environment / contact:
  • Policy / guardrail versions and risk/incident owners:
  • Reviewer / owner:
  • Review date:
  • Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

  1. Copy this shared template to project-specific review records; keep every checkbox in the repository unchecked.
  2. Scope adversarial testing to systems, models, datasets, and endpoints the project owns or is authorized to test. Use synthetic identities, dummy data, sandboxed tools, and safe endpoints; do not attack unrelated systems or disclose real secrets.
  3. Mark [x] only after verifying a check through an approved threat model, model/tool permission test, adversarial result, policy decision, safety drill, or production-equivalent guardrail validation. Track evidence, defects, accepted exceptions, and justified N/A in project-specific notes/issues keyed by RED ID, not in this template.
  4. Separate security objective, actual observed violation, blocked attempt, inconclusive result, and benign false positive. Do not credit a model refusal if unauthorized tool effects or external disclosure already occurred.
  5. Evaluate ## API Checks and ## Frontend Checks independently where those surfaces exist; document N/A outside the template for systems without them.
  6. Repeat targeted adversarial and benign evaluation after changes to the model, prompts, retrieved data, tool scopes, guardrails, modalities, or application policy. Review general AI governance and detailed agent, RAG, and regression requirements in their other checklists.

Threat modeling and abuse cases#

  • RED-001 Define the AI product's intended purpose, users, allowed content, model authority, external dependencies, and highest-consequence decisions.
  • RED-002 Assign named security, product, safety, privacy, and operational owners to the threat-model review and risk acceptance.
  • RED-003 Inventory models, prompts, retrieval collections, fine-tuning sources, guardrail services, tools, agents, and data-processing providers.
  • RED-004 Identify all trusted and untrusted entry points, including user messages, files, retrieved documents, websites, tool responses, and conversation memory.
  • RED-005 Map where content crosses boundaries between untrusted data, privileged instructions, authenticated identities, and executable tool actions.
  • RED-006 Identify sensitive assets such as secrets, protected customer data, proprietary prompts, privileged workflows, and financial operations.
  • RED-007 Enumerate plausible threats to confidentiality, integrity, availability, user safety, and trustworthy product decisions.
  • RED-008 Classify direct and indirect prompt injection, malicious context, forged tool messages, and instruction laundering as separate test surfaces.
  • RED-009 Model data poisoning in retrieval sources, memory, feedback loops, model artifacts, and other relevant supply-chain inputs.
  • RED-010 Identify excessive agency and privilege-escalation paths from model-suggested tool calls or delegated credentials.
  • RED-011 Consider cross-user, cross-tenant, and cross-session access and leakage via prompts, caches, retrieval, or persistent state.
  • RED-012 Identify risks from unsafe output handling, including generated markup, code, SQL, shell commands, and network destinations.
  • RED-013 Model misinformation, unsupported factual claims, harmful advice, and overconfident outputs where consequential to users.
  • RED-014 Assess possible discrimination, harassment, self-harm, or other safety policy failures within the application's allowed use case.
  • RED-015 Account for misuse of multimodal inputs, hidden document layers, captions, metadata, and mixed-script content when supported.
  • RED-016 Identify resource-exhaustion, denial-of-wallet, rate-limit evasion, and unbounded inference or tool-loop risks.
  • RED-017 Map attacker capabilities, reachable surfaces, likely preconditions, and expected impact without claiming the test set is exhaustive.
  • RED-018 Prioritize attack paths by severity, exposure, exploitability, and affected people, not just frequency in a benchmark.
  • RED-019 Define which unacceptable outcomes automatically block a release and what residual risks require human authorization.
  • RED-020 Identify tests that require specialist legal, privacy, child-safety, security, or domain expertise and obtain qualified review.
  • RED-021 Record authorized red-team scope, permitted environments, test identities, prohibited targets, and the response contact for exercises.
  • RED-022 Revisit the threat model when model versions, tool permissions, retrieval sources, markets, modalities, or guardrail configurations change.

Adversarial evaluation#

  • RED-023 Develop an adversarial test plan linked to the threat model, defined security objectives, and observable failure outcomes.
  • RED-024 Include a realistic benign control set so aggressive refusals are not mistaken for successful safeguards.
  • RED-025 Test direct attacks through authorized user input and indirect attacks through each actually supported untrusted content channel.
  • RED-026 Use sandboxed tools, dummy accounts, and synthetic secrets so testing cannot trigger live harmful or unauthorized side effects.
  • RED-027 Test prompt-injection attempts that impersonate trusted roles or try to override the application's instruction hierarchy.
  • RED-028 Test malicious directions hidden within retrieved text, file contents, tool responses, and formatted documents where relevant.
  • RED-029 Include encoding, obfuscation, Unicode confusables, and multilingual or code-switched attempts suitable for supported inputs.
  • RED-030 Test attacks that ask for internal instructions, authentication tokens, private user histories, or unauthorized documents using dummy values.
  • RED-031 Check whether an injection can change tool selection, arguments, permissions, destinations, or human-approval requirements.
  • RED-032 Test attempts to exploit multi-agent handoffs, shared memories, resumed sessions, and stale approval state where supported.
  • RED-033 Probe output handling with harmless simulated markup, code, commands, URLs, and structured payloads that must remain untrusted.
  • RED-034 Evaluate ingestion or retrieval poisoning with a controlled, reversible test corpus and independent source-access checks.
  • RED-035 Test whether the system refuses unsafe or disallowed tasks while still completing permitted tasks in the same domain.
  • RED-036 Include ambiguous, contradictory, boundary-condition, and missing-context prompts to evaluate safe clarification or abstention.
  • RED-037 Test long-context crowding, repeated turns, partial tool failures, and adversarially timed input within authorized limits.
  • RED-038 Exercise multimodal attack surfaces using synthetic test images, audio, or files where those modalities exist.
  • RED-039 Measure actual unauthorized actions, data disclosure, policy violations, and invalid outputs rather than relying only on refusal wording.
  • RED-040 Distinguish blocked attacks, successful attacks, inconclusive cases, and test-environment failures in results.
  • RED-041 Record attack identifier, intended violation, input channel, expected behavior, and observed outcome for every test case.
  • RED-042 Version the model, prompt, guardrails, tools, corpus, dataset, decoding settings, and test harness for each evaluation.
  • RED-043 Repeat nondeterministic cases within approved budgets to measure variability and avoid conclusions from a single passing sample.
  • RED-044 Report severity-weighted failure counts and attack success with explicit denominators, sampling limitations, and uncertainty where justified.
  • RED-045 Measure legitimate-request false-positive blocks, false-negative harmful passes, latency, and user-task completion together.
  • RED-046 Review adversarial case quality and automated judge decisions against competent human assessments on a representative sample.
  • RED-047 Retest fixed issues with regression cases and confirm remediations do not merely suppress visible error messages.
  • RED-048 Run targeted tests after each material change to the model, prompt, retrieval, tool permissions, guardrails, or user workflow.

Content and tool guardrails#

  • RED-049 Define a written policy for allowed, restricted, disallowed, and escalated content and actions that matches the deployed use case.
  • RED-050 Separate input screening, model response checking, tool-call authorization, and final user-visible output controls by their trust boundaries.
  • RED-051 Select moderation or classification tools suited to the supported modalities and documented policy categories.
  • RED-052 Validate guardrail accuracy with domain-specific allowed and prohibited examples, including ambiguous and long-tail cases.
  • RED-053 Measure classifier performance by language, modality, category, and relevant user context rather than assuming one global threshold is sufficient.
  • RED-054 Document moderation thresholds, manual-review routes, and the consequences of false positives and false negatives.
  • RED-055 Apply appropriate screening to retrieved documents and untrusted tool output in addition to the initial user request.
  • RED-056 Do not rely on prompt wording, delimiter syntax, regular expressions, or classifier verdicts as the sole authorization boundary.
  • RED-057 Enforce user, tenant, role, scope, resource, and destination permissions in trusted execution code for every consequential tool call.
  • RED-058 Validate model-generated tool names and arguments against allowlists, schemas, type constraints, and business invariants.
  • RED-059 Require action-specific authenticated human approval for risk-defined irreversible or high-impact tool actions before execution.
  • RED-060 Ensure guardrails that must block side effects run before those effects, including through handoffs, nested agents, and resumed tasks.
  • RED-061 Check the actual runtime's coverage of agent-level, tool-level, hosted-tool, and delegated-agent guardrail hooks.
  • RED-062 Choose fail-safe behavior for guardrail timeout, service outage, malformed verdict, or missing required classification.
  • RED-063 Test whether disabling or bypassing a guardrail causes the appropriate service degradation or blocks privileged actions.
  • RED-064 Prevent generated HTML, Markdown, scripts, SQL, and URLs from being used in sensitive downstream sinks without context-appropriate validation.
  • RED-065 Restrict code execution, filesystem access, browsing, and network egress to permitted sandboxes and destinations.
  • RED-066 Protect secrets and private data through scoped credentials and data minimization, not concealment in system prompts.
  • RED-067 Apply safe storage and rendering rules to uploaded and model-generated text, images, audio, and files where applicable.
  • RED-068 Check that policy changes and moderation model updates are versioned, reviewed, and tested for regressions.
  • RED-069 Limit input/output size, tokens, calls, retries, recursion depth, concurrency, and spend independently of model requests.
  • RED-070 Ensure safety filters do not silently erase important evidence needed for authorized incident investigation or escalation.
  • RED-071 Record guardrail decisions and tool authorization outcomes without retaining unnecessary sensitive full prompts or outputs.
  • RED-072 Test that user permissions and policy changes take effect in caches, sessions, agents, and pending tool executions.
  • RED-073 Define clear and auditable exception handling for allowed research, security testing, or regulated domain workflows.
  • RED-074 Reassess the entire guardrail chain after adding a new model, modality, connector, output channel, or privileged capability.

Safety response and monitoring#

  • RED-075 Define severity levels, containment actions, decision owners, and escalation timelines for AI-specific security and safety incidents.
  • RED-076 Instrument relevant safety events such as policy blocks, injection attempts, tool denials, data exposure indicators, and unsafe output reports.
  • RED-077 Track harmful-output passes and benign-task blocks with separate metrics to avoid optimizing only for refusals.
  • RED-078 Monitor changes in moderation results, refusal categories, policy exceptions, and observed failure rates across releases.
  • RED-079 Monitor anomalous tool sequences, unusual destinations, privilege requests, repeated abuse attempts, and cost spikes where relevant.
  • RED-080 Record safe identifiers for model, prompt, policy, guardrail, tool, dataset, and release versions to support investigations.
  • RED-081 Restrict access to security telemetry and redact protected content, credentials, and test secrets by policy.
  • RED-082 Set actionable alert thresholds with named responders and test routing before production exposure.
  • RED-083 Define procedures to freeze a rollout, disable a capability, revoke credentials, quarantine data, or suspend a connector.
  • RED-084 Test that emergency suspension reaches active runs, queued actions, child agents, and external integrations where feasible.
  • RED-085 Ensure error handling does not substitute a fabricated success, source citation, or action confirmation after a safety stop.
  • RED-086 Provide safe user messaging for blocked requests, uncertain outcomes, service failures, and human escalation.
  • RED-087 Offer a visible user reporting channel for harmful, inaccurate, or unauthorized AI behavior where users interact with the system.
  • RED-088 Ensure reporting and complaint workflows distinguish immediate safety concerns from normal quality feedback.
  • RED-089 Record significant findings with owners, severity, mitigation, user impact, and validated retest evidence in project-specific records.
  • RED-090 Define appropriate customer, partner, regulator, or provider notification review when exposure or contractual obligations may apply.
  • RED-091 Review incidents for root causes involving model behavior, unsafe tool authority, guardrail gaps, and organizational process.
  • RED-092 Add confirmed incidents and newly discovered attack paths to maintained regression suites using sanitized fixtures.
  • RED-093 Reassess thresholds, monitoring, and escalation arrangements as traffic, threat patterns, and supported languages change.
  • RED-094 Perform post-deployment safety review during initial canary exposure and after material model or policy updates.
  • RED-095 Train designated reviewers on automation bias, sensitive data handling, escalation limits, and evidence preservation.
  • RED-096 Track residual high-severity risks and prohibit release without explicit qualified authorization and documented mitigations.

Release gate#

  • A reviewed threat model covers intended use, attacker-controlled inputs, privileged operations, sensitive data, relevant harms, and approved test boundaries.
  • A versioned adversarial evaluation set covers representative direct/indirect attacks, meaningful tool and data boundaries, high-impact policy failures, and benign controls.
  • No unaccepted high-severity violation remains, including unauthorized access, protected-data disclosure, disallowed external side effects, or serious user harm.
  • Moderation and other guardrails have documented coverage, known limitations, false-positive/false-negative measurements, and safe behavior during outages and handoffs.
  • Independent authentication, authorization, parameter validation, resource limits, and action-specific approval prevent prompt wording from granting privilege or bypassing protections.
  • API, retrieval, and frontend paths have been tested for unsafe output handling, information leakage, improper caching, failure states, and supported multilingual/multimodal behavior where relevant.
  • Safety monitoring, incident escalation, emergency disablement, user reporting, and validated retest/rollback procedures have named owners.
  • Release approvers have reviewed exact model/prompt/policy/tool versions, test results, residual risks, and written risk-acceptance decisions with deadlines.

Domain-specific approval rule: Block sign-off if an agent or model can bypass required access controls, execute material unauthorized actions, disclose protected content, or cause serious harm without acceptable mitigation; if required guardrails fail open on dangerous paths; or if critical attack surfaces have not been meaningfully evaluated. No red-team result establishes perfect security, comprehensive legal compliance, or immunity to new attacks.

Trusted references and scope#

These official risk frameworks, test guidance, and provider documentation inform the review items; applicability depends on system capability and the relevant policy.

Applicability: OWASP GenAI LLM Top 10 (2026) and Agentic Top 10 (2026) are awareness and risk frameworks, not certifications. MITRE ATLAS is a living adversary knowledge base; do not treat its tactics as an exhaustive checklist. NIST AI RMF and its generative AI profile are voluntary guidance. Provider moderation models and agent-guardrail SDKs differ in modalities, integration points, blocking semantics, and coverage; verify the actual runtime. Safety classifiers and prompt instructions are defense in depth, not authorization controls. Only conduct red-teaming where the project has explicit authority.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

  • Test focus: injection, jailbreak, exfiltration, unsafe tools, and guardrail bypass; include model and dataset behaviors where applicable.
  • Cross-stack: garak + promptfoo.
  • Python: garak + pytest; for pytest-based suites, run python -m pytest -q.
  • TypeScript: promptfoo + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
  • JavaScript: promptfoo + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
  • Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

How to check: In a project-specific copy, have the agent audit the AI-specific scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.

Reference, not a certification. Check applicability, verify important claims against current primary sources, and seek expert review where necessary. Suggest an improvement.