awesomechecklists
llms.txtGitHub ↗
/

22 — AI Red Teaming and Guardrails Checklist#

Verify that AI-enabled products have risk-based adversarial coverage and enforceable defenses against harmful outputs, unauthorized actions, data disclosure, and abuse. Red teaming seeks to discover failures; passing a finite test suite does not prove a model or product cannot be bypassed.

Scope: Backend / API-specific. This file is one part of the 22_AI_Red_Teaming_and_Guardrails.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

  • Project / product:
  • Release / version / environment:
  • Scope and applicable jurisdictions / markets:
  • AI capabilities, model/provider, tools, data sources, and protected assets:
  • Authorized red-team scope / test environment / contact:
  • Policy / guardrail versions and risk/incident owners:
  • Reviewer / owner:
  • Review date:
  • Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

  1. Copy this shared template to project-specific review records; keep every checkbox in the repository unchecked.
  2. Scope adversarial testing to systems, models, datasets, and endpoints the project owns or is authorized to test. Use synthetic identities, dummy data, sandboxed tools, and safe endpoints; do not attack unrelated systems or disclose real secrets.
  3. Mark [x] only after verifying a check through an approved threat model, model/tool permission test, adversarial result, policy decision, safety drill, or production-equivalent guardrail validation. Track evidence, defects, accepted exceptions, and justified N/A in project-specific notes/issues keyed by RED ID, not in this template.
  4. Separate security objective, actual observed violation, blocked attempt, inconclusive result, and benign false positive. Do not credit a model refusal if unauthorized tool effects or external disclosure already occurred.
  5. Evaluate ## API Checks and ## Frontend Checks independently where those surfaces exist; document N/A outside the template for systems without them.
  6. Repeat targeted adversarial and benign evaluation after changes to the model, prompts, retrieved data, tool scopes, guardrails, modalities, or application policy. Review general AI governance and detailed agent, RAG, and regression requirements in their other checklists.

API Checks#

  • RED-097 Authenticate and authorize safety-sensitive endpoints, moderation routes, policy overrides, and privileged model integrations.
  • RED-098 Apply caller, resource, tenant, and purpose restrictions before model-generated tool or retrieval operations execute.
  • RED-099 Validate structured outputs, tool arguments, URL destinations, and user-supplied media against strict server-side rules.
  • RED-100 Test direct and indirect injection through every API-supported data ingress, including uploaded files and external tool results.
  • RED-101 Verify safety filtering covers API requests and relevant outputs without assuming all SDK or hosted tool paths share guardrail hooks.
  • RED-102 Check guardrail failure, timeout, and degraded-service behavior for safe treatment of sensitive actions.
  • RED-103 Enforce bounded prompt length, response size, tool count, per-user usage, retries, concurrency, and model spending.
  • RED-104 Record guardrail verdicts and security decisions with safe correlation IDs and policy revisions, without logging raw secrets.
  • RED-105 Test that policy-blocked content cannot leak through streaming fragments, cached responses, asynchronous jobs, or alternate routes.
  • RED-106 Measure unauthorized tool attempts and actual state changes rather than trusting only model-produced refusal messages.
  • RED-107 Verify API error contracts distinguish blocked, refused, review-required, malformed, unavailable, and timed-out results.
  • RED-108 Test moderation and action checks with encoded, non-Latin, long, malformed, and multimodal input where supported.
  • RED-109 Ensure approved human-review decisions bind to the correct identity, action, arguments, and current permissions.
  • RED-110 Validate that rate limiting and abuse controls cannot be bypassed by switching model aliases, endpoints, or tenants.
  • RED-111 Check cross-tenant isolation in retrieval indexes, agent memory, response caches, monitoring, and evaluation exports.
  • RED-112 Test safe rollback or feature disablement after a critical guardrail or model-policy regression.
  • RED-113 Ensure API integrations with moderation and model providers use approved regions, storage settings, and processing terms.
  • RED-114 Run safety and authorization contract tests in CI with isolated sandbox accounts and synthetic payloads.
  • RED-115 Check model-, guardrail-, prompt-, and tool-version alignment against the approved production release.
  • RED-116 Monitor p95 latency, policy-decision availability, false blocks, and risk-related throughput under representative load.

Release gate#

  • A reviewed threat model covers intended use, attacker-controlled inputs, privileged operations, sensitive data, relevant harms, and approved test boundaries.
  • A versioned adversarial evaluation set covers representative direct/indirect attacks, meaningful tool and data boundaries, high-impact policy failures, and benign controls.
  • No unaccepted high-severity violation remains, including unauthorized access, protected-data disclosure, disallowed external side effects, or serious user harm.
  • Moderation and other guardrails have documented coverage, known limitations, false-positive/false-negative measurements, and safe behavior during outages and handoffs.
  • Independent authentication, authorization, parameter validation, resource limits, and action-specific approval prevent prompt wording from granting privilege or bypassing protections.
  • API, retrieval, and frontend paths have been tested for unsafe output handling, information leakage, improper caching, failure states, and supported multilingual/multimodal behavior where relevant.
  • Safety monitoring, incident escalation, emergency disablement, user reporting, and validated retest/rollback procedures have named owners.
  • Release approvers have reviewed exact model/prompt/policy/tool versions, test results, residual risks, and written risk-acceptance decisions with deadlines.

Domain-specific approval rule: Block sign-off if an agent or model can bypass required access controls, execute material unauthorized actions, disclose protected content, or cause serious harm without acceptable mitigation; if required guardrails fail open on dangerous paths; or if critical attack surfaces have not been meaningfully evaluated. No red-team result establishes perfect security, comprehensive legal compliance, or immunity to new attacks.

Trusted references and scope#

These official risk frameworks, test guidance, and provider documentation inform the review items; applicability depends on system capability and the relevant policy.

Applicability: OWASP GenAI LLM Top 10 (2026) and Agentic Top 10 (2026) are awareness and risk frameworks, not certifications. MITRE ATLAS is a living adversary knowledge base; do not treat its tactics as an exhaustive checklist. NIST AI RMF and its generative AI profile are voluntary guidance. Provider moderation models and agent-guardrail SDKs differ in modalities, integration points, blocking semantics, and coverage; verify the actual runtime. Safety classifiers and prompt instructions are defense in depth, not authorization controls. Only conduct red-teaming where the project has explicit authority.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

  • Test focus: injection, jailbreak, exfiltration, unsafe tools, and guardrail bypass; include API and service behavior where applicable.
  • Cross-stack: garak + promptfoo.
  • Python: garak + pytest; for pytest-based suites, run python -m pytest -q.
  • TypeScript: promptfoo + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
  • JavaScript: promptfoo + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
  • Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

How to check: In a project-specific copy, have the agent audit the backend / API scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.

Reference, not a certification. Check applicability, verify important claims against current primary sources, and seek expert review where necessary. Suggest an improvement.