awesomechecklists
llms.txtGitHub ↗
/

10 — AI Evaluation and Governance Checklist#

Review whether a proposed AI use is appropriate, its risks are understood, the system has credible evaluation evidence, human oversight is operational, and organizational accountability is maintained across its lifecycle. This cross-cutting checklist complements detailed model, RAG, agent, red-team, data, and monitoring templates in their respective AI domain checklists.

Scope: Backend / API-specific. This file is one part of the 10_AI_Evaluation_and_Governance.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

  • Project / product:
  • Release / version / environment:
  • AI use case / intended users and deployment context:
  • Model(s), providers, and system-version references:
  • Risk category and applicable jurisdictions / markets:
  • Governance, evaluation, human-oversight, and compliance owners:
  • Reviewer / owner:
  • Review date:
  • Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

  1. Copy the shared template to project-specific review records; leave all repository checkboxes unchecked.
  2. Define the AI system boundary, intended purpose, affected people, risk tolerance, and responsible human approvers before selecting evaluation and release controls.
  3. Mark [x] only after verifying the check through a risk assessment, versioned model/system documentation, evaluation result, control test, supplier review, or authorized governance decision. Record evidence, failures, accepted exceptions, and justified N/A outside the shared template, keyed by AIG ID.
  4. Complete ## API Checks and ## Frontend Checks independently where applicable. For non-API or non-interactive products, document why relevant checks do not apply.
  5. Adapt tests and oversight to foreseeable harms and actual system authority. Re-evaluate after significant model, policy, prompt, data, tool, market, or regulatory changes. Do not treat a benchmark pass as proof of safety or legal compliance.
  6. Route deep specialist reviews to the appropriate specialized AI checklists; keep this file focused on cross-cutting risk, evidence, approval, and accountability.

API Checks#

  • AIG-091 Inventory all model inference, embeddings, retrieval, moderation, feedback, and action-execution APIs that form the AI product.
  • AIG-092 Describe the intended purpose and authority of each endpoint, including authenticated caller roles and tenant boundaries.
  • AIG-093 Enforce authorization on the server for model outputs, retrieval access, and every external tool action regardless of prompt content.
  • AIG-094 Treat model-generated arguments and retrieved content as untrusted input; validate them against strict schemas and permissions.
  • AIG-095 Apply output validation and downstream encoding before model content is rendered, stored, executed, or sent to other services.
  • AIG-096 Restrict high-impact API actions with explicit user consent or human approval gates appropriate to the operation.
  • AIG-097 Protect model credentials, provider tokens, customer input, sensitive prompt content, and retrieved documents from accidental logging.
  • AIG-098 Document provider data-use and retention settings and enforce approved endpoint regions and account configuration where applicable.
  • AIG-099 Use scoped timeouts, quotas, concurrency bounds, rate limits, and spending controls to prevent unbounded inference or tool calls.
  • AIG-100 Define safe retry and idempotency behavior so AI-mediated API timeouts do not duplicate purchases, messages, or data changes.
  • AIG-101 Include structured error states for refusal, abstention, invalid output, tool failure, timeout, and policy block where appropriate.
  • AIG-102 Make API response contracts distinguish verified facts, model-generated content, citations, tool results, and confidence estimates when supported.
  • AIG-103 Do not expose internal instructions, proprietary system prompts, or confidential retrieval data to unauthorized API consumers.
  • AIG-104 Propagate safe audit and correlation identifiers across model calls, retrieval steps, guardrails, and human approval actions.
  • AIG-105 Record the model, policy, prompt, retrieval, and tool configuration version associated with relevant AI responses for investigation.
  • AIG-106 Test multi-tenant isolation in vector searches, caches, prompts, conversation history, and tool results.
  • AIG-107 Verify moderation or safety gates fail according to an approved safe policy rather than silently bypassing checks during outages.
  • AIG-108 Exercise injection attempts and malicious tool or retrieved-document content against production-equivalent API permissions.
  • AIG-109 Monitor per-operation quality, refusal, incident, latency, error, and cost signals in proportion to the deployment risk.
  • AIG-110 Ensure API fallback or model substitution does not silently introduce an unapproved capability, data region, or policy change.

Release gate#

  • The AI use case, appropriate non-AI alternatives, intended population, system boundaries, and responsible owners have been documented and approved.
  • Significant foreseeable harms and applicable prohibited uses or legal/contractual obligations have been reviewed, with risk classification and accountable decisions recorded.
  • Representative evaluation evidence meets predeclared task-quality and safety thresholds, including tests for consequential failures, adversarial inputs, and relevant population differences.
  • Human-review, appeal, intervention, and emergency-disablement procedures have been tested for high-impact or externally acting AI workflows where appropriate.
  • Model/provider licensing, processing terms, data handling, versioning, and change controls are reviewed; unauthorized access and unbounded tool actions are not permitted.
  • API and frontend behavior clearly communicates AI limitations and errors and enforces approved roles, output handling, and user-facing disclosures where applicable.
  • Monitoring, incident-response ownership, complaint intake, and change-triggered reevaluation are established for the deployed system.
  • Material gaps or residual risks are either resolved or explicitly accepted by authorized owners with mitigation, review date, and conditions for suspension.

Domain-specific approval rule: Block AI governance sign-off if a prohibited use or unaccepted serious harm remains, a critical system action lacks required human authorization, major security/privacy exposures remain uncontrolled, or representative evaluation cannot support the intended use. Human approval must not be inferred from automated quality scores. This checklist does not establish ISO certification, NIST endorsement, or compliance with any specific jurisdiction's AI law.

Trusted references and scope#

These are primary and authoritative sources for cross-cutting AI risk and governance decisions. Recommendations from non-binding frameworks are not universal legal duties, and management-system standards do not themselves certify a particular AI deployment.

Applicability: NIST AI RMF 1.0 and its Playbook are voluntary and were under revision as of this source review. ISO/IEC 42001 and 23894 are published standards with full texts subject to license/access; these links identify their scope but are not a substitute for reading licensed requirements. The EU AI Act has phased, revised application dates as of 2026; distinguish providers, deployers, risk classes, markets, and effective dates with qualified legal counsel. OWASP GenAI LLM Top 10 2026 applies primarily to relevant LLM-integrated systems. Only implement controls proportionate to system risk and actual interfaces.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

  • Test focus: AI policy thresholds, task/safety slices, escalation, and traceable evaluations; include API and service behavior where applicable.
  • Cross-stack: promptfoo + OpenTelemetry.
  • Python: DeepEval + pytest; for pytest-based suites, run python -m pytest -q.
  • TypeScript: promptfoo + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
  • JavaScript: promptfoo + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
  • Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

  • agent-governance — Assess AI oversight and operational policies (github/awesome-copilot, 41,073 repo stars).
  • llm-evaluation — Review quality/safety metrics and datasets (wshobson/agents, 42,437 repo stars).
  • agentic-eval — Check repeatable evaluations and failure thresholds (github/awesome-copilot, 41,073 repo stars).

How to check: In a project-specific copy, have the agent audit the backend / API scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.

Reference, not a certification. Check applicability, verify important claims against current primary sources, and seek expert review where necessary. Suggest an improvement.