awesomechecklists
llms.txtGitHub ↗
/

03 — AI Evaluation and Governance Checklist#

Review whether a proposed AI use is appropriate, its risks are understood, the system has credible evaluation evidence, human oversight is operational, and organizational accountability is maintained across its lifecycle. This cross-cutting checklist complements detailed model, RAG, agent, red-team, data, and monitoring templates in their respective AI domain checklists.

Scope: AI-specific. This file is one part of the 03_AI_Evaluation_and_Governance.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

  • Project / product:
  • Release / version / environment:
  • AI use case / intended users and deployment context:
  • Model(s), providers, and system-version references:
  • Risk category and applicable jurisdictions / markets:
  • Governance, evaluation, human-oversight, and compliance owners:
  • Reviewer / owner:
  • Review date:
  • Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

  1. Copy the shared template to project-specific review records; leave all repository checkboxes unchecked.
  2. Define the AI system boundary, intended purpose, affected people, risk tolerance, and responsible human approvers before selecting evaluation and release controls.
  3. Mark [x] only after verifying the check through a risk assessment, versioned model/system documentation, evaluation result, control test, supplier review, or authorized governance decision. Record evidence, failures, accepted exceptions, and justified N/A outside the shared template, keyed by AIG ID.
  4. Complete ## API Checks and ## Frontend Checks independently where applicable. For non-API or non-interactive products, document why relevant checks do not apply.
  5. Adapt tests and oversight to foreseeable harms and actual system authority. Re-evaluate after significant model, policy, prompt, data, tool, market, or regulatory changes. Do not treat a benchmark pass as proof of safety or legal compliance.
  6. Route deep specialist reviews to the appropriate specialized AI checklists; keep this file focused on cross-cutting risk, evidence, approval, and accountability.

AI use-case and risk definition#

  • AIG-001 Describe the intended AI-assisted task, user groups, operating context, and expected benefit in terms that can be evaluated.
  • AIG-002 Determine whether a non-AI solution could achieve the same user outcome with lower risk or operational complexity.
  • AIG-003 Identify the AI system owner, business sponsor, model/application operator, and accountable release approver.
  • AIG-004 Maintain an inventory of in-scope AI models, providers, versions, datasets, prompts, retrieval sources, tools, and deployment locations.
  • AIG-005 Classify the system by potential impact and likelihood of harm using a documented, project-appropriate risk methodology.
  • AIG-006 Identify affected people, bystanders, downstream recipients, and vulnerable populations who may bear different risks.
  • AIG-007 Map foreseeable normal use, misuse, out-of-scope use, and high-consequence failure scenarios across the full user journey.
  • AIG-008 Record whether AI outputs inform or determine decisions in health, employment, education, finance, safety, legal, or other consequential contexts.
  • AIG-009 Assess potential physical, psychological, economic, reputational, privacy, security, and societal impacts relevant to the deployment.
  • AIG-010 Identify discrimination or disparate-impact risks across relevant groups and contexts using lawful, appropriate data and methods.
  • AIG-011 Define unacceptable uses, prohibited outcomes, and risk tolerance before design or release approval.
  • AIG-012 Assess whether the system should use generation, classification, retrieval, prediction, ranking, or deterministic rules for each task.
  • AIG-013 Map the trust boundaries between user content, retrieved data, model instructions, tool results, APIs, and privileged operations.
  • AIG-014 Identify data sensitivity, provenance, intellectual-property restrictions, and permitted processing purposes before using external AI services.
  • AIG-015 Review model-provider geography, data handling, subprocessor, and retention terms against the intended data flows.
  • AIG-016 Document AI capability limitations, missing knowledge, uncertainty, and circumstances requiring abstention or escalation.
  • AIG-017 Evaluate how users may over-rely on seemingly authoritative outputs, including automation bias and misunderstood confidence.
  • AIG-018 Identify the jurisdictions, deployment roles, and AI-specific legal or contractual requirements that require qualified review.
  • AIG-019 Assess EU AI Act applicability and changing phased deadlines for any relevant EU-market use case using current official guidance.
  • AIG-020 Record the risk classification, assumptions, material open issues, and the decision whether this AI use case is justified.
  • AIG-021 Define clear boundaries for any external actions, financial commitments, messages, or changes the AI system may initiate.
  • AIG-022 Reassess risk assumptions when the model, audience, context, connected tools, or policy requirements materially change.

Evaluation approach#

  • AIG-023 Define measurable task success criteria and unacceptable failures before selecting an evaluation dataset or model.
  • AIG-024 Select evaluation metrics appropriate to the use case, including task quality, accuracy, recall, consistency, abstention, and error severity where relevant.
  • AIG-025 Set explicit acceptance thresholds, review rules, and responsible approvers for high-impact outcomes.
  • AIG-026 Establish a deterministic or human-workflow baseline where feasible to show the added value of the AI system.
  • AIG-027 Create representative test cases covering routine, ambiguous, long-tail, multilingual, and high-impact scenarios in the target population.
  • AIG-028 Include realistic user goals and full workflow context rather than evaluating only isolated model text outputs.
  • AIG-029 Include clearly defined negative tests for prohibited behaviors, unsupported requests, and out-of-scope questions.
  • AIG-030 Evaluate factual accuracy and unsupported assertions against trusted references when factual outputs are expected.
  • AIG-031 Assess whether the system states uncertainty, asks for missing context, and abstains or escalates appropriately.
  • AIG-032 Evaluate whether model outputs include unsafe instructions, privacy leaks, harmful stereotyping, or discriminatory outcomes relevant to the context.
  • AIG-033 Test prompt injection, malicious retrieved content, untrusted tool output, and attempted instruction or permission escalation at system boundaries.
  • AIG-034 Test whether tool-use authorization and external side effects remain controlled under adversarial and ambiguous inputs.
  • AIG-035 Measure false positives and false negatives separately where errors have materially different costs.
  • AIG-036 Evaluate performance by justified subgroups, languages, devices, and operating environments while avoiding unlawful or unreliable inference.
  • AIG-037 Separate development, tuning, validation, and final held-out evaluation data to reduce contamination and overfitting.
  • AIG-038 Track test set provenance, allowed use, version, sampling limitations, and changes to expected outcomes.
  • AIG-039 Use appropriate human evaluation rubrics, reviewer qualification, and adjudication of disagreements where automated scores are insufficient.
  • AIG-040 Validate that model-based judges or automated evaluators align with competent human assessments before relying on their scores.
  • AIG-041 Check evaluator bias, judge position sensitivity, scoring drift, and failure to identify harmful outputs where applicable.
  • AIG-042 Report confidence intervals, sample sizes, uncertainty, and critical-error counts when interpreting quantitative results.
  • AIG-043 Run repeat evaluations under controlled settings to characterize nondeterminism and meaningful output variability.
  • AIG-044 Record model, prompt, retrieval, tool, guardrail, dataset, and configuration versions for every acceptance result.
  • AIG-045 Evaluate changes against a trusted baseline and flag regressions in safety, fairness, task quality, and user-impacting behavior.
  • AIG-046 Document limitations of benchmarks and offline datasets, including where they may not predict real-world outcomes.
  • AIG-047 Include staged or shadow evaluation with controlled real-world signals when risk and data permissions permit.
  • AIG-048 Ensure serious failures have clear owner, severity, remediation action, and retest requirements before sign-off.

Human oversight and accountability#

  • AIG-049 Assign responsibility for AI policy, deployment approval, operational oversight, user complaints, and model-incident escalation.
  • AIG-050 Define which decisions AI may support, which require human approval, and which must remain fully human-controlled.
  • AIG-051 Identify high-impact or irreversible actions and require an appropriately authorized human checkpoint before execution.
  • AIG-052 Ensure reviewers have enough context, source material, and system limitations to make independent decisions.
  • AIG-053 Give authorized humans practical means to pause, override, reject, correct, or disable unsafe AI behavior.
  • AIG-054 Test that escalation to a human actually reaches a responsible person within an appropriate service window.
  • AIG-055 Define who can approve exceptional AI uses, accept residual risks, and revoke earlier approval.
  • AIG-056 Train reviewers on automation bias, hallucinations, limitations, relevant safety risks, and appropriate verification methods.
  • AIG-057 Set workload and escalation limits that make genuine human review feasible rather than purely nominal.
  • AIG-058 Distinguish recommendations, drafts, and automated decisions in internal workflows and customer interactions.
  • AIG-059 Define a clear process for users to report harmful, incorrect, or discriminatory AI outputs and request follow-up.
  • AIG-060 Provide a route for contesting or seeking human review of consequential AI-supported outcomes where appropriate or legally required.
  • AIG-061 Document when users and affected parties must be informed they are interacting with an AI system or receiving AI-generated content.
  • AIG-062 Ensure user-facing disclosures accurately describe capabilities, limits, available human assistance, and relevant data handling.
  • AIG-063 Define the criteria and authorized process for suspending AI features after a serious incident or evidence of emerging harm.
  • AIG-064 Keep incident records including affected scope, causal hypotheses, temporary mitigations, user notifications, and corrective actions.
  • AIG-065 Review anonymized or appropriately protected user feedback and incident trends for previously unrecognized harms.
  • AIG-066 Ensure accountability for AI-generated external communications, transactions, or changes remains with a named human or organization.
  • AIG-067 Test handoff, appeal, and emergency-disablement paths rather than relying on policy documents alone.
  • AIG-068 Review oversight and escalation arrangements after staffing, supplier, or critical workflow changes.

Model and policy governance#

  • AIG-069 Maintain an AI system register identifying owners, use case, model/provider versions, deployment context, risk class, and approval state.
  • AIG-070 Define a documented AI governance policy covering acceptable use, prohibited uses, data handling, human oversight, and incident response.
  • AIG-071 Assign review responsibilities across product, engineering, security, privacy, legal, domain experts, and affected-user representatives as warranted.
  • AIG-072 Keep decisions and risk treatments traceable to owners, review dates, evidence, and residual-risk acceptance.
  • AIG-073 Document the provenance, licensing, allowed purposes, and known restrictions of models and third-party AI components.
  • AIG-074 Review supplier contracts, usage policies, data-processing terms, audit rights, retention, and model-change notification practices.
  • AIG-075 Maintain model or system documentation describing intended uses, capabilities, evaluation results, known limitations, and warnings.
  • AIG-076 Identify training, fine-tuning, retrieval, or feedback data that could introduce bias, sensitive information, or intellectual-property risk.
  • AIG-077 Define controls to prevent unauthorized training or secondary use of customer data by internal systems or suppliers where prohibited.
  • AIG-078 Document prompt, system policy, retrieval configuration, guardrail, tool permissions, and model changes through version-controlled review.
  • AIG-079 Set a change classification that distinguishes low-risk prompt/content edits from significant model, capability, autonomy, and data-use changes.
  • AIG-080 Require renewed evaluation and authorization before material changes to intended purpose, model, tool authority, or deployment audience.
  • AIG-081 Define operational monitors for unexpected outputs, misuse, policy violations, user complaints, cost, and model-quality degradation.
  • AIG-082 Establish a controlled incident response for AI-specific failures, including containment, investigation, remediation, and notification owners.
  • AIG-083 Document how AI features can be rolled back, disabled, or routed to a safe alternative without corrupting user work.
  • AIG-084 Set retention and restricted access for evaluations, prompts, model inputs/outputs, incident records, and audit artifacts.
  • AIG-085 Reassess regulatory applicability and required documentation against the law and effective dates relevant to the deployment.
  • AIG-086 Distinguish voluntary NIST/OECD guidance and ISO management-system practices from mandatory laws and contract clauses.
  • AIG-087 Require a qualified compliance review before asserting conformity, certification, or legal approval for the system.
  • AIG-088 Define supplier-exit, model-replacement, and data-deletion plans for critical third-party AI dependencies.
  • AIG-089 Assign periodic policy review, independent challenge, and management reporting proportional to risk.
  • AIG-090 Record the justification for accepting residual harm risks and the conditions that require reconsidering approval.

Release gate#

  • The AI use case, appropriate non-AI alternatives, intended population, system boundaries, and responsible owners have been documented and approved.
  • Significant foreseeable harms and applicable prohibited uses or legal/contractual obligations have been reviewed, with risk classification and accountable decisions recorded.
  • Representative evaluation evidence meets predeclared task-quality and safety thresholds, including tests for consequential failures, adversarial inputs, and relevant population differences.
  • Human-review, appeal, intervention, and emergency-disablement procedures have been tested for high-impact or externally acting AI workflows where appropriate.
  • Model/provider licensing, processing terms, data handling, versioning, and change controls are reviewed; unauthorized access and unbounded tool actions are not permitted.
  • API and frontend behavior clearly communicates AI limitations and errors and enforces approved roles, output handling, and user-facing disclosures where applicable.
  • Monitoring, incident-response ownership, complaint intake, and change-triggered reevaluation are established for the deployed system.
  • Material gaps or residual risks are either resolved or explicitly accepted by authorized owners with mitigation, review date, and conditions for suspension.

Domain-specific approval rule: Block AI governance sign-off if a prohibited use or unaccepted serious harm remains, a critical system action lacks required human authorization, major security/privacy exposures remain uncontrolled, or representative evaluation cannot support the intended use. Human approval must not be inferred from automated quality scores. This checklist does not establish ISO certification, NIST endorsement, or compliance with any specific jurisdiction's AI law.

Trusted references and scope#

These are primary and authoritative sources for cross-cutting AI risk and governance decisions. Recommendations from non-binding frameworks are not universal legal duties, and management-system standards do not themselves certify a particular AI deployment.

Applicability: NIST AI RMF 1.0 and its Playbook are voluntary and were under revision as of this source review. ISO/IEC 42001 and 23894 are published standards with full texts subject to license/access; these links identify their scope but are not a substitute for reading licensed requirements. The EU AI Act has phased, revised application dates as of 2026; distinguish providers, deployers, risk classes, markets, and effective dates with qualified legal counsel. OWASP GenAI LLM Top 10 2026 applies primarily to relevant LLM-integrated systems. Only implement controls proportionate to system risk and actual interfaces.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

  • Test focus: AI policy thresholds, task/safety slices, escalation, and traceable evaluations; include model and dataset behaviors where applicable.
  • Cross-stack: promptfoo + OpenTelemetry.
  • Python: DeepEval + pytest; for pytest-based suites, run python -m pytest -q.
  • TypeScript: promptfoo + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
  • JavaScript: promptfoo + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
  • Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

  • agent-governance — Assess AI oversight and operational policies (github/awesome-copilot, 41,073 repo stars).
  • llm-evaluation — Review quality/safety metrics and datasets (wshobson/agents, 42,437 repo stars).
  • agentic-eval — Check repeatable evaluations and failure thresholds (github/awesome-copilot, 41,073 repo stars).

How to check: In a project-specific copy, have the agent audit the AI-specific scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.

Reference, not a certification. Check applicability, verify important claims against current primary sources, and seek expert review where necessary. Suggest an improvement.