21 — AI Evaluation and Regression Checklist#

Verify that AI-enabled applications have representative, measurable, repeatable evaluation of task quality and safety, and that changes are held to defined regression and release standards. Scores from a benchmark, a model judge, or a dashboard alone do not prove real-world correctness or absence of harm.

Scope: Frontend-specific. This file is one part of the 21_AI_Evaluation_and_Regression.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

• Project / product:
• Release / version / environment:
• Scope and applicable jurisdictions / markets:
• Evaluated tasks, models, agents, retrieval, and user populations:
• Golden dataset / evaluator / model and prompt versions:
• Task, safety, latency, cost, and regression thresholds:
• Evaluator / domain / risk approvers:
• Review date:
• Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

1. Copy this shared checklist into project-specific evaluation records. Leave the repository's reusable checkbox items unchecked.
2. Define task outcomes, severe failures, baseline performance, and release thresholds before inspecting candidate results. Use separate measures for task success, policy/safety, retrieval, downstream tool actions, user experience, and operational cost where applicable.
3. Mark [x] only when verified from versioned dataset manifests, reproducible runs, validated scorers, qualified human reviews, CI outcomes, or a documented approver decision. Keep evidence, failures, exceptions, and genuine N/A reasons outside this template, indexed by EVAL ID.
4. Compare under equivalent settings; account for stochastic generation, sample size, evaluator bias, and real-world coverage limitations. A high aggregate score must never hide a severe individual failure.
5. Apply ## API Checks and ## Frontend Checks independently for supported interfaces. Document inapplicable items in project review records.
6. Re-run appropriate evaluations after model, prompt, retrieval, dataset, guardrail, API, tool, or frontend changes. Coordinate model selection, data curation, governance, safety/red teaming, and monitoring in their separate checklists.

Frontend Checks#

• EVAL-117 Evaluate complete user journeys and representative device/browser conditions in addition to isolated model text outputs.
• EVAL-118 Show appropriate uncertainty, no-answer, refusal, or escalation UI for cases where the model cannot meet the task.
• EVAL-119 Verify citations and factual evidence displayed in the UI support the corresponding claims when source-grounding is promised.
• EVAL-120 Test how users perceive model-generated suggestions compared with independently verified facts or committed external actions.
• EVAL-121 Check output rendering for malformed markup, long text, unsupported characters, streaming, tables, and structured widgets.
• EVAL-122 Ensure user-visible failures, timeouts, missing context, and rejected tool actions have accessible recovery paths.
• EVAL-123 Test keyboard, assistive-technology, and focus behavior for AI-generated dynamic content and error states.
• EVAL-124 Validate that localization, multilingual input, and supported multimodal flows are represented in regression tests.
• EVAL-125 Compare page/task latency, time-to-first-content, and completion time before and after model or prompt changes.
• EVAL-126 Test how frontend caches, old browser tabs, or feature flags behave with new prompt and response formats.
• EVAL-127 Confirm client-side validation and rendering cannot make an unsafe model response executable.
• EVAL-128 Verify agent actions appear as proposed, pending, approved, or confirmed only when supported by authoritative state.
• EVAL-129 Include approved user research and comprehension checks where automation bias or overreliance is a material risk.
• EVAL-130 Ensure test interfaces do not display restricted golden answers, private evaluation prompts, or secret reference data to unauthorized users.
• EVAL-131 Make feedback and report-an-error controls usable for incorrect, harmful, or inaccessible AI output.
• EVAL-132 Check that production feedback tagging records the deployed model/configuration without collecting unnecessary personal data.
• EVAL-133 Evaluate user experience of staged rollout and fallback when a different model changes output length, tone, or available features.
• EVAL-134 Test visible safety warnings and confirmations for high-impact workflows before release.
• EVAL-135 Review mobile and low-bandwidth behavior under slow or failing evaluation-relevant AI services.
• EVAL-136 Check important frontend regressions and support escalations against release-approved user-task metrics.

Release gate#

• The exact release has approved task definitions, meaningful primary metrics, severe-failure conditions, risk guardrails, and predeclared acceptance thresholds.
• Golden and held-out evaluation cases are representative of the intended deployment, versioned, permissioned, contamination-checked, and reviewed for major coverage gaps.
• Deterministic assertions and model-based graders are fit for purpose; human rubrics and automated judge scores have suitable calibration and auditability.
• Candidate and approved baseline results are compared with reproducible settings, adequate repeatability, and explicit statistical or practical uncertainty where needed.
• All release-blocking privacy, safety, authorization, and consequential task failures are resolved or have the specifically required authorized disposition; averaging does not waive critical failures.
• API, retrieval, agent/tool, and frontend end-to-end paths meet the relevant quality, contract, usability, accessibility, latency, and cost requirements.
• CI/controlled release gates, anomaly and regression review, safe rollback, production observation, and accountable incident response are ready.
• A named release approver has reviewed run artifacts, per-case regressions, coverage limitations, accepted risks, and the precise model/data/prompt/guardrail versions promoted.

Domain-specific approval rule: Block sign-off when the evaluation suite is materially unrepresentative or contaminated, scoring is unvalidated for decisions it controls, a severe user-impacting or unauthorized-action failure is unresolved, or an important metric regresses beyond its approved threshold without qualified acceptance. Passing these checks is not proof of complete AI safety, impartiality, or compliance with any particular jurisdiction.

Trusted references and scope#

The following official frameworks, provider documentation, and tool-maintainer references informed these project-specific verification questions. They are not universal requirements or substitutes for independent evaluation design.

• NIST AI RMF Playbook — Measure — risk-based evaluation methods, uncertainty, deployment conditions, and measurement gaps.
• NIST AI RMF Generative AI Profile (AI 600-1) — generative AI risk identification, evaluation, and trustworthiness considerations.
• OpenAI API — Evaluation best practices — task-specific evals, representative cases, model variability, calibration, and development-driven regression.
• OpenAI API — Graders — deterministic and model-based grading, human reference scores, grader limitations; note platform deprecation.
• Microsoft Foundry — Evaluation overview — evaluation workflow, datasets, response generation, quality/safety metrics, and production traces.
• Microsoft Foundry — Built-in Evaluators — task completion, retrieval, groundedness, safety, and textual similarity metrics.
• Microsoft Foundry — RAG Evaluators — separate retrieval-process and grounded-answer system evaluation.
• Google Cloud — Gen AI evaluation service overview — adaptive task rubrics, version/model comparisons, production sampling, and synthetic test cases.
• Amazon Bedrock — Automatic model evaluation jobs — task-specific dataset jobs, settings, built-in metrics, and evaluation artifacts.
• Amazon Bedrock — Human model evaluation jobs — human review teams, task-specific instructions, qualitative comparisons, and grading controls.
• Amazon Bedrock — LLM-as-a-judge evaluation jobs — judge-model evaluation configuration, IAM, datasets, and permissions.
• Ragas — Available evaluation metrics — RAG faithfulness/retrieval and agent task/tool metrics where this library is chosen.

Applicability: NIST AI RMF and its GenAI profile are voluntary risk frameworks, not certifications. As of November 2026, the OpenAI Evals platform and associated graders are being retired (read-only after October 31; scheduled closure November 30); avoid building new long-lived release gates dependent on that service and verify the latest migration guidance. Microsoft, Google, and AWS evaluation features, model judges, and scoring thresholds are platform-specific and may have version, regional, or preview limitations. Ragas metrics are implementation choices, not a universal scientific gold standard. Protect evaluation data and check legal rights before sharing it with third-party judges.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

• Test focus: held-out cases, severe failures, judge calibration, and CI quality thresholds; include browser and client behavior where applicable.
• Cross-stack: promptfoo.
• Python: DeepEval + Ragas + pytest; for pytest-based suites, run python -m pytest -q.
• TypeScript: promptfoo + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
• JavaScript: promptfoo + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
• Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

• agentic-eval — Check end-to-end agent outcome regressions (github/awesome-copilot, 41,073 repo stars).
• eval-driven-dev — Test evaluation-based development gates (github/awesome-copilot, 41,073 repo stars).
• llm-evaluation — Review datasets, graders, and quality thresholds (wshobson/agents, 42,437 repo stars).

How to check: In a project-specific copy, have the agent audit the frontend scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.