11 — AI Model Selection and Benchmarking Checklist#

Choose AI models against the actual application workload, operating constraints, safety requirements, and measurable outcomes. Public leaderboards help identify candidates but do not prove suitability, compliance, or reliability for a particular deployment.

Scope: Frontend-specific. This file is one part of the 11_AI_Model_Selection_and_Benchmarking.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

• Project / product:
• Release / version / environment:
• Scope and applicable jurisdictions / markets:
• Intended task / affected users / impact level:
• Candidate models, providers, and exact versions:
• Benchmark datasets, scoring framework, and review owner:
• Cost, latency, privacy, safety, and quality budgets:
• Reviewer / owner:
• Review date:
• Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

1. Copy the shared template to project-specific evaluation records; keep all repository checkboxes unchecked.
2. Establish requirements, disqualifying risks, and acceptance thresholds before comparing model scores. Mark [x] only when verified through model documentation, reproducible benchmark results, human evaluation, provider review, or representative application tests.
3. Record evidence, failures, candidate exclusions, accepted risks, and justified N/A outside this shared template, keyed by MODELS ID. Do not add evidence tables or real customer data to the repository.
4. Compare candidates under equivalent workloads and document version, benchmark data, prompts, sampling, scoring, hosting, and resource settings. Adjust for measurable differences in real API/application behavior.
5. Complete ## API Checks and ## Frontend Checks where those interfaces exist. For non-interactive, offline, or non-API deployments, document what does not apply in project records.
6. Route deeper prompt, RAG, agent, red-team, fine-tuning, monitoring, model-card, and inference reviews to the dedicated AI files. Re-run selection when models, tasks, providers, risk profiles, prices, or serving conditions materially change.

Frontend Checks#

• MODELS-115 Validate the selected model's response quality on realistic user journeys rather than only offline text prompts.
• MODELS-116 Test whether users understand when answers are generated, uncertain, source-grounded, or incomplete.
• MODELS-117 Measure time to first visible response and completion time under representative browser and mobile network conditions.
• MODELS-118 Confirm streaming or partial results do not appear to be final decisions before validation or safety checks finish.
• MODELS-119 Ensure long answers, markdown, code, structured data, and multilingual text render safely and accessibly.
• MODELS-120 Test frontend handling of refusals, abstentions, model errors, timeouts, rate limits, and offline use.
• MODELS-121 Provide understandable feedback and recovery options when the selected model cannot perform the requested task.
• MODELS-122 Check that important citations and evidence are presented truthfully and do not imply unsupported factual verification.
• MODELS-123 Verify model-specific UI affordances such as attachments, image/audio inputs, and tool approvals work across supported devices.
• MODELS-124 Do not expose private model API keys, provider credentials, or confidential evaluation data in client bundles.
• MODELS-125 Test output variability for repeated user tasks and ensure the interface does not promise determinism where none exists.
• MODELS-126 Validate localization and accessibility for error states, safety interventions, and human escalation controls.
• MODELS-127 Test whether the model's tendency to be verbose, terse, or hedged harms task completion or comprehension.
• MODELS-128 Check whether fallback models alter response format, latency, language, or available actions in a way users need to understand.
• MODELS-129 Verify user consent and warning copy reflect the approved provider's actual data handling and model capabilities.
• MODELS-130 Define frontend guardrails for maximum input length, upload types, output length, and cost-sensitive operations as appropriate.
• MODELS-131 Confirm human-review and confirmation steps remain effective for consequential actions regardless of model ranking.
• MODELS-132 Correlate client-side failures and user reports with the deployed model version without exposing sensitive prompts or outputs.
• MODELS-133 Review a representative set of task completions with target users before declaring a model suitable for the workflow.
• MODELS-134 Ensure the selected model's production rollout has client-side health signals and feature-disablement paths.

Release gate#

• Intended use, stakeholder impact, mandatory capabilities, and measurable quality/safety, latency, cost, and data-handling requirements are approved.
• A relevant baseline and candidate shortlist were tested on representative and held-out workloads using documented, reproducible evaluation settings.
• Task quality, severe failure rates, robustness, relevant group/language performance, and uncertainty meet predeclared acceptance rules or have approved mitigations.
• Realistic serving measurements show acceptable latency, throughput, availability, resource use, and cost per successful user task.
• Provider terms, model licenses, data residency/retention, model-version controls, and safety/authorization constraints are reviewed for intended use.
• API and frontend application behavior is verified where applicable, including schema adherence, refusals, fallbacks, error states, and user-facing comprehension.
• Decision makers have reviewed limitations, disqualifying risks, supporting evidence, alternatives, and a rollback/re-evaluation plan.
• A named approver has accepted the exact model/provider/configuration decision and recorded any residual risks, owners, and trigger for renewed assessment.

Domain-specific approval rule: Block model selection when a critical task or safety threshold is missed, validation data is compromised, material license or data-handling constraints are unresolved, or latency/cost at required capacity is not viable without accepted mitigation. A high public benchmark ranking alone cannot override an unsafe or unfit result.

Trusted references and scope#

These primary framework, benchmark, and tool-maintainer sources inform the review questions. They are not proof that any specific model is suitable or that the product is certified by a standards body.

• NIST AI RMF Playbook — Measure — measurement methods, uncertainty, robustness, operating context, and validation.
• NIST AI RMF — Core Measure function — formal measurement of trustworthiness, uncertainty, independent assessment, and monitoring.
• NIST AI RMF Generative AI Profile (AI 600-1) — generative-AI risk categories for candidate evaluation when relevant.
• Hugging Face — Model Cards — intended use, model limitations, bias, experimental setup, and evaluation documentation.
• Hugging Face — Dataset Cards — benchmark-data provenance, licenses, languages, limitations, and representativeness.
• Stanford CRFM — HELM — multi-scenario, multi-metric comparison of foundation models; compare only relevant tasks.
• Stanford CRFM — HELM source and maintenance note — evaluation tooling; entered maintenance mode June 2026, so verify current support before adopting.
• EleutherAI — LM Evaluation Harness — reproducible prompt/task settings and open evaluation tooling.
• EleutherAI — Custom task guide — definition and validation of task-specific benchmark configurations.
• MLCommons — MLPerf Inference — fair and representative performance benchmarking across inference configurations.
• MLCommons — MLPerf Inference Rules — reproducible workload scenarios, result integrity, and latency/throughput measurements.
• Amazon Bedrock — Human model evaluation — human evaluation workflows for comparing candidate outputs when applicable.
• Google Cloud — Define generative AI evaluation metrics — task-appropriate grounding, safety, fluency, and customized rubric evaluation.

Applicability: NIST AI RMF material is voluntary guidance and the framework was under revision at the source review. HELM entered maintenance mode in June 2026. MLPerf measures defined hardware/software inference scenarios, not arbitrary end-user task quality. Benchmark harnesses and model/dataset cards require independent review of actual model versions, prompt templates, evaluation rights, and workload relevance. The listed proprietary platform documentation applies only when using those platforms; model/provider pricing and capabilities may change without advance notice.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

• Test focus: candidate models with identical datasets, metrics, latency, and safety thresholds; include browser and client behavior where applicable.
• Cross-stack: lm-evaluation-harness + MLPerf.
• Python: Hugging Face Evaluate + pytest; for pytest-based suites, run python -m pytest -q.
• TypeScript: promptfoo + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
• JavaScript: promptfoo + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
• Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

• eval-harness-first — Verify benchmark design before selection (wshobson/agents, 42,437 repo stars).
• llm-evaluation — Compare held-out and safety evaluation criteria (wshobson/agents, 42,437 repo stars).
• agentic-eval — Review repeatability, baselines, and regression gating (github/awesome-copilot, 41,073 repo stars).

How to check: In a project-specific copy, have the agent audit the frontend scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.