24 — AI Monitoring and Drift Checklist#

Verify that production AI systems are observable, that distribution changes are separated from demonstrated model-quality regressions, and that emerging user harm or operational failures trigger accountable responses. A drift alarm alone does not establish a decline in accuracy, fairness, or safety.

Scope: Backend / API-specific. This file is one part of the 24_AI_Monitoring_and_Drift.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

• Project / product:
• Release / version / environment:
• Scope and applicable jurisdictions / markets:
• Deployed model/provider, prompt, data, and monitoring revisions:
• Monitoring baselines, signal owners, and alert thresholds:
• User-harm, security, privacy, and incident escalation owners:
• Reviewer / owner:
• Review date:
• Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

1. Copy this reusable checklist into project-specific monitoring review records; leave the shared repository checkboxes unchecked.
2. Define meaningful production quality, risk, availability, latency, cost, and data-change indicators using the actual AI task and deployment environment. Distinguish feature skew, data drift, model-quality drift, and delayed or absent outcome labels.
3. Mark [x] only when verified using approved dashboards, trace samples, monitoring jobs, drift reports, ground-truth comparisons, alert tests, incident exercises, or qualified monitoring and governance decisions. Record evidence, failed checks, accepted exceptions, and justified N/A in project-specific notes/issues keyed by DRIFT ID; do not add evidence tables to this shared template.
4. Apply ## API Checks and ## Frontend Checks independently where the corresponding product interfaces exist; document why any surface does not apply.
5. Do not infer task correctness or safe impact from low drift scores or endpoint uptime. Sampling gaps, changing cohorts, delayed labels, and provider/model updates can invalidate simple comparisons.
6. Reassess baselines and alert thresholds after major data, population, model, prompt, retrieval, tool, provider, or regulatory changes. Coordinate incident response and evaluation with AI Governance, AI Evaluation and Regression, MLOps, and Monitoring and Observability checklists.

API Checks#

• DRIFT-093 Emit safe, correlatable request and operation IDs across inference, retrieval, tool calls, and external providers.
• DRIFT-094 Capture the actual serving model and relevant approved configuration revision without exposing internal secrets.
• DRIFT-095 Measure HTTP and provider errors, timeouts, rate limiting, refusal responses, and business validation failures separately.
• DRIFT-096 Instrument streaming time to first chunk, interrupted streams, completion rates, and malformed partial outputs where applicable.
• DRIFT-097 Measure inference token consumption and other billable units consistently with vendor-reported totals when available.
• DRIFT-098 Monitor safe schema and contract failures at API boundaries after model or prompt updates.
• DRIFT-099 Check that tool-call and agent-run telemetry distinguish proposed actions from authoritative completed external effects.
• DRIFT-100 Apply high-cardinality and sensitive-data restrictions to telemetry attributes, sampling, and logs.
• DRIFT-101 Test that instrumentation does not persist private prompts, retrieved documents, access tokens, or protected model output without explicit approval.
• DRIFT-102 Correlate delayed labels and confirmed outcomes to the original prediction or task using stable permitted IDs.
• DRIFT-103 Validate that API monitoring includes fallback routes, canary deployments, batch operations, and retries.
• DRIFT-104 Verify tenant isolation and role-based access for monitoring endpoints, traces, dashboards, and anomaly exports.
• DRIFT-105 Test monitoring behavior when a provider silently changes a model alias or serving version.
• DRIFT-106 Check input validation, response output-format compliance, and authorization signals alongside quality telemetry.
• DRIFT-107 Apply server-enforced cost, quota, and tool-loop limits so alerts are not the only control against runaway spend.
• DRIFT-108 Test whether monitoring survives API deployments, provider outages, queue backlogs, and instrumentation agent restarts.
• DRIFT-109 Expose safe operational health and monitoring-job status without leaking protected model internals.
• DRIFT-110 Ensure alert callbacks or remediation automation require appropriate authentication, authorization, and approval.
• DRIFT-111 Verify that telemetry retention, export, deletion, and geographic processing satisfy the actual data policy.
• DRIFT-112 Include model-specific latency, success, quality, and security signals in production API release checks.

Release gate#

• Critical AI user journeys, production components, responsible teams, and actual serving model/configuration versions have verifiable monitoring coverage.
• Availability, error, tail-latency, workload, cost, safety, and high-impact task-quality indicators have approved baselines, thresholds, and accountable escalation paths.
• Feature input quality, training-serving skew, output changes, and where feasible ground-truth quality are tracked with suitable reference populations, sampling, and freshness controls.
• Monitoring distinguishes observed distribution drift from confirmed quality degradation, accounts for missing or delayed labels, and identifies unreliable/inconclusive comparisons.
• Production privacy, security, access, sampling, cardinality, and retention controls protect telemetry, feedback, and evaluation records.
• Alerts and monitoring-failure detection have been exercised; critical incidents trigger tested containment, human response, emergency disablement, or model rollback as applicable.
• Relevant API and frontend monitoring detects errors, incomplete/unsafe outputs, user-visible failures, and problematic model or feature-rollout changes.
• A named risk/release approver has reviewed remaining monitoring blind spots, observation windows, accepted residual risk, and the plan for drift investigation and remediation.

Domain-specific approval rule: Block monitoring readiness when serious user harm or unauthorized tool effects cannot be detected or escalated appropriately, monitoring is materially blind on critical workflows, an essential safety guardrail can fail open unnoticed, or required privacy controls for monitored data are absent. Stable feature distributions and passing infrastructure health checks do not certify model quality or the safety of the user experience.

Trusted references and scope#

The following primary and authoritative sources informed the monitoring questions. Vendor-specific services and algorithms are examples rather than universal requirements.

• NIST AI RMF Playbook — Manage — post-deployment risk monitoring, impact tracking, escalation, and decommissioning.
• Google Cloud — Model Monitoring overview — training-serving feature skew, prediction drift, and supported data types.
• Google Cloud — Monitor feature skew and drift — reference datasets, analysis windows, sampling, alert thresholds, and monitoring jobs.
• Google Cloud — Feature attribution skew and drift — monitoring changes in feature contributions where model explanations are available.
• Evidently — Data drift — reference/current comparison, statistical tests, and missingness limitations.
• AWS — SageMaker Model Monitor availability change — legacy-service status, replacement monitoring architectures, delayed labels, and operational considerations.
• AWS — Model quality monitoring with delayed ground truth — prediction-to-label joining and quality monitoring in existing SageMaker Model Monitor deployments.
• Microsoft Foundry — Evaluation of deployed AI applications — production conversation and trace evaluation as complements to offline regression.
• OpenTelemetry — Generative AI semantic conventions (maintained repository) — GenAI spans, metrics, events, and model/tool metadata; schema and capture rules are version-specific.
• Google SRE — Alerting on SLOs — reliable user-impact indicators, actionable alerts, and error budgets.
• Prometheus — Alerting best practices — actionable alert names, metadata, and effective incident routing.
• OpenTelemetry — General metrics semantic conventions — metric-unit, naming, and cardinality conventions applicable to service telemetry.

Applicability: The NIST AI RMF is voluntary guidance. Statistical drift test behavior depends on baseline size, measurement type, missing values, thresholds, and cohort composition; drift may warrant investigation but is not proof of concept drift or degraded ground-truth performance. AWS states SageMaker Model Monitor is no longer open to new customers as of this review; its older model-quality documentation applies to existing deployments, and the availability-change page describes recommended alternatives. Google Cloud feature-skew/drift features depend on supported deployment types and configurations. In 2026, OpenTelemetry GenAI semantic conventions moved to the dedicated open-telemetry/semantic-conventions-genai repository; verify schema and instrumentation support before adopting fields. Monitor sensitive content only where approved and avoid exporting raw prompts, protected documents, or personally identifying labels by default.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

• Test focus: drift thresholds, sampling, alert routing, and inference telemetry; include API and service behavior where applicable.
• Cross-stack: Prometheus promtool + Grafana.
• Python: Evidently + pytest; for pytest-based suites, run python -m pytest -q.
• TypeScript: OpenTelemetry + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
• JavaScript: OpenTelemetry + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
• Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

• distributed-tracing — Check inference trace and context propagation (wshobson/agents, 42,437 repo stars).
• llm-evaluation — Check quality changes over time (wshobson/agents, 42,437 repo stars).
• agent-platform-eval-flywheel — Review continuous evaluation where Google Agent Platform is used (google/skills, 24,012 repo stars).

How to check: In a project-specific copy, have the agent audit the backend / API scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.