23 — MLOps and Model Lifecycle Checklist#

Verify that AI/ML experiments, data, models, approvals, deployment configurations, and retirement decisions are traceable and managed across the system lifecycle. Registry entries and passing offline tests are inputs to a release decision, not proof of safe real-world operation.

Scope: Backend / API-specific. This file is one part of the 23_MLOps_and_Model_Lifecycle.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

• Project / product:
• Release / version / environment:
• Scope and applicable jurisdictions / markets:
• Training, evaluation, registry, and serving owners:
• Dataset, feature, pipeline, and model artifact versions:
• Approval, deployment, rollback, and monitoring policy:
• Reviewer / owner:
• Review date:
• Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

1. Copy this shared checklist into project-specific records; keep all repository template boxes unchecked.
2. Verify each item using version-controlled code, immutable data and model manifests, experiment-tracking records, registry and deployment audits, pipeline tests, monitoring evidence, and authorized approval decisions. Document evidence, failures, justified N/A, and risk exceptions outside the shared template, keyed by MLOPS ID.
3. Tailor the controls to the deployment type: internally trained models, fine-tuned or adapter models, hosted third-party snapshots, batch prediction jobs, online endpoints, and generative AI workloads have different lifecycle and observability needs.
4. Review ## API Checks and ## Frontend Checks separately where those surfaces exist; document why items do not apply for headless or offline workflows.
5. Re-run appropriate lineage, evaluation, contract, security, and lifecycle checks after changes to data, model versions, features, prompts, providers, deployment platforms, tools, or user populations. Use the separate model-evaluation, drift-monitoring, fine-tuning, and governance templates for deeper requirements.
6. Follow current provider/runtime documentation for concrete implementation details. Do not assume that a registry alias, approval tag, or model stage automatically enforces your organization's security or release policy.

API Checks#

• MLOPS-091 Expose inference, model metadata, and lifecycle endpoints through authenticated APIs with purpose-scoped permissions.
• MLOPS-092 Verify production API responses and diagnostic headers identify the serving model revision where appropriate without leaking sensitive configuration.
• MLOPS-093 Reject consumer requests that try to bypass approved routing, model versions, or safety policies.
• MLOPS-094 Validate request feature schemas, input lengths, modalities, context sizes, and output contract requirements before inference.
• MLOPS-095 Test authentication, authorization, and tenant isolation when selecting models, viewing experiment results, or invoking private endpoints.
• MLOPS-096 Protect registry and promotion APIs with separate rights from ordinary inference access.
• MLOPS-097 Require authorized approvals, invariant checks, and audit trails for model promotion, alias changes, and deactivation calls.
• MLOPS-098 Validate model package signatures, digests, and runtime compatibility before deploy operations accept artifacts.
• MLOPS-099 Test mixed-version inference clients and legacy consumers during canary and rollback.
• MLOPS-100 Define rate limits, budget ceilings, timeouts, bounded retries, and behavior under quota exhaustion.
• MLOPS-101 Measure model-load errors, inference tail latency, memory pressure, throughput, and provider failure rates by release.
• MLOPS-102 Check data and prediction schema compatibility between training features, feature stores, and online/batch inference.
• MLOPS-103 Validate retraining job triggers, input data version, deduplication, and scheduling permissions.
• MLOPS-104 Prevent user-supplied model identifiers or paths from causing unsafe artifact loading or untrusted code execution.
• MLOPS-105 Protect model weights, prompts, customer features, and prediction payloads from unintended logs or unapproved data transfers.
• MLOPS-106 Test fallback routing for unavailable providers, revoked versions, incompatible outputs, and policy updates.
• MLOPS-107 Ensure telemetry provides safe experiment, registry, deployment, and inference correlation IDs.
• MLOPS-108 Confirm canceled, failed, and partial training jobs cannot be mistaken for approved model packages.
• MLOPS-109 Verify model-retirement APIs invalidate serving routes, caches, and dependent scheduled jobs as designed.
• MLOPS-110 Run production-equivalent API contract tests against approved model promotion and rollback configurations.

Release gate#

• Critical model training and tuning runs are traceable to reviewed code, data, environment, configuration, evaluation, and identifiable accountable owners.
• The approved model artifact or provider snapshot has verifiable identity, provenance, integrity, permissioned registry metadata, and documented intended use and limitations.
• Quality, safety, privacy, performance, compatibility, and resource/cost thresholds have been evaluated against representative conditions and a known-good baseline.
• Authorized promotion gates validate the specific model and associated feature, schema, prompt, tool, and serving configurations; unapproved versions cannot be deployed.
• Production-equivalent deployment, staged rollout, mixed-version compatibility, endpoint health, and rollback or safe-disablement tests pass.
• Ongoing monitoring, incident escalation, user feedback, drift investigation, retraining approvals, and retention/decommissioning procedures have named owners.
• Relevant API and frontend behaviors remain safe and comprehensible across model changes, fallback, and retired capabilities.
• Residual material risks and unresolved failures are documented and approved by appropriately qualified humans for the exact release.

Domain-specific approval rule: Block sign-off when a production model cannot be traced to approved data and code, integrity or access controls permit unreviewed artifacts to be deployed, severe evaluation or privacy findings are unresolved, or rollback/emergency disablement is required but untested. Automatic retraining does not confer automatic deployment authorization.

Trusted references and scope#

These primary specifications, project documentation, and authoritative risk-management resources support the lifecycle checks. They describe different architectures and are not interchangeable certification requirements.

• Google Cloud Architecture Center — MLOps CI/CD/CT — automation, testing, metadata, continuous training, and serving lifecycle; emphasis on predictive ML.
• MLflow — Experiment Tracking — experiment run metadata, datasets, artifacts, metrics, parameters, and run comparison.
• MLflow — Model Registry — model versioning, registry ownership, aliases, lineage, and approval metadata.
• MLflow — Model Registry Workflows — promotion workflows and migration away from deprecated fixed model stages.
• Kubeflow Pipelines — ML Metadata — pipeline executions, artifact lineage, metadata, and operational migration considerations.
• Kubeflow Pipelines — ML Artifacts — typed datasets, model artifacts, provenance, and pipeline artifact lifecycle.
• TensorFlow Extended — ML Metadata — tracking relationships among data, training runs, model artifacts, and software dependencies.
• Amazon SageMaker AI — Model Registry — model package approvals, lineage, metadata, and controlled deployment.
• Amazon SageMaker AI — Pipelines Overview — processing, training, evaluation, condition gates, and model registration workflows.
• NIST AI RMF Playbook — Manage — human risk decisions, continual monitoring, recovery, vendor oversight, and decommissioning.
• SLSA v1.2 — Provenance — verifiable artifact build/source provenance relevant to model packaging and serving supply chains.
• OpenTelemetry — Generative AI semantic conventions — model, operation, token, and response observability with sensitivity safeguards.

Applicability: Google Cloud's MLOps architecture article principally covers predictive machine learning; generative AI may use provider-managed models rather than locally trained artifacts. MLflow fixed Staging/Production/Archived model stages are deprecated in favor of model aliases, tags, and suitable environment access control; a mutable alias must never replace an immutable release identity. Kubeflow, TensorFlow ML Metadata, and SageMaker capabilities depend on the deployed version and provider configuration. SLSA is supply-chain provenance guidance, not proof of permission to use training data. OpenTelemetry's GenAI conventions and implementation support may evolve; avoid logging protected prompt or response content without an approved policy. NIST AI RMF is voluntary and not a certification.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

• Test focus: lineage, promotion gates, artifacts, deployment, and rollback; include API and service behavior where applicable.
• Cross-stack: DVC + MLflow.
• Python: Evidently + pytest; for pytest-based suites, run python -m pytest -q.
• TypeScript: Testcontainers + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
• JavaScript: Testcontainers + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
• Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

• ml-pipeline-workflow — Review training/deployment lifecycle controls (wshobson/agents, 42,437 repo stars).
• checkpoint-promotion — Check model approval and rollback gates (wshobson/agents, 42,437 repo stars).
• agent-platform-model-registry — Check registry provenance where Google Agent Platform is used (google/skills, 24,012 repo stars).

How to check: In a project-specific copy, have the agent audit the backend / API scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.