17 — MLOps and Model Lifecycle Checklist#

Verify that AI/ML experiments, data, models, approvals, deployment configurations, and retirement decisions are traceable and managed across the system lifecycle. Registry entries and passing offline tests are inputs to a release decision, not proof of safe real-world operation.

Scope: AI-specific. This file is one part of the 17_MLOps_and_Model_Lifecycle.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

• Project / product:
• Release / version / environment:
• Scope and applicable jurisdictions / markets:
• Training, evaluation, registry, and serving owners:
• Dataset, feature, pipeline, and model artifact versions:
• Approval, deployment, rollback, and monitoring policy:
• Reviewer / owner:
• Review date:
• Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

1. Copy this shared checklist into project-specific records; keep all repository template boxes unchecked.
2. Verify each item using version-controlled code, immutable data and model manifests, experiment-tracking records, registry and deployment audits, pipeline tests, monitoring evidence, and authorized approval decisions. Document evidence, failures, justified N/A, and risk exceptions outside the shared template, keyed by MLOPS ID.
3. Tailor the controls to the deployment type: internally trained models, fine-tuned or adapter models, hosted third-party snapshots, batch prediction jobs, online endpoints, and generative AI workloads have different lifecycle and observability needs.
4. Review ## API Checks and ## Frontend Checks separately where those surfaces exist; document why items do not apply for headless or offline workflows.
5. Re-run appropriate lineage, evaluation, contract, security, and lifecycle checks after changes to data, model versions, features, prompts, providers, deployment platforms, tools, or user populations. Use the separate model-evaluation, drift-monitoring, fine-tuning, and governance templates for deeper requirements.
6. Follow current provider/runtime documentation for concrete implementation details. Do not assume that a registry alias, approval tag, or model stage automatically enforces your organization's security or release policy.

Experiments and reproducibility#

• MLOPS-001 Define the model's intended task, business owner, supported use, and measurable performance and risk thresholds before starting experiments.
• MLOPS-002 Keep training, evaluation, and pipeline code under reviewed version control with traceable change history.
• MLOPS-003 Record the exact source revision, training entry point, execution pipeline, and runtime environment for each consequential experiment.
• MLOPS-004 Version data snapshots, sample-selection rules, feature transformations, labels, and split assignments consumed by each run.
• MLOPS-005 Capture model architecture, base checkpoint, tuning method, optimizer, hyperparameters, and configuration for every promoted candidate.
• MLOPS-006 Record random seeds, deterministic settings, accelerator/software libraries, and known sources of nondeterminism.
• MLOPS-007 Preserve task metrics, validation results, severe failures, and statistical uncertainty alongside model outputs.
• MLOPS-008 Record training and evaluation hardware type, runtime, utilization, costs, and resource limits for material experiments.
• MLOPS-009 Use an experiment tracker or equivalent immutable run records so results can be compared and audited.
• MLOPS-010 Assign globally unique run IDs and link child trials, hyperparameter searches, and resumed training jobs to their parents.
• MLOPS-011 Validate input schema, missing fields, corrupted examples, class distributions, and data quality before training begins.
• MLOPS-012 Check data-preparation and feature-pipeline tests with known-correct fixtures, including edge values and encoding.
• MLOPS-013 Prevent training-to-validation contamination and evaluation leakage in snapshots, feature stores, prompts, and augmented data.
• MLOPS-014 Test pipeline reproducibility by rerunning representative experiments against approved inputs and comparing expected tolerances.
• MLOPS-015 Distinguish reproducible computation from statistically identical nondeterministic model outcomes.
• MLOPS-016 Track preprocessing, tokenization, embedding, retrieval, and safety configurations that change model behavior where relevant.
• MLOPS-017 Store experiment artifacts and logs with scoped access and retention; exclude production secrets and unnecessary protected training examples.
• MLOPS-018 Validate checkpoint and artifact integrity before using resumed or transferred training outputs.
• MLOPS-019 Define how failed, interrupted, partial, or canceled runs are classified and excluded from approval calculations.
• MLOPS-020 Separate research experimentation resources and credentials from production training and inference systems.
• MLOPS-021 Publish a clear comparison of candidate versus baseline quality, cost, reliability, fairness, and other risk-relevant metrics.
• MLOPS-022 Document experiment owner, conclusions, data caveats, and go/no-go decision before a candidate is registered for release.

Model registries and lineage#

• MLOPS-023 Use a controlled registry or equivalent artifact catalog for every model or externally hosted snapshot approved for production.
• MLOPS-024 Assign immutable model version identifiers and maintain the exact artifact digest or provider revision where available.
• MLOPS-025 Track model provider, license, original base model, adapter or fine-tune, framework, and intended serving runtime.
• MLOPS-026 Link each registered model version to the experiment or training job that produced it.
• MLOPS-027 Record provenance for training, validation, and evaluation datasets and the transformations used to create them.
• MLOPS-028 Link each model to its evaluation suite, task scores, severe failures, risk decisions, and approval evidence.
• MLOPS-029 Store relevant model-card documentation, intended uses, constraints, and known limitations alongside the registry entry.
• MLOPS-030 Define registry ownership and role-based permission for publishing, reviewing, promoting, archiving, and deleting versions.
• MLOPS-031 Prevent unreviewed models or model aliases from being promoted by users without authorization.
• MLOPS-032 Verify that aliases or deployment pointers resolve to an approved immutable version at release time.
• MLOPS-033 Audit alias reassignment and registry edits with actor, timestamp, prior target, new target, and change reason.
• MLOPS-034 Check compatibility between model artifact format, serialization library, inference framework, runtime, and required hardware.
• MLOPS-035 Scan serialized artifacts and dependencies for unsafe deserialization paths or untrusted executable content before loading.
• MLOPS-036 Use signed provenance, hashes, or equivalent integrity controls for models and supporting build artifacts where warranted.
• MLOPS-037 Make feature schemas, tokenizer versions, prompt templates, output contracts, and serving configuration discoverable from the model version.
• MLOPS-038 Link feature store snapshots and materialized feature definitions to the registered model where that architecture is used.
• MLOPS-039 Track deployment location, endpoint version, traffic share, and active consumers for every released model.
• MLOPS-040 Keep release-critical prior model versions and compatible dependencies available for the agreed rollback window.
• MLOPS-041 Define data, artifact, registry, and audit retention separately, including required privacy and intellectual-property controls.
• MLOPS-042 Propagate correction or deletion requirements to copies, caches, re-exported artifacts, and downstream uses when applicable.
• MLOPS-043 Validate registry lookup, model download, and dependency resolution in a production-equivalent environment before promotion.
• MLOPS-044 Reconcile registry metadata with actual deployed model revisions rather than assuming metadata proves runtime state.

Promotion, deployment, and rollback#

• MLOPS-045 Define development, evaluation, staging, production, and retirement states using explicit approval policy, not only a registry label.
• MLOPS-046 Require a recorded qualified approver and current evaluation evidence for risk-defined model promotion.
• MLOPS-047 Gate training and deployment pipelines on data-schema tests, reproducible artifacts, and model-quality and safety acceptance results.
• MLOPS-048 Run model and pipeline integration tests against the actual feature, retrieval, tool, and inference dependencies.
• MLOPS-049 Verify container images, packages, model files, configuration, and build provenance match approved release artifacts.
• MLOPS-050 Check resource compatibility, model load time, memory headroom, accelerators, runtime versions, and warm-up behavior.
• MLOPS-051 Test online, batch, streaming, and asynchronous inference paths supported by the selected deployment architecture.
• MLOPS-052 Record model version, configuration, endpoint, feature flags, planned audience, and rollout window in a release manifest.
• MLOPS-053 Use staged traffic, shadow inference, canary, or blue-green deployment where risk and architecture warrant it.
• MLOPS-054 Define initial cohort, traffic promotion increments, observation periods, and stop criteria before rollout.
• MLOPS-055 Test p95/p99 latency, errors, throughput, capacity, and unit cost under representative production-like load.
• MLOPS-056 Verify output quality, refusal rates, drift indicators, and high-impact user outcomes against predeclared guardrails.
• MLOPS-057 Confirm model and associated prompt, tokenizer, schema, feature, retrieval, and safety versions remain compatible while mixed versions serve traffic.
• MLOPS-058 Check whether a rolled-back model can read data written by the new version and whether any migration is irreversible.
• MLOPS-059 Test traffic rollback, registry-pointer reversal, feature disablement, or safe non-AI fallback in a controlled environment.
• MLOPS-060 Define recovery options when models rely on mutable external providers or revisions that cannot be restored exactly.
• MLOPS-061 Verify failed or partial deployments do not leave orphaned endpoints, duplicate jobs, unbounded cost, or stale traffic routing.
• MLOPS-062 Use infrastructure-as-code or reproducible deployment configuration where applicable to avoid manual environment drift.
• MLOPS-063 Limit production promotion rights and protect credentials, model artifacts, and high-impact environment settings.
• MLOPS-064 Ensure rollout automation stops on failed health checks rather than silently expanding exposure.
• MLOPS-065 Record release decisions and actual serving versions and compare them with the approved manifest after deployment.
• MLOPS-066 Maintain model-specific on-call runbooks, alert routes, emergency disabling, and provider support contacts.
• MLOPS-067 Verify downstream API clients and frontend experiences remain compatible across staged rollouts and rollback.
• MLOPS-068 Define the stabilization period and accountable reviewer for final post-deployment acceptance.

Retraining and lifecycle governance#

• MLOPS-069 Assign lifecycle owners for model quality, data pipeline integrity, serving operations, regulatory review, and eventual decommissioning.
• MLOPS-070 Define retraining triggers based on observed drift, performance degradation, data changes, scheduled refresh, or business requirements.
• MLOPS-071 Distinguish label delay and data-availability lag from true concept or model-performance drift before triggering retraining.
• MLOPS-072 Keep retraining data selection, rights, consent, retention, and sensitive-data handling subject to current approved policies.
• MLOPS-073 Define whether retraining is automatic, manually approved, or prohibited for specific high-impact model categories.
• MLOPS-074 Ensure a retraining run does not automatically replace a production model without the required evaluation and promotion gate.
• MLOPS-075 Monitor incoming training data schema, source quality, class balance, distribution shift, and possible poisoning.
• MLOPS-076 Repeat necessary subgroup, safety, task, regression, and robustness evaluations for each new candidate.
• MLOPS-077 Compare retrained models against a known-good baseline using representative held-out and post-deployment cases.
• MLOPS-078 Record new data, label, algorithm, provider, and tuning changes so improvements and regressions are attributable.
• MLOPS-079 Test pipeline retries, interruption recovery, parallel training, and duplicate-trigger prevention to avoid uncontrolled costs.
• MLOPS-080 Maintain appropriate retraining compute quotas, budget alerts, and workload priority controls.
• MLOPS-081 Capture model predictions or authorized outcome signals needed for monitoring without retaining unnecessary protected content.
• MLOPS-082 Define monitoring metrics and escalation for input changes, output anomalies, severe failures, and business outcome degradation.
• MLOPS-083 Document incident workflows for unexpected bias, hallucination, leakage, unsafe agent behavior, or inappropriate autonomous actions.
• MLOPS-084 Retain a tested ability to quarantine a candidate, roll back a release, or disable a model after emerging harm.
• MLOPS-085 Review third-party model/provider deprecations, license changes, pricing changes, and data-processing policy updates.
• MLOPS-086 Define how feedback and user complaints become sanitized, versioned training or evaluation records after review.
• MLOPS-087 Update model cards, release notes, risk evaluations, and user disclosures after material changes.
• MLOPS-088 Periodically audit registry ownership, obsolete artifacts, inactive endpoints, access rights, and deployment costs.
• MLOPS-089 Define retirement triggers including unsupported use, expired rights, unmitigated risk, or declining operational value.
• MLOPS-090 Decommission models with a plan for endpoint shutdown, dependent clients, data retention, deletion, and required audit records.

Release gate#

• Critical model training and tuning runs are traceable to reviewed code, data, environment, configuration, evaluation, and identifiable accountable owners.
• The approved model artifact or provider snapshot has verifiable identity, provenance, integrity, permissioned registry metadata, and documented intended use and limitations.
• Quality, safety, privacy, performance, compatibility, and resource/cost thresholds have been evaluated against representative conditions and a known-good baseline.
• Authorized promotion gates validate the specific model and associated feature, schema, prompt, tool, and serving configurations; unapproved versions cannot be deployed.
• Production-equivalent deployment, staged rollout, mixed-version compatibility, endpoint health, and rollback or safe-disablement tests pass.
• Ongoing monitoring, incident escalation, user feedback, drift investigation, retraining approvals, and retention/decommissioning procedures have named owners.
• Relevant API and frontend behaviors remain safe and comprehensible across model changes, fallback, and retired capabilities.
• Residual material risks and unresolved failures are documented and approved by appropriately qualified humans for the exact release.

Domain-specific approval rule: Block sign-off when a production model cannot be traced to approved data and code, integrity or access controls permit unreviewed artifacts to be deployed, severe evaluation or privacy findings are unresolved, or rollback/emergency disablement is required but untested. Automatic retraining does not confer automatic deployment authorization.

Trusted references and scope#

These primary specifications, project documentation, and authoritative risk-management resources support the lifecycle checks. They describe different architectures and are not interchangeable certification requirements.

• Google Cloud Architecture Center — MLOps CI/CD/CT — automation, testing, metadata, continuous training, and serving lifecycle; emphasis on predictive ML.
• MLflow — Experiment Tracking — experiment run metadata, datasets, artifacts, metrics, parameters, and run comparison.
• MLflow — Model Registry — model versioning, registry ownership, aliases, lineage, and approval metadata.
• MLflow — Model Registry Workflows — promotion workflows and migration away from deprecated fixed model stages.
• Kubeflow Pipelines — ML Metadata — pipeline executions, artifact lineage, metadata, and operational migration considerations.
• Kubeflow Pipelines — ML Artifacts — typed datasets, model artifacts, provenance, and pipeline artifact lifecycle.
• TensorFlow Extended — ML Metadata — tracking relationships among data, training runs, model artifacts, and software dependencies.
• Amazon SageMaker AI — Model Registry — model package approvals, lineage, metadata, and controlled deployment.
• Amazon SageMaker AI — Pipelines Overview — processing, training, evaluation, condition gates, and model registration workflows.
• NIST AI RMF Playbook — Manage — human risk decisions, continual monitoring, recovery, vendor oversight, and decommissioning.
• SLSA v1.2 — Provenance — verifiable artifact build/source provenance relevant to model packaging and serving supply chains.
• OpenTelemetry — Generative AI semantic conventions — model, operation, token, and response observability with sensitivity safeguards.

Applicability: Google Cloud's MLOps architecture article principally covers predictive machine learning; generative AI may use provider-managed models rather than locally trained artifacts. MLflow fixed Staging/Production/Archived model stages are deprecated in favor of model aliases, tags, and suitable environment access control; a mutable alias must never replace an immutable release identity. Kubeflow, TensorFlow ML Metadata, and SageMaker capabilities depend on the deployed version and provider configuration. SLSA is supply-chain provenance guidance, not proof of permission to use training data. OpenTelemetry's GenAI conventions and implementation support may evolve; avoid logging protected prompt or response content without an approved policy. NIST AI RMF is voluntary and not a certification.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

• Test focus: lineage, promotion gates, artifacts, deployment, and rollback; include model and dataset behaviors where applicable.
• Cross-stack: DVC + MLflow.
• Python: Evidently + pytest; for pytest-based suites, run python -m pytest -q.
• TypeScript: Testcontainers + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
• JavaScript: Testcontainers + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
• Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

• ml-pipeline-workflow — Review training/deployment lifecycle controls (wshobson/agents, 42,437 repo stars).
• checkpoint-promotion — Check model approval and rollback gates (wshobson/agents, 42,437 repo stars).
• agent-platform-model-registry — Check registry provenance where Google Agent Platform is used (google/skills, 24,012 repo stars).

How to check: In a project-specific copy, have the agent audit the AI-specific scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.