08 — AI Fine-tuning and Training Checklist#

Verify that training or fine-tuning a model is justified, reproducible, permissioned, numerically stable, evaluated for unintended behavioral changes, and released under explicit control. Lower training loss or a successful provider job alone does not establish model quality, safety, data rights, or deployment readiness.

Scope: AI-specific. This file is one part of the 08_AI_FineTuning_and_Training.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

• Project / product:
• Release / version / environment:
• Scope and applicable jurisdictions / markets:
• Training objective, base model, adaptation method, and intended use:
• Data / split / preprocessing / tokenizer versions and processing rights:
• Training code, environment, hyperparameters, checkpoint, and evaluator versions:
• ML engineering / data / risk / security / release approvers:
• Review date:
• Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

1. Make a project-specific copy of this shared checklist; leave the repository's reusable checkboxes unchecked.
2. Decide whether fine-tuning is needed by comparing the expected benefit with a baseline, prompt or retrieval improvements, external inference, and the full training/serving cost. Scope checks to the actual model type, adaptation method, modality, training provider, and deployment.
3. Verify applicable checks using versioned training data, dataset rights reviews, training recipes, instrumentation, checkpoint recovery tests, held-out evaluations, registry artifacts, and approved deployment records. Track failures, evidence, justified N/A, and risk acceptances in project-specific records keyed by TRAIN ID, not in this shared template.
4. Keep ## API Checks and ## Frontend Checks separate. Hosted training, offline training, adapter-only tuning, and models without an end-user interface have different applicability; document what does not apply and why outside this file.
5. Validate training data formatting, response-token loss masking, checkpoint compatibility, and exact base/adapter revisions against the chosen framework. Techniques such as LoRA, SFT, DPO, mixed precision, packing, and distributed checkpointing are options, not universal mandates.
6. Reassess training and release gates when source rights, data distributions, model/provider versions, tuning objectives, label rules, evaluation results, training environment, or serving integrations change. Use the separate AI Data Curation, Model Selection, AI Evaluation, MLOps, Model Cards, and AI Red Teaming checklists for deeper domain reviews.

Training objectives and configuration#

• TRAIN-001 Define the intended task, affected users, observable improvement target, and reason training is preferable to prompting, retrieval, or workflow changes.
• TRAIN-002 Identify the exact approved base model or checkpoint, revision, license, provider restrictions, and permitted fine-tuning or redistribution uses.
• TRAIN-003 Choose full fine-tuning, supervised fine-tuning, adapter-based training, preference optimization, or another approach based on task evidence and resources.
• TRAIN-004 Document explicit quality, safety, privacy, cost, latency, and deployment acceptance criteria before starting training.
• TRAIN-005 Choose a non-tuned or current-production baseline and maintain comparable evaluation settings for the candidate.
• TRAIN-006 Record the training objective, loss function, expected label semantics, and any reward or preference signals used.
• TRAIN-007 Specify the trainable parameters, frozen modules, adapter targets, LoRA rank and scaling, or equivalent method settings when applicable.
• TRAIN-008 Verify a parameter-efficient adapter is compatible with the precise base model, tokenizer, architecture, and inference runtime.
• TRAIN-009 Document optimizer, learning-rate schedule, warm-up, regularization, weight decay, gradient accumulation, and clipping settings.
• TRAIN-010 Set effective batch size, sequence length, padding, truncation, example packing, and sampling policies with explicit rationale.
• TRAIN-011 Verify conversational training uses the correct role and chat templates for the target model and release.
• TRAIN-012 Check that loss masking trains on intended assistant/completion tokens rather than inadvertently optimizing prompts or protected context.
• TRAIN-013 Define how tool-call formats, special tokens, structured outputs, and end-of-turn markers are represented in training examples.
• TRAIN-014 Set validation cadence, checkpoint selection criterion, early-stop policy, and maximum training steps or epochs.
• TRAIN-015 Define numerical precision, quantization, distributed training, and accelerator-memory settings supported by the actual runtime.
• TRAIN-016 Validate mixed-precision and quantized training with checks for NaNs, gradient underflow/overflow, and changed task quality.
• TRAIN-017 Estimate compute, GPU memory, storage, checkpoint volume, run duration, and total training cost against approved limits.
• TRAIN-018 Specify distributed worker configuration, sharding, accumulation, optimizer state, and any expected scaling bottlenecks.
• TRAIN-019 Review training-job identities, network egress, secret handling, approved artifact destinations, and access to private datasets.
• TRAIN-020 Plan training-risk mitigations for overfitting, catastrophic forgetting, reward hacking, and unexpected behavioral changes.
• TRAIN-021 Assign responsible ML engineering, evaluation, data rights, security, and deployment approvers.
• TRAIN-022 Version and freeze the approved training configuration, base artifact, and environment before starting a release-candidate run.

Training data and splits#

• TRAIN-023 Confirm training and preference data have approved provenance, licensing, consent, and permitted model-development uses.
• TRAIN-024 Link each training run to immutable or versioned datasets, source manifests, transformation code, and labeling guidelines.
• TRAIN-025 Check training records against required schemas, field types, role formats, media metadata, and sample-size constraints.
• TRAIN-026 Verify target labels, assistant completions, selected/rejected preference pairs, and scoring rubrics match the chosen objective.
• TRAIN-027 Ensure preference datasets preserve the same prompt context for chosen and rejected outputs where the method requires it.
• TRAIN-028 Detect missing responses, inconsistent roles, malformed conversations, corrupted media, and invalid token sequences before training.
• TRAIN-029 Validate examples retain relevant task context and that truncation does not delete necessary answers or safety qualifiers.
• TRAIN-030 Measure token-length distributions, language and task frequencies, label balance, and outlier examples before scheduling runs.
• TRAIN-031 Prevent training, validation, and held-out evaluation overlap across near-duplicates, translations, source documents, users, and synthetic variants.
• TRAIN-032 Use group-based or temporal splits when related records would make random row splits misleading.
• TRAIN-033 Protect the final release-evaluation holdout from data selection, prompt tuning, hyperparameter search, and training feedback.
• TRAIN-034 Check that known benchmark questions or private grading answers have not leaked into fine-tuning examples.
• TRAIN-035 Review sample weighting, class rebalance, filtering, and oversampling for unintended distribution or fairness effects.
• TRAIN-036 Audit model-generated synthetic training examples for factual errors, unsafe content, memorization, and copied private records.
• TRAIN-037 Check the quality and consistency of human labels and adjudicate ambiguous or high-impact cases before training.
• TRAIN-038 Verify inclusion of task-relevant refusals, clarifications, boundary cases, and policy-consistent behavior where appropriate.
• TRAIN-039 Review model behavior risks introduced by demonstrations containing unsafe instructions, privileged tool calls, or untrusted document text.
• TRAIN-040 Measure potential overrepresentation of narrow sources, annotator preferences, dialects, or demographic proxies where lawful and relevant.
• TRAIN-041 Apply necessary minimization or redaction to personal data, credentials, confidential business records, and protected identifiers.
• TRAIN-042 Define how source deletion, license withdrawal, or consent changes are propagated to future training runs and affected model artifacts.
• TRAIN-043 Keep private datasets and golden evaluations access-controlled; prevent accidental publication to experiment dashboards or model hubs.
• TRAIN-044 Document augmentation, example packing, shuffling, sampling seeds, and preprocessing version so the training input can be reconstructed.
• TRAIN-045 Validate tokenization, attention masks, loss masks, and data collators on a small known-correct fixture before large training.
• TRAIN-046 Record excluded records, their exclusion reasons, unresolved data quality issues, and any accepted representation limitations.

Checkpoints and reproducibility#

• TRAIN-047 Assign stable run identifiers linking code revision, data version, base weights, training configuration, and execution environment.
• TRAIN-048 Record library versions, container image digests, accelerator models, driver/runtime versions, and distributed-training topology.
• TRAIN-049 Set and capture random seeds and determinism controls supported by the framework while documenting limits across devices and releases.
• TRAIN-050 Test that a small training run can be repeated within realistic numerical and statistical tolerances.
• TRAIN-051 Configure loss, validation metrics, learning rate, gradient norms, memory usage, and throughput logging at useful intervals.
• TRAIN-052 Detect NaN or infinite loss, diverging gradients, unstable reward metrics, and unexpected learning-rate schedule changes.
• TRAIN-053 Monitor train-versus-validation divergence and define investigation criteria for overfitting or data contamination.
• TRAIN-054 Validate automatic mixed precision, gradient scaling where applicable, and numerical stability for the selected hardware.
• TRAIN-055 Choose checkpoint cadence based on data loss tolerance, expected preemption, storage cost, and recovery time.
• TRAIN-056 Persist the necessary model or adapter weights, optimizer state, scheduler state, step counters, and random-number states for exact supported resumption.
• TRAIN-057 Distinguish inference-only weight exports from resumable training checkpoints in names and metadata.
• TRAIN-058 Test interruption and resume from a representative checkpoint and compare resulting run behavior with expected tolerances.
• TRAIN-059 Verify partial, corrupted, or incomplete checkpoints cannot be promoted or silently selected for resumption.
• TRAIN-060 Validate distributed sharded checkpoint restore with the selected training topology and supported framework version.
• TRAIN-061 Test checkpoint restore when storage, rank count, node placement, or topology changes if that recovery mode is supported.
• TRAIN-062 Check model/adaptor save and load compatibility with its exact base revision and required special-token configuration.
• TRAIN-063 Record model artifact checksums or cryptographic digests and protect training outputs against unauthorized overwrites.
• TRAIN-064 Restrict access to checkpoints because weights, optimizer state, and metadata may contain sensitive information.
• TRAIN-065 Ensure training artifact storage, retention, and deletion satisfy applicable data policy and contractual constraints.
• TRAIN-066 Test job timeouts, worker loss, transient network failures, and storage quota exhaustion with safe resource cleanup.
• TRAIN-067 Prevent duplicate training jobs or repeated scheduled runs from silently replacing reviewed artifacts or exhausting compute budgets.
• TRAIN-068 Keep traceable history of experiment failures, restart reasons, configuration changes, and checkpoint selection rationale.
• TRAIN-069 Document the final candidate's exact artifact, adapter composition, source run, and evaluation checkpoint identifier.
• TRAIN-070 Verify the archived training recipe can regenerate a functionally comparable candidate under its documented constraints.

Post-training evaluation and deployment#

• TRAIN-071 Compare the trained candidate with the unchanged approved baseline on representative, held-out, task-specific evaluations.
• TRAIN-072 Verify improvement in the intended target task is meaningful and not confined to the training examples.
• TRAIN-073 Measure regressions on unrelated but important baseline capabilities to detect catastrophic forgetting.
• TRAIN-074 Evaluate safety refusals, legitimate-request false blocks, unsupported assertions, and harmful outputs after tuning.
• TRAIN-075 Test unauthorized tool use, prompt-injection susceptibility, private-data exposure, and policy compliance where relevant.
• TRAIN-076 Assess structured output, tool-call argument validity, role semantics, and required response formats against production contracts.
• TRAIN-077 Measure relevant language, cohort, dialect, and domain performance differences with adequate samples and permitted data use.
• TRAIN-078 Test memorization or unintended reproduction of private training examples using controlled, authorized fixtures.
• TRAIN-079 Report confidence intervals, repeatability limits, and severe failures rather than relying solely on one aggregate quality score.
• TRAIN-080 Validate downstream RAG, agents, multimodal processing, and human-approval workflows when the fine-tuned model participates in them.
• TRAIN-081 Benchmark serving latency, model/adapter load time, throughput, memory, and inference cost against approved budgets.
• TRAIN-082 Test inference using the precise model weights, adapter, tokenizer, chat template, and quantization settings selected for deployment.
• TRAIN-083 Ensure the release model card captures training origin, intended uses, dataset limits, evaluation evidence, and known safety restrictions.
• TRAIN-084 Register the candidate with immutable artifact/version identifiers, traceable lineage, and an auditable approval status.
• TRAIN-085 Require a qualified human to review critical quality, safety, privacy, and licensing findings before promotion.
• TRAIN-086 Verify no automatic training job or model-registry alias can deploy an unreviewed checkpoint to production.
• TRAIN-087 Stage rollout using controlled traffic or a testing environment appropriate to impact, with predeclared stop criteria.
• TRAIN-088 Test compatibility with the current API response formats, stored conversations, client versions, and fallback models.
• TRAIN-089 Maintain an approved previous version or safe non-AI fallback and test model/adapter rollback.
• TRAIN-090 Monitor post-release task performance, safety blocks, token/cost changes, drift, and user complaints by model revision.
• TRAIN-091 Define triggers for retuning, retraining, revocation, deprecation, or emergency disablement based on observed harm or poor quality.
• TRAIN-092 Record release decision, risk acceptances, owners, deployment manifest, and post-launch review date.

Release gate#

• The training purpose, permitted base-model use, accountable owners, chosen method, expected improvement, and material privacy/licensing restrictions have been reviewed.
• Versioned datasets, preprocessing, labels and preference examples pass defined quality checks; evaluation holdouts are appropriately separated and contamination assessed.
• The exact training job has a reviewed configuration, identifiable base checkpoint, correct loss/format semantics, approved resource budgets, numerical stability evidence, and reproducible lineage.
• Candidate weights or adapters, checkpoints, and artifact metadata are integrity-checked; required resume and failure-recovery paths are tested for the training environment.
• Held-out task and baseline comparisons meet predeclared quality/safety criteria, with checks for catastrophic forgetting, memorization, behavior regression, and relevant cohort differences.
• API/inference and frontend compatibility have been evaluated for the exact tuned model, including schema validity, latency, costs, fallback, safety, and user-visible limitations as relevant.
• Registry, model card, deployment approval, staged rollout, monitoring, and rollback or emergency disablement are ready for the approved artifact version.
• No unaccepted high-severity privacy, data-rights, unsafe-content, or unauthorized-action failure remains; a qualified human has signed off on the specific release and accepted residual risks.

Domain-specific approval rule: Block model promotion when material training data lack permitted use, evaluation holdouts are compromised, the approved base/adapter artifact cannot be identified or restored, the candidate shows serious unmitigated behavior or safety regression, or deployment bypasses required authorization. A low training loss, efficient fine-tuning method, or model-card publication is not an independent release approval.

Trusted references and scope#

The following primary maintainer documentation and authoritative guidance inform these training-related checks. Methods and configuration flags are conditional on the exact framework and release in use; they are not mandatory for every AI system.

• Hugging Face TRL — Supervised Fine-Tuning Trainer — conversational/prompt-completion data, masking, packing, and supervised training configuration.
• Hugging Face TRL — Direct Preference Optimization Trainer — chosen/rejected preference pair formats, reference policies, and preference optimization.
• Hugging Face PEFT — LoRA — adapter-based parameter-efficient fine-tuning, base-model compatibility, and configuration.
• Hugging Face Transformers — Trainer features — checkpoints, resumable training, optimizer and random state, and interruptions.
• Hugging Face Datasets — File Names and Splits — separating and documenting training, validation, and test data.
• Hugging Face — Dataset Cards — source rights, dataset description, known limitations, and versioned metadata.
• Hugging Face — Model Cards — model intended use, provenance, training and evaluation disclosures.
• PyTorch — Reproducibility — randomness controls and limits of reproducibility across framework/device versions.
• PyTorch — Automatic Mixed Precision — mixed precision, gradient scaling where applicable, and numerical stability.
• PyTorch — Distributed Checkpoint — sharded and distributed state saving, restoration, and topology requirements.
• Amazon SageMaker AI — Training Output Artifacts — managed training artifact output and separation of final model from other job data.
• Google Cloud Architecture Center — MLOps CI/CD/CT — reproducible pipelines, model evaluation, controlled release, and continuous training.
• NIST AI RMF Playbook — Measure — trustworthiness evaluations, dataset splits, robustness, representativeness, and ongoing monitoring.

Applicability: Hugging Face TRL SFT and preference trainers vary across versions in chat-template support, loss masking, packing semantics, and trainer defaults; inspect actual deployed code and dataset format. LoRA/PEFT adapters depend on the exact base model and are not universally interchangeable. PyTorch documents that complete reproducibility cannot be guaranteed across releases and hardware even with identical random seeds. Restoring training exactly may require optimizer, scheduler and RNG states beyond model weights; distributed checkpoints have topology and compatibility constraints. Mixed precision benefits and numerical risks depend on hardware and dtype. AWS SageMaker output behavior applies only to that managed training platform; Google Cloud MLOps examples do not make its architecture universal. NIST AI RMF practices are voluntary guidance, not certification.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

• Test focus: reproducible runs, training splits, checkpoints, and post-training quality; include model and dataset behaviors where applicable.
• Cross-stack: MLflow + DVC.
• Python: Transformers Trainer + pytest; for pytest-based suites, run python -m pytest -q.
• TypeScript: promptfoo + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
• JavaScript: promptfoo + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
• Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

• checkpoint-promotion — Review promotion conditions and rollback (wshobson/agents, 42,437 repo stars).
• eval-harness-first — Check pre/post-training evaluation (wshobson/agents, 42,437 repo stars).
• dataset-curation — Review training-data quality and split hygiene (wshobson/agents, 42,437 repo stars).

How to check: In a project-specific copy, have the agent audit the AI-specific scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.