awesomechecklists
llms.txtGitHub ↗
/

06 — AI Data Curation and Labeling Checklist#

Verify that datasets used for AI training, tuning, evaluation, or retrieval are lawfully and responsibly sourced, technically consistent, properly labeled, traceable to origin, and separated to avoid misleading evaluation. A large dataset or high annotator agreement alone does not establish correctness or representativeness.

Scope: AI-specific. This file is one part of the 06_AI_Data_Curation_and_Labeling.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

  • Project / product:
  • Release / version / environment:
  • Scope and applicable jurisdictions / markets:
  • Dataset purpose, sources, data stewards, and intended users:
  • Dataset / labeling-guideline / transformation versions:
  • Applicable privacy, licensing, and retention rules:
  • Reviewer / owner:
  • Review date:
  • Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

  1. Make a project-specific copy of this reusable template; keep the repository checkboxes unchecked.
  2. Verify applicable items from permitted-source registers, dataset manifests and cards, privacy or licensing reviews, quality reports, annotation guidelines, agreement/adjudication results, and reproducible dataset builds. Record evidence, failures, justified N/A, and risk acceptances outside this shared template, keyed by CUR ID.
  3. Scope each check to the actual data source, modality, task, jurisdiction, labeling method, and intended consumers. Some products use unlabeled pretraining data, synthetic examples, weak labels, or externally curated evaluation sets; do not assume every label or review workflow applies.
  4. Keep ## API Checks and ## Frontend Checks separate. Record N/A and a rationale in project records when a surface does not exist.
  5. Reassess after adding new data sources, changing acquisition rights, collection populations, cleaning rules, label taxonomies, split strategies, processors, or deployment use cases.
  6. Coordinate specialist privacy, AI evaluation, model cards, training, and monitoring review with related repository checklists. Source rights and personal-data obligations require qualified legal/privacy review where relevant.

Dataset sourcing and rights#

  • CUR-001 Define each dataset's intended model task, target population, owners, and permitted training, tuning, and evaluation uses.
  • CUR-002 Inventory each original data source, supplier, collection method, acquisition date, and source-of-truth location.
  • CUR-003 Record provenance at a level that can trace a released example back to its permitted source and transformation history.
  • CUR-004 Verify licensing and terms explicitly permit the contemplated machine-learning, derivative, redistribution, and commercial uses.
  • CUR-005 Review contractual restrictions on scraping, retention, transfer, benchmarking, sharing, and downstream model outputs where relevant.
  • CUR-006 Determine applicable lawful basis, notice, consent, and other privacy obligations with qualified reviewers before using personal data.
  • CUR-007 Document any sensitive or regulated attributes and limit access and processing to what the specific use case requires.
  • CUR-008 Assess the rights and consent associated with identifiable image, voice, video, location, and biometric data where applicable.
  • CUR-009 Distinguish publicly accessible content from content actually authorized for collection or model development.
  • CUR-010 Record geographic, age-group, language, and temporal coverage and compare it with the intended deployment audience.
  • CUR-011 Identify affected or underrepresented communities and include relevant domain stakeholders in source selection.
  • CUR-012 Define sampling inclusion and exclusion criteria before collecting or curating data where feasible.
  • CUR-013 Quantify selection bias risks from collection channels, demographics, opt-in patterns, and device or network access.
  • CUR-014 Inspect sources for known quality problems, unreliable labels, synthetic material, spam, and adversarial contamination.
  • CUR-015 Define required provenance and integrity checks for data received from vendors, annotators, and third-party datasets.
  • CUR-016 Validate external transfer, provider retention, geographic storage, and subprocessor arrangements before supplying data to labeling or model vendors.
  • CUR-017 Assign an owner for source corrections, takedowns, rights challenges, and consent withdrawals.
  • CUR-018 Define how an erasure or source revocation reaches raw datasets, label stores, caches, exports, and derived model-training artifacts where required.
  • CUR-019 Use documented collection minimization and an approved retention schedule rather than keeping all source data indefinitely.
  • CUR-020 Restrict access to original sensitive sources and prevent accidental disclosure via dataset cards, examples, or shared repositories.
  • CUR-021 Document known limitations, collection gaps, consent-related biases, and suitability restrictions in a maintained dataset description.
  • CUR-022 Approve source additions or major licensing, population, and purpose changes before adding them to released training or evaluation data.

Data quality and preparation#

  • CUR-023 Define a versioned data schema, expected types, nullability, valid ranges, and allowed categorical values for every relevant field.
  • CUR-024 Verify ingest completeness with record counts, source IDs, checksums, and expected update windows where available.
  • CUR-025 Detect invalid encodings, truncated documents, unreadable media, corrupted records, and parser exceptions before labeling.
  • CUR-026 Apply documented normalization of text, Unicode, timestamps, identifiers, units, and other format-dependent features.
  • CUR-027 Record preprocessing order and parameters so curated datasets can be rebuilt reproducibly.
  • CUR-028 Measure missingness, outliers, duplicates, distribution shape, and class balance for relevant features and labels.
  • CUR-029 Detect exact and near-duplicate examples across sources before training and split assignment.
  • CUR-030 Preserve meaningful signals while removing or flagging spam, mislabeled, harmful, or irrelevant records.
  • CUR-031 Review whether cleaning rules disproportionately exclude specific languages, dialects, accessibility cases, or population groups.
  • CUR-032 Verify transformations do not silently strip negation, qualifiers, dialect features, units, or context needed for correct labels.
  • CUR-033 Define special handling for low-resource languages, scripts, mixed-language text, code-switching, and non-Latin inputs as applicable.
  • CUR-034 Validate document and multimedia preprocessing preserves speaker, timing, coordinate, or page-reference metadata where important.
  • CUR-035 Screen for unnecessary credentials, personal identifiers, protected fields, and prohibited confidential source content.
  • CUR-036 Document redaction and pseudonymization methods and test that supposedly removed sensitive information cannot still be recovered trivially.
  • CUR-037 Assess the privacy and fidelity of synthetic or augmented examples rather than treating synthetic data as automatically anonymous.
  • CUR-038 Keep source, human-generated, model-generated, and augmented records distinguishable for bias and contamination analysis.
  • CUR-039 Test that data augmentation does not change intended semantics or replicate protected source records beyond permitted uses.
  • CUR-040 Check training, evaluation, and serving input feature schemas for mismatches and missing required values.
  • CUR-041 Compare representative curated data distributions with the expected deployment distribution, including relevant time and segment changes.
  • CUR-042 Track anomaly thresholds, false alarms, owner decisions, and exceptions for data-quality checks.
  • CUR-043 Test batch and incremental curation for deterministic or otherwise documented behavior across reruns.
  • CUR-044 Version scripts, dependencies, extraction settings, feature definitions, and transformation outputs alongside curated datasets.
  • CUR-045 Quarantine or reject records with material unresolvable errors instead of silently guessing their content or labels.
  • CUR-046 Maintain data-quality dashboards or reports with row counts, missing fields, drift, duplicates, and source-specific failure rates.

Labeling and adjudication#

  • CUR-047 Define the annotation unit and clear label taxonomy, including mutually exclusive, multi-label, hierarchical, or span-based cases.
  • CUR-048 Write task-specific guidelines with positive and negative examples, boundary cases, and consistent decision criteria.
  • CUR-049 Include explicit labels or procedures for uncertainty, ambiguity, insufficient evidence, and genuinely unlabelable examples.
  • CUR-050 Describe how annotators should treat mixed-language, dialectal, culturally dependent, and domain-specific input.
  • CUR-051 Specify how annotators should handle subjective labels, value judgments, and disputed or context-sensitive categories.
  • CUR-052 Validate labeling UI and instructions with representative annotators before scaling collection.
  • CUR-053 Recruit or qualify annotators with language, domain, cultural, and accessibility expertise appropriate to the task.
  • CUR-054 Provide paid or otherwise approved training, calibration, and feedback consistent with applicable labor and contractual rules.
  • CUR-055 Restrict annotator access to only the data needed, and train workers on sensitive-data handling and reporting.
  • CUR-056 Design ergonomic, accessible labeling tools and safe support procedures for exposure to potentially distressing content.
  • CUR-057 Specify task assignment, randomization, overlap, and independence rules to reduce systematic annotator bias.
  • CUR-058 Create expert-reviewed gold or calibration items where the task permits an objectively defensible reference.
  • CUR-059 Measure intra-annotator and inter-annotator consistency using a metric appropriate to label type and chance agreement.
  • CUR-060 Review agreement by label, source, language, demographic proxy where lawful, and difficulty instead of relying on one global score.
  • CUR-061 Investigate low agreement as a possible ambiguity or guideline defect rather than always blaming annotators.
  • CUR-062 Define escalation and independent adjudication for disagreements, rare cases, and high-impact labels.
  • CUR-063 Record adjudication decisions and rationale so confusing guideline cases can be clarified.
  • CUR-064 Version annotation guidelines and document whether a guideline change requires affected labels to be reworked.
  • CUR-065 Track labeling corrections, revisions, annotator identifiers or permitted pseudonyms, timestamps, and approval state.
  • CUR-066 Use targeted double review, audit sampling, or expert review for high-impact and low-confidence labels.
  • CUR-067 Evaluate pre-labels and model-assisted annotation for anchoring bias, automation bias, and compounding model errors.
  • CUR-068 Check that annotators can correct or reject machine suggestions rather than being forced to approve them.
  • CUR-069 Monitor labeling throughput and error rates without trading away the quality necessary for the stated use.
  • CUR-070 Verify labeled exports preserve sample IDs, label definitions, geometry/span references, adjudication status, and provenance.

Splits, leakage, and provenance#

  • CUR-071 Define training, validation, test, and any holdout or temporal split policy before tuning the model.
  • CUR-072 Partition by the appropriate independent unit, such as person, account, document, subject, device, or time window.
  • CUR-073 Prevent the same user, source document, event family, or near-duplicate content from leaking across independent splits.
  • CUR-074 Use group-based or time-based splits for correlated records when random row splits would inflate reported performance.
  • CUR-075 Keep a final holdout evaluation set separate from prompt tuning, labeling-rule adjustments, and model selection decisions.
  • CUR-076 Audit train/evaluation overlap after augmentation, translation, synthetic generation, and near-duplicate expansion.
  • CUR-077 Track contamination risks when benchmark questions, answers, or evaluation examples become available to model training.
  • CUR-078 Recheck splits for new data batches and changed entity identity rules rather than assuming historical partitions remain safe.
  • CUR-079 Assess whether each split contains enough critical classes, languages, scenarios, and relevant groups to evaluate the intended system.
  • CUR-080 Document when stratification or oversampling alters evaluation prevalence and report limitations when interpreting metrics.
  • CUR-081 Record dataset version, manifest, sample IDs, split membership, checksums, and relevant source snapshot identifiers.
  • CUR-082 Version labeling instructions, adjudication changes, schemas, and preprocessing code with their resulting dataset artifacts.
  • CUR-083 Ensure a model run can identify the exact dataset and transformation revisions used for training and evaluation.
  • CUR-084 Maintain end-to-end lineage from acquired source through cleaning, labeling, split assignment, and exported training records.
  • CUR-085 Control who can modify a frozen validation or holdout set, and audit authorized revisions to it.
  • CUR-086 Test export/import and storage integrity with checksums or equivalent consistency validation where appropriate.
  • CUR-087 Define how corrected labels propagate to downstream versions without silently changing prior benchmark results.
  • CUR-088 Remove withdrawn, revoked, or expired records from future dataset releases and assess whether prior model artifacts need remediation.
  • CUR-089 Publish a dataset card or equivalent description covering creation, composition, labels, intended use, licensing, limitations, and risks.
  • CUR-090 Report known gaps in data provenance, subgroup coverage, and uncertain ground truth as limitations rather than hiding them.
  • CUR-091 Establish a periodic review for dataset drift, obsolescence, label-taxonomy changes, and new source restrictions.
  • CUR-092 Require responsible owner approval and an auditable change record before publishing a training or evaluation dataset release.

Release gate#

  • The dataset's intended use, source inventories, stewardship, relevant licensing and privacy rights, and prohibited uses have been reviewed by accountable owners.
  • Collection scope, sensitive fields, retention, vendor processing, deletion or withdrawal mechanisms, and access restrictions are approved for the actual use case.
  • Data-quality checks cover critical schema fields, record counts, duplicates, corrupt examples, preprocessing integrity, and relevant population/language representation.
  • Human or machine-assisted labeling follows versioned guidelines, with measured agreement or quality review and resolved high-impact ambiguities where labels are used.
  • Training, validation, test, and holdout split rules have been reviewed for entity, temporal, near-duplicate, and synthetic/augmentation leakage that could invalidate evaluation.
  • Dataset and label versions have reconstructable provenance, controlled changes, accessible documentation of limitations, and verified export integrity.
  • Relevant API and frontend workflows protect annotator and source data, preserve review states, and prevent unauthorized releases or overwritten labels.
  • Severe unresolved rights, privacy, contamination, bias, or label-integrity risks have been remediated or explicitly dispositioned by qualified human approvers before dataset use.

Domain-specific approval rule: Block dataset-release sign-off when collection/use rights are unknown for material records, restricted personal data is exposed, severe unresolved labeling defects undermine the target task, evaluation data leaks into training, or required source deletion cannot be honored within accepted controls. No dataset documentation or quality score guarantees freedom from bias, lawful use in every jurisdiction, or a reliable downstream model.

Trusted references and scope#

These primary and authoritative resources support practical dataset-quality, labeling, provenance, and risk review. Some technical examples are vendor-specific and do not establish universal legal obligations or independent certification.

Applicability: Hugging Face dataset cards and MLCommons Croissant are useful documentation/metadata conventions, not mandatory formats for every dataset. TensorFlow Data Validation illustrates schema, skew, and drift techniques; equivalent tools are acceptable. HumanSignal Label Studio agreement and quality features can be edition-specific. W3C PROV describes interoperable lineage, not proof of permission to collect data. UK ICO guidance concerns applicable UK data-protection law; evaluate other markets separately. Human labels may be subjective and disagreement can reveal genuine task ambiguity.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

  • Test focus: data lineage, nulls, duplicates, labeling agreement, PII, and train/test leakage; include model and dataset behaviors where applicable.
  • Cross-stack: Great Expectations + DVC.
  • Python: Pandera + pytest; for pytest-based suites, run python -m pytest -q.
  • TypeScript: Zod + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
  • JavaScript: Ajv + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
  • Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

  • dataset-curation — Inspect quality, splits, provenance, and leakage (wshobson/agents, 42,437 repo stars).
  • data-quality-frameworks — Check schema, null, duplicate, and anomaly tests (wshobson/agents, 42,437 repo stars).
  • arize-dataset — Assess evaluation-data lifecycle where Arize is used (github/awesome-copilot, 41,073 repo stars).

How to check: In a project-specific copy, have the agent audit the AI-specific scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.

Reference, not a certification. Check applicability, verify important claims against current primary sources, and seek expert review where necessary. Suggest an improvement.