awesomechecklists
llms.txtGitHub ↗
/

19 — Multimodal AI Quality Checklist#

Verify that AI features using images, audio, speech, video, documents, or combinations of modalities produce fit-for-purpose, safe, accessible, and properly grounded results. An AI model's claim to support a modality does not establish accuracy for a specific task, language, device, input quality, or deployment configuration.

Scope: Backend / API-specific. This file is one part of the 19_Multimodal_AI_Quality.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

  • Project / product:
  • Release / version / environment:
  • Scope and applicable jurisdictions / markets:
  • Supported input / output modalities and intended tasks:
  • Model/provider, media preprocessing, and evaluation versions:
  • Quality, accessibility, safety, privacy, latency, and cost targets:
  • Reviewer / owner:
  • Review date:
  • Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

  1. Create a project-specific copy of this reusable checklist; leave all checkboxes in the repository template unchecked.
  2. Scope checks to the actual supported modalities, formats, audiences, languages, and product workflows. Image-only, speech-only, video, and cross-modal systems require different test fixtures and quality measures.
  3. Verify applicable checks using representative approved test media, independent reference judgments, actual deployed endpoints, preprocessing configurations, end-to-end user tests, privacy/security reviews, and accessibility assessments. Record failed checks, evidence, risk exceptions, and justified N/A outside this shared template using the MULTI ID.
  4. Keep ## API Checks and ## Frontend Checks separate, documenting inapplicability when a surface does not exist.
  5. Separate extraction/transcription, reasoning, generation, transport, playback, and presentation errors. A usable transcript, credible image, or plausible video summary does not prove downstream factual or safety correctness.
  6. Repeat representative multimodal and accessibility evaluations after changes to model versions, media transforms, codecs, cropping, frame sampling, provider capabilities, streaming logic, policy, or user interfaces. Coordinate deeper AI Evaluation, Red Teaming, Data Curation, and Accessibility reviews with those dedicated templates.

API Checks#

  • MULTI-095 Enforce authenticated user, tenant, and resource authorization for each media upload, processing request, retrieval, and generated output.
  • MULTI-096 Validate actual file magic bytes, supported MIME types, codecs, sizes, duration, sample rates, and dimensions server-side.
  • MULTI-097 Reject corrupted, encrypted, unsupported, or oversized media with documented safe error states.
  • MULTI-098 Scan or sandbox untrusted file parsing and prohibit execution of embedded content or metadata-driven commands.
  • MULTI-099 Check supported modality fields and combinations against the exact deployed model endpoint and API schema.
  • MULTI-100 Record whether image resizing, frame sampling, audio transcoding, or PDF rendering changes model-visible evidence.
  • MULTI-101 Enforce approved upper bounds on media counts, frame rates, audio duration, token usage, retries, and costs.
  • MULTI-102 Validate signed media URLs, temporary file handles, upload destinations, and storage permissions before processing.
  • MULTI-103 Keep sensitive media, speaker identifiers, transcripts, and generated files out of broad-access logs and traces.
  • MULTI-104 Test that temporary and retained media are deleted or inaccessible after revocation, expiry, or account removal as required.
  • MULTI-105 Prevent cross-tenant access through caches, model sessions, media asset IDs, and provider-side storage references.
  • MULTI-106 Check video timestamps, frame IDs, transcript segment IDs, and stream event ordering against API contracts.
  • MULTI-107 Handle partial realtime transcription or audio generation without reporting unfinished text or playback as completed.
  • MULTI-108 Verify retries and cancellation do not duplicate downstream communications, actions, or billed jobs unnecessarily.
  • MULTI-109 Test simultaneous media uploads and processing for rate-limit fairness and bounded resource utilization.
  • MULTI-110 Protect model-generated structured responses, extracted metadata, and tool arguments with schema and authorization checks.
  • MULTI-111 Check input, output, and transformed-media provenance identifiers remain traceable across API processing steps.
  • MULTI-112 Test unsupported features such as diarization, timestamp outputs, and alternate media tracks with explicit documented fallbacks.
  • MULTI-113 Validate privacy, consent, data residency, and retention settings for external multimodal model providers.
  • MULTI-114 Run production-equivalent multimodal API contract, safety, quality, and overload tests before deployment.

Release gate#

  • Every advertised modality, combination, language, file format, and input limit is supported by the exact deployed model/API/client configuration and correctly documented.
  • Representative image/video, audio/speech, and supported cross-modal tasks pass predeclared factuality, fidelity, temporal alignment, and user-task quality criteria.
  • Low-quality input, unsupported metadata, missing evidence, ambiguous media, streaming failures, and incomplete results trigger tested safe fallback, abstention, or clarification behavior.
  • Important privacy, consent, source-use rights, synthetic-media disclosures, and data lifecycle requirements have appropriate qualified approval.
  • Unsafe media content and embedded instructions cannot bypass independent access controls, moderation, tool authorization, or protected data boundaries.
  • Required text alternatives, transcripts, captions, visual descriptions, accessible controls, and supported assistive-technology journeys have been checked for the actual content and applicable WCAG requirements.
  • Production-equivalent API and frontend tests verify format validation, size limits, media handling, cancellation, latency/cost budgets, and safe completion or failure states.
  • Accountable owners approve relevant versioned evaluation results, known modality gaps, remaining high-impact risks, rollback paths, and the exact release.

Domain-specific approval rule: Block release where critical media interpretation or synthesis is materially misleading, protected recordings or images can be exposed, unsafe embedded media instructions enable unauthorized actions, or necessary accessible alternatives are missing for a supported consequential workflow without approved mitigation. Passing individual modality benchmarks does not prove correct cross-modal reasoning or regulatory compliance.

Trusted references and scope#

The primary technical documentation and accessibility standards below inform project-specific verification. These sources cover different vendors and media modalities; use only the standards and provider features applicable to the deployed system.

Applicability: OpenAI vision, audio, transcription, and file-input capabilities differ by model, endpoint, and revision; verify current formats, quotas, metadata, and any deprecation dates instead of copying them across models. As of this source review, OpenAI documentation lists a February 26, 2027 shutdown for certain legacy transcription models and a January 6, 2027 shutdown for several legacy TTS models, so new deployments should check current alternatives. Amazon Bedrock multimodal indexing and retrieval are also workflow- and provider-specific; some modes do not search speech or support mixed text/image queries. WCAG 2.2 obligations depend on media type, live/prerecorded status, conformance level, and jurisdiction. NIST AI RMF guidance is voluntary and not a safety certification. Do not use automated visual or audio inference as unreviewed proof for consequential identity or health decisions.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

  • Test focus: image/audio/video fixtures, timing, captioning, invalid media, and accessibility; include API and service behavior where applicable.
  • Cross-stack: FFmpeg + Playwright.
  • Python: Pillow + librosa + pytest; for pytest-based suites, run python -m pytest -q.
  • TypeScript: Playwright + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
  • JavaScript: Playwright + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
  • Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

  • caption-qa — Assess captions and media accessibility (github/awesome-copilot, 41,073 repo stars).
  • remotion-captions — Check video caption workflows when applicable (remotion-dev/skills, 6,895 repo stars).
  • webapp-testing — Exercise media upload, rendering, and failures (anthropics/skills, 193,495 repo stars).

How to check: In a project-specific copy, have the agent audit the backend / API scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.

Reference, not a certification. Check applicability, verify important claims against current primary sources, and seek expert review where necessary. Suggest an improvement.