awesomechecklists
llms.txtGitHub ↗
/

13 — Multimodal AI Quality Checklist#

Verify that AI features using images, audio, speech, video, documents, or combinations of modalities produce fit-for-purpose, safe, accessible, and properly grounded results. An AI model's claim to support a modality does not establish accuracy for a specific task, language, device, input quality, or deployment configuration.

Scope: AI-specific. This file is one part of the 13_Multimodal_AI_Quality.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

  • Project / product:
  • Release / version / environment:
  • Scope and applicable jurisdictions / markets:
  • Supported input / output modalities and intended tasks:
  • Model/provider, media preprocessing, and evaluation versions:
  • Quality, accessibility, safety, privacy, latency, and cost targets:
  • Reviewer / owner:
  • Review date:
  • Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

  1. Create a project-specific copy of this reusable checklist; leave all checkboxes in the repository template unchecked.
  2. Scope checks to the actual supported modalities, formats, audiences, languages, and product workflows. Image-only, speech-only, video, and cross-modal systems require different test fixtures and quality measures.
  3. Verify applicable checks using representative approved test media, independent reference judgments, actual deployed endpoints, preprocessing configurations, end-to-end user tests, privacy/security reviews, and accessibility assessments. Record failed checks, evidence, risk exceptions, and justified N/A outside this shared template using the MULTI ID.
  4. Keep ## API Checks and ## Frontend Checks separate, documenting inapplicability when a surface does not exist.
  5. Separate extraction/transcription, reasoning, generation, transport, playback, and presentation errors. A usable transcript, credible image, or plausible video summary does not prove downstream factual or safety correctness.
  6. Repeat representative multimodal and accessibility evaluations after changes to model versions, media transforms, codecs, cropping, frame sampling, provider capabilities, streaming logic, policy, or user interfaces. Coordinate deeper AI Evaluation, Red Teaming, Data Curation, and Accessibility reviews with those dedicated templates.

Modality support and requirements#

  • MULTI-001 Inventory the exact input and output modalities supported by each model, endpoint, client, and production workflow.
  • MULTI-002 Define the intended tasks and acceptable outcomes for image understanding, generation, audio, speech, video, and multimodal combinations actually used.
  • MULTI-003 Record the model/provider snapshot, modality capabilities, deployment region, and any feature-specific limitations for each route.
  • MULTI-004 Document supported file formats, codecs, dimensions, sample rates, durations, frame rates, and request-size limits for every applicable modality.
  • MULTI-005 Specify user populations, languages, scripts, accents, device types, and assistive-technology requirements relevant to the deployment.
  • MULTI-006 Define the task-specific quality, factuality, safety, latency, throughput, and cost thresholds before comparing variants.
  • MULTI-007 Identify consequential uses requiring specialist review rather than inferring medical, legal, biometric, or safety-critical accuracy.
  • MULTI-008 Distinguish capabilities that are native to a multimodal model from chained OCR, ASR, TTS, retrieval, and other processing components.
  • MULTI-009 Map media preprocessing, upload, storage, inference, postprocessing, playback, and deletion across the complete data flow.
  • MULTI-010 Verify media processing rights, licenses, speaker permissions, privacy notices, and any required user consent before collecting samples.
  • MULTI-011 Document how raw, transformed, compressed, cropped, sampled, or otherwise degraded inputs may change model-visible information.
  • MULTI-012 Specify whether the system may analyze metadata, EXIF, hidden document layers, timestamps, or only the actual media content exposed by its API.
  • MULTI-013 Define failure behavior for unsupported media, missing tracks, corrupt files, encrypted files, and media that exceeds provider limits.
  • MULTI-014 Set resource and token budgets for high-resolution images, long recordings, video frames, and repeated multimodal processing.
  • MULTI-015 Specify whether media input is ephemeral, retained for quality review, or used for model improvement under the applicable data policy.
  • MULTI-016 Establish source and generated-media provenance, attribution, and disclosure requirements where relevant to use and policy.
  • MULTI-017 Define expected behavior for ambiguous scenes, inaudible speech, incomplete frames, unknown speakers, and contradictory modalities.
  • MULTI-018 Treat audio/video content, embedded text, captions, and transcribed speech as untrusted data for instruction-following and tool use.
  • MULTI-019 Identify which downstream actions require validated evidence and human confirmation instead of relying on perception alone.
  • MULTI-020 Record fallback paths to accessible text, human support, alternate media formats, or non-AI features when a modality fails.
  • MULTI-021 Define versioned evaluation fixtures for single-modality tasks and paired or synchronized cross-modal tasks.
  • MULTI-022 Assign responsible product, engineering, accessibility, privacy, safety, and evaluation owners to supported multimodal workflows.

Image and video quality#

  • MULTI-023 Evaluate image understanding on representative photographs, screenshots, diagrams, charts, scans, and documents as applicable.
  • MULTI-024 Test small text, dense tables, rotated pages, blur, glare, shadows, occlusion, and low-resolution content.
  • MULTI-025 Measure OCR or visual text-extraction errors separately from factual reasoning errors made after extraction.
  • MULTI-026 Verify correct handling of multilingual and non-Latin text embedded in images where supported.
  • MULTI-027 Check that image orientation, cropping, resizing, tiling, and detail settings preserve task-critical evidence.
  • MULTI-028 Test object counting, spatial relationships, relative position, and coordinate-sensitive requests against ground-truth fixtures.
  • MULTI-029 Evaluate chart legends, units, axis labels, colors, line styles, and uncertainty when interpreting data visualizations.
  • MULTI-030 Verify generated descriptions distinguish visible evidence from guesses about identity, intent, location, or unseen events.
  • MULTI-031 Test refusal, abstention, or escalation when an image is insufficient to support a consequential conclusion.
  • MULTI-032 Check image generation and editing for prompt adherence, requested objects, composition, text accuracy, and preservation constraints.
  • MULTI-033 Assess generated images for unexpected additions, missing subjects, artifacts, distortions, and unsafe or misleading details.
  • MULTI-034 Evaluate unintended changes to protected visual attributes or user-supplied elements during requested image edits.
  • MULTI-035 Record image generation parameters, seeds where available, sizes, model versions, and variation across repeated outputs.
  • MULTI-036 Define appropriate provenance and AI-generation disclosure for synthetic images when required by product policy or law.
  • MULTI-037 Validate that image uploads cannot expose private EXIF/location metadata or unapproved file content downstream.
  • MULTI-038 Test video ingestion against supported codecs, containers, duration, aspect ratio, variable frame rates, and audio tracks.
  • MULTI-039 Document how video frames are sampled and which events may be missed between sampled frames.
  • MULTI-040 Test temporal ordering, action recognition, scene transitions, continuity, and event start/end timing with labeled video clips.
  • MULTI-041 Check whether summaries correctly distinguish actions visible in frames from events inferred without supporting video evidence.
  • MULTI-042 Evaluate visual changes in the presence of camera motion, panning, zoom, cuts, low light, and motion blur.
  • MULTI-043 Verify synchronized audio and video references preserve the correct timestamp and source segment.
  • MULTI-044 Test long-form video behavior when the clip exceeds the effective frame, token, or context budget.
  • MULTI-045 Assess generated image or video safety with representative permitted and disallowed cases under the approved policy.
  • MULTI-046 Ensure downloaded, stored, or displayed visual media follows access controls, licensing, retention, and deletion rules.

Audio and speech quality#

  • MULTI-047 Define separate quality thresholds for speech recognition, translation, diarization, synthesis, and realtime conversation as applicable.
  • MULTI-048 Benchmark speech recognition against correctly transcribed reference audio using word or character error rate suited to the language.
  • MULTI-049 Test accents, dialects, code-switching, non-native speech, low-volume speakers, and regional pronunciation.
  • MULTI-050 Include real background noise, echo, reverberation, crosstalk, microphone variation, and telephony compression in evaluation audio.
  • MULTI-051 Test recognition of domain terminology, names, identifiers, dates, amounts, addresses, and negation that affect user outcomes.
  • MULTI-052 Measure empty, truncated, duplicated, reordered, and hallucinated transcript segments separately from overall transcription error rates.
  • MULTI-053 Verify language detection and explicit language hints behave correctly for supported languages and mixed-language conversations.
  • MULTI-054 Test actual availability of word timestamps, speaker labels, and confidence fields rather than assuming all transcription models return them.
  • MULTI-055 Evaluate speaker diarization with overlaps, interruptions, speaker changes, and unknown speakers where supported.
  • MULTI-056 Validate sample rate, channel layout, codec, clipping, loudness, silence detection, and resampling behavior for audio input.
  • MULTI-057 Test voice-activity detection or turn segmentation against pauses, short replies, hesitations, cross-talk, and non-speech sounds.
  • MULTI-058 Measure final transcript latency and partial-transcript corrections under realistic realtime connection conditions.
  • MULTI-059 Ensure partial ASR output is not treated as a verified final transcript for irreversible actions.
  • MULTI-060 Check speech translation for meaning preservation, negation, named entities, tone, and relevant professional terminology.
  • MULTI-061 Evaluate synthesized speech for intelligibility, pronunciation, pacing, pauses, naturalness, and supported-language quality.
  • MULTI-062 Check synthesized speech for unexpected content, omitted words, unstable pronunciation, and misleading emotional or speaker presentation.
  • MULTI-063 Confirm the selected synthetic voice has appropriate consent/rights and that AI-generated speech is disclosed where required.
  • MULTI-064 Test streaming audio playback for missing chunks, reordering, clipping, jitter, long pauses, and reconnection artifacts.
  • MULTI-065 Test conversational interruptions, barge-in, cancellation, and truncation so unplayed responses are not treated as heard or confirmed.
  • MULTI-066 Verify spoken audio and visible transcript or subtitles refer to the same actual turn and do not falsely claim exact timing when unavailable.
  • MULTI-067 Validate processing of long recordings, silence-only files, unsupported channels, and audio exceeding provider limits.
  • MULTI-068 Assess content safety and privacy for speech and audio, including inadvertent disclosure of background conversations.
  • MULTI-069 Test audio fallback to text input/output or accessible transcripts when microphone access, playback, or synthesis fails.
  • MULTI-070 Verify audio storage, playback logs, transcription retention, and speaker identifiers comply with approved access and deletion controls.

Multimodal evaluation and accessibility#

  • MULTI-071 Maintain a held-out, versioned evaluation set containing each supported modality and meaningful cross-modal combinations.
  • MULTI-072 Record evaluation inputs, expected outcomes, source rights, modality metadata, and reference judgments for every case.
  • MULTI-073 Create examples where image, speech, video, and text convey complementary information needed for the correct result.
  • MULTI-074 Test contradictory visual and spoken instructions and document which information should govern the response.
  • MULTI-075 Measure answer grounding to the actual relevant visual frame, audio segment, or document region instead of a related but insufficient source.
  • MULTI-076 Evaluate cross-modal temporal alignment, speaker-to-scene association, and event chronology when the task depends on synchronization.
  • MULTI-077 Check that text-only and single-modality baselines are compared fairly with multimodal results where useful.
  • MULTI-078 Report quality by input type, language, device quality, content difficulty, and other risk-relevant cohorts with sufficient sample sizes.
  • MULTI-079 Separate extraction failures, model reasoning failures, unsafe generation, input transport errors, and UI presentation defects.
  • MULTI-080 Use trained human reviewers or calibrated rubrics for subjective visual/audio quality and multimodal factual support.
  • MULTI-081 Document model-judge limitations and cross-modal evaluation bias rather than trusting one aggregate automated score.
  • MULTI-082 Repeat variable generation tests where needed and record versioned models, transforms, seeds, settings, and run identifiers.
  • MULTI-083 Test adversarial instructions embedded in images, audio transcripts, subtitles, and video frames using authorized benign fixtures.
  • MULTI-084 Verify that untrusted media content cannot elevate tool permissions, change system instructions, or access unrelated user data.
  • MULTI-085 Evaluate text alternatives for informative images and controls according to applicable accessibility requirements.
  • MULTI-086 Provide appropriate captions for synchronized audio/video, including necessary non-speech information where required.
  • MULTI-087 Provide transcripts, audio descriptions, or other media alternatives appropriate to the content and applicable WCAG level.
  • MULTI-088 Check that generated captions and transcripts are edited or verified where consequential accuracy or accessibility depends on them.
  • MULTI-089 Test caption timing, speaker labels, visual legibility, font scaling, and playback speed controls where provided.
  • MULTI-090 Verify keyboard operation, focus indication, accessible controls, and screen-reader labeling of media upload and playback experiences.
  • MULTI-091 Test multimodal use under poor network conditions and on assistive devices without losing critical state or accessibility information.
  • MULTI-092 Check harmful or misleading synthetic-media outputs and provenance/disclosure limitations as part of the actual product risk review.
  • MULTI-093 Measure time to first useful output, total completion, resource usage, and cost for each supported modality mix.
  • MULTI-094 Require qualified approval of significant quality, privacy, security, and accessibility shortcomings before release.

Release gate#

  • Every advertised modality, combination, language, file format, and input limit is supported by the exact deployed model/API/client configuration and correctly documented.
  • Representative image/video, audio/speech, and supported cross-modal tasks pass predeclared factuality, fidelity, temporal alignment, and user-task quality criteria.
  • Low-quality input, unsupported metadata, missing evidence, ambiguous media, streaming failures, and incomplete results trigger tested safe fallback, abstention, or clarification behavior.
  • Important privacy, consent, source-use rights, synthetic-media disclosures, and data lifecycle requirements have appropriate qualified approval.
  • Unsafe media content and embedded instructions cannot bypass independent access controls, moderation, tool authorization, or protected data boundaries.
  • Required text alternatives, transcripts, captions, visual descriptions, accessible controls, and supported assistive-technology journeys have been checked for the actual content and applicable WCAG requirements.
  • Production-equivalent API and frontend tests verify format validation, size limits, media handling, cancellation, latency/cost budgets, and safe completion or failure states.
  • Accountable owners approve relevant versioned evaluation results, known modality gaps, remaining high-impact risks, rollback paths, and the exact release.

Domain-specific approval rule: Block release where critical media interpretation or synthesis is materially misleading, protected recordings or images can be exposed, unsafe embedded media instructions enable unauthorized actions, or necessary accessible alternatives are missing for a supported consequential workflow without approved mitigation. Passing individual modality benchmarks does not prove correct cross-modal reasoning or regulatory compliance.

Trusted references and scope#

The primary technical documentation and accessibility standards below inform project-specific verification. These sources cover different vendors and media modalities; use only the standards and provider features applicable to the deployed system.

Applicability: OpenAI vision, audio, transcription, and file-input capabilities differ by model, endpoint, and revision; verify current formats, quotas, metadata, and any deprecation dates instead of copying them across models. As of this source review, OpenAI documentation lists a February 26, 2027 shutdown for certain legacy transcription models and a January 6, 2027 shutdown for several legacy TTS models, so new deployments should check current alternatives. Amazon Bedrock multimodal indexing and retrieval are also workflow- and provider-specific; some modes do not search speech or support mixed text/image queries. WCAG 2.2 obligations depend on media type, live/prerecorded status, conformance level, and jurisdiction. NIST AI RMF guidance is voluntary and not a safety certification. Do not use automated visual or audio inference as unreviewed proof for consequential identity or health decisions.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

  • Test focus: image/audio/video fixtures, timing, captioning, invalid media, and accessibility; include model and dataset behaviors where applicable.
  • Cross-stack: FFmpeg + Playwright.
  • Python: Pillow + librosa + pytest; for pytest-based suites, run python -m pytest -q.
  • TypeScript: Playwright + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
  • JavaScript: Playwright + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
  • Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

  • caption-qa — Assess captions and media accessibility (github/awesome-copilot, 41,073 repo stars).
  • remotion-captions — Check video caption workflows when applicable (remotion-dev/skills, 6,895 repo stars).
  • webapp-testing — Exercise media upload, rendering, and failures (anthropics/skills, 193,495 repo stars).

How to check: In a project-specific copy, have the agent audit the AI-specific scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.

Reference, not a certification. Check applicability, verify important claims against current primary sources, and seek expert review where necessary. Suggest an improvement.