awesomechecklists
llms.txtGitHub ↗
/

09 — Prompt Engineering and Versioning Checklist#

Review instructions, templates, prompt revisions, and evaluation gates used in model-integrated software. Treat prompt wording as a behavioral control and engineering artifact, not a guarantee of security or deterministic correctness. Protect trust boundaries and external actions with independent code-level authorization.

Scope: AI-specific. This file is one part of the 09_Prompt_Engineering_and_Versioning.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

  • Project / product:
  • Release / version / environment:
  • Scope and applicable jurisdictions / markets:
  • Prompt purpose / workflow / output contract:
  • Model / provider / snapshot / prompt revision:
  • Safety, evaluation, and release owners:
  • Reviewer / owner:
  • Review date:
  • Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

  1. Copy this shared template into project-specific review records; leave the repository's checkboxes unchecked.
  2. Mark [x] only after checking versioned source, actual rendered messages, execution-boundary controls, reproducible evaluation results, deployment evidence, or an accountable risk decision. Keep failed checks, N/A reasons, and exceptions in project-specific notes/issues keyed by PROMPT ID, not in an evidence table here.
  3. Choose prompt strategies based on the target model and its current API. Clear role separation, examples, and formatting may improve output consistency but do not guarantee safety, confidentiality, or obedience.
  4. Review ## API Checks and ## Frontend Checks separately if those surfaces exist; document inapplicability outside the shared template.
  5. Re-run relevant evaluations when prompt text, model revision, retrieval, tools, parsing, policy, traffic, or user interface changes. Detailed RAG, agent orchestration, adversarial red-teaming, evaluation regression, and model-monitoring checks belong in the relevant dedicated AI domain checklists.

Prompt design and requirements#

  • PROMPT-001 Define the intended task, user population, allowed inputs, and desired business outcome for every production prompt.
  • PROMPT-002 Identify the approved model family or snapshot and any provider-specific behavior the prompt depends on.
  • PROMPT-003 Document the output's intended audience, expected level of detail, and appropriate language or tone.
  • PROMPT-004 Specify measurable task-success criteria and disallowed outcomes before iterating on prompt wording.
  • PROMPT-005 Assign a named engineering owner and product or domain approver to each critical prompt.
  • PROMPT-006 Distinguish trusted application instructions, end-user requests, retrieved documents, and tool output in the prompt design.
  • PROMPT-007 Express essential requirements with clear, noncontradictory instructions and explicit priorities.
  • PROMPT-008 Describe the scope and limits of the assistant's role without overstating expertise, factual certainty, or authority.
  • PROMPT-009 Define what the model should do when information is absent, contradictory, stale, or outside its permitted task.
  • PROMPT-010 Specify when the assistant must ask for clarification, abstain, escalate, or hand off to a human.
  • PROMPT-011 Define how citations or source evidence must support claims when the task requires grounded answers.
  • PROMPT-012 Choose an output contract such as structured JSON, prose, or tool arguments that downstream consumers can validate.
  • PROMPT-013 Describe any safety, privacy, legal, or accessibility restrictions relevant to the application's task.
  • PROMPT-014 Specify the permitted external actions and which actions require explicit, action-specific approval.
  • PROMPT-015 Include a representative set of positive and negative examples only when they improve measured task quality.
  • PROMPT-016 Ensure examples are diverse and reflect realistic inputs, languages, edge cases, and expected outputs.
  • PROMPT-017 Check that examples do not accidentally teach unwanted biases, shortcuts, or unsafe behavior.
  • PROMPT-018 Use clear delimiters or structured messages to separate instructions, context, examples, and variable inputs.
  • PROMPT-019 Avoid relying on particular delimiters, polite wording, or hidden text as a security or authorization boundary.
  • PROMPT-020 Remove redundant, conflicting, stale, or excessively prescriptive instructions that worsen measured output quality.
  • PROMPT-021 Document model-specific prompting techniques and avoid assuming instructions work identically across vendors or snapshots.
  • PROMPT-022 Set explicit prompt length, token, latency, and inference cost budgets appropriate to the user journey.

Templates and input handling#

  • PROMPT-023 Store approved prompt templates in reviewed application code or another supported, versioned configuration system.
  • PROMPT-024 Define each template variable with a type, purpose, source, allowable length, and validation policy.
  • PROMPT-025 Render untrusted user and external content as data values rather than executable template source.
  • PROMPT-026 Use a template rendering system that does not evaluate arbitrary expressions from user-controlled text.
  • PROMPT-027 Keep developer/system instructions separate from untrusted user, document, search, or tool-provided content wherever the model API supports roles.
  • PROMPT-028 Identify and label retrieved passages or tool results as untrusted evidence, while recognizing labels are not an enforcement boundary.
  • PROMPT-029 Validate inputs before invoking provider APIs, including modality, size, encoding, and supported formats.
  • PROMPT-030 Apply appropriate truncation, summarization, or chunking policies and document what information may be lost.
  • PROMPT-031 Budget system instructions, conversation history, retrieved context, tools, and expected output within the selected model context window.
  • PROMPT-032 Verify that context-window overflow and provider truncation do not silently discard critical instructions or necessary evidence.
  • PROMPT-033 Check that conversation-history selection does not mix one user's or tenant's data into another context.
  • PROMPT-034 Define expiration and deletion of prompt variables, retrieved content, and conversation history according to policy.
  • PROMPT-035 Ensure prompts and examples never contain real API keys, production secrets, or unnecessary protected personal information.
  • PROMPT-036 Protect internal prompt templates and proprietary context with appropriate code/configuration access controls.
  • PROMPT-037 Treat model outputs as untrusted for HTML, code execution, SQL, command invocation, and downstream integrations.
  • PROMPT-038 Enforce tool permissions and action-specific approvals in deterministic code rather than relying on model obedience.
  • PROMPT-039 Validate model-generated structured outputs and tool arguments against strict schemas at execution boundaries.
  • PROMPT-040 Test direct and indirect prompt-injection attempts across user, document, web, tool, and history channels.
  • PROMPT-041 Test disguised instructions, encoded content, unexpected markup, and multilingual variations with authorized harmless fixtures.
  • PROMPT-042 Measure false refusals on legitimate inputs alongside successful blocking of malicious requests.
  • PROMPT-043 Specify stable error handling for malformed variables, missing placeholders, oversized inputs, and failed template rendering.
  • PROMPT-044 Confirm request logging, traces, and debugging tools exclude prohibited sensitive prompt data and confidential templates.

Versioning and change management#

  • PROMPT-045 Assign an immutable identifier to every production prompt revision and link it to the related feature or workflow.
  • PROMPT-046 Keep prompt source in version control with a reviewable history, author, date, and reason for the change.
  • PROMPT-047 Capture the exact model snapshot or provider revision associated with each approved prompt version.
  • PROMPT-048 Version dependent tool schemas, retrieved-context instructions, safety policies, and output validators together with prompts.
  • PROMPT-049 Distinguish prompt content changes from model, temperature, tool, retrieval, and other configuration changes during reviews.
  • PROMPT-050 Require review by the responsible domain and safety/security owners for material behavior changes.
  • PROMPT-051 Define a risk-based approval process for low-risk copy changes versus changes to tool access, data processing, or sensitive decisions.
  • PROMPT-052 Maintain separate development, evaluation, staging, and approved production prompt configurations.
  • PROMPT-053 Deploy immutable prompt revisions or recorded code releases rather than mutable, untracked 'latest' behavior.
  • PROMPT-054 Document any provider-managed prompt feature's support lifecycle and a migration path if it is deprecated.
  • PROMPT-055 Use an auditable release process linking prompt revision, artifact, evaluation results, and approver.
  • PROMPT-056 Run required evaluation checks before promoting a prompt revision to production.
  • PROMPT-057 Use canary exposure, feature flags, or equivalent staged rollout controls for high-impact prompt changes.
  • PROMPT-058 Define measurable post-deployment health signals and rollback conditions for changed prompts.
  • PROMPT-059 Keep an approved previous prompt revision and compatible model/configuration available for rollback where feasible.
  • PROMPT-060 Confirm rollback does not silently break output schemas, stored conversations, or client expectations.
  • PROMPT-061 Require reevaluation when the model version changes even if the prompt text remains unchanged.
  • PROMPT-062 Review the user experience and safety implications of prompt changes after incident findings or customer complaints.
  • PROMPT-063 Track deprecation or retirement of unused prompts and prevent obsolete versions from being called accidentally.
  • PROMPT-064 Record migration guidance for teams and APIs consuming a prompt whose output contract has changed.
  • PROMPT-065 Audit access to production prompt editing and promotion, and remove obsolete editor permissions.
  • PROMPT-066 Document ownership and expiry for emergency prompt hotfixes, with retrospective review and regression testing.

Prompt evaluation and regression#

  • PROMPT-067 Define representative evaluation fixtures based on realistic production tasks and approved synthetic or sanitized data.
  • PROMPT-068 Keep a held-out regression set separate from examples used to tune the prompt.
  • PROMPT-069 Define task-specific acceptance thresholds for accuracy, completeness, helpfulness, factuality, and format compliance where relevant.
  • PROMPT-070 Include expected refusals, clarifications, abstentions, and human escalation cases.
  • PROMPT-071 Evaluate ambiguous, adversarial, multilingual, long-context, and missing-context prompts appropriate to deployment risk.
  • PROMPT-072 Test both typical and severe-error cases rather than relying only on a small set of happy paths.
  • PROMPT-073 Compare prompt candidates against the same model and environment configuration before attributing score changes to prompt text.
  • PROMPT-074 Compare new prompt revisions against the last approved production baseline using paired or otherwise comparable cases.
  • PROMPT-075 Record prompt/model version, variable fixtures, decoding parameters, evaluation harness, and evaluator versions for every run.
  • PROMPT-076 Repeat stochastic tests sufficiently to characterize output variance when results may change between runs.
  • PROMPT-077 Use objective assertions for schemas, required fields, forbidden side effects, and supported format constraints.
  • PROMPT-078 Use qualified human rubrics where there is no single correct output or automated metrics are insufficient.
  • PROMPT-079 Calibrate model-based graders against human judgments before trusting their scores or letting them block deployment.
  • PROMPT-080 Test graders for positional, verbosity, language, and self-preference bias when applicable.
  • PROMPT-081 Assess how well retrieved evidence and actual output citations support claims for grounded tasks.
  • PROMPT-082 Evaluate safety refusals and false positives together so a system that refuses all requests is not deemed successful.
  • PROMPT-083 Test prompt injection against meaningful trust boundaries and verify tool permissions independently.
  • PROMPT-084 Report security outcomes from observed tool actions and data access, not just harmless-sounding model responses.
  • PROMPT-085 Measure p50/p95 latency, output length, token usage, and cost impact of prompt revisions under representative workload.
  • PROMPT-086 Test expected structured outputs across error conditions, partial outputs, tool call branches, and streaming where applicable.
  • PROMPT-087 Verify multi-turn state and long-context behavior, including stale instructions and conflicting conversational requests.
  • PROMPT-088 Include high-impact policy and regulated-workflow cases reviewed by appropriate domain specialists.
  • PROMPT-089 Track per-case regressions and severe failures with owners and remediation rather than hiding them in averaged scores.
  • PROMPT-090 Collect post-release feedback and incidents as candidates for privacy-preserving regression tests.
  • PROMPT-091 Define stop/go criteria and require documented risk acceptance when a new prompt regresses a material metric.
  • PROMPT-092 Rerun targeted and full evaluations after changes to provider snapshots, tools, retrieval, or safety controls.

Release gate#

  • The production prompt's intended task, trusted instructions, allowed inputs, output contract, limits, and accountable owners are documented.
  • Prompt templates safely handle variable inputs and untrusted content; model-generated tool calls are checked by independent permissions and schema validators.
  • A reviewed, identifiable prompt revision is tied to an approved model/configuration snapshot, relevant schemas, and the release artifact.
  • Representative held-out and adversarial evaluations meet predefined task-quality, policy, and format thresholds, with severe failures documented.
  • The new prompt is compared against the currently approved baseline for critical regressions, false refusals, task completion, latency, and resource cost.
  • API and frontend integration tests cover prompt-version-specific behavior, unsafe output handling, and useful recovery states where applicable.
  • Rollout, monitoring, incident escalation, prompt rollback, and documented change authorization are ready for the intended release.
  • Material residual privacy, security, quality, or user-harm risks have explicit owner, mitigation, retest deadline, and authorized release disposition.

Domain-specific approval rule: Block release when a prompt change can bypass required authorization, misroute sensitive content, break critical output contracts, or introduce significant unaccepted quality or safety regressions. No amount of instruction wording substitutes for enforced permission boundaries, input/output validation, or meaningful evaluation.

Trusted references and scope#

These primary or authoritative sources inform the verification checks. Apply vendor-specific suggestions only to models and platforms where they have been tested; they are not universal guarantees of reliable or safe model behavior.

Applicability: OpenAI's November 2026 prompt guidance prefers reviewed, code-managed prompts and notes deprecation of its reusable prompt-object API; check the official migration and deadline before relying on that feature. Prompt advice differs by model generation and vendor. OWASP string delimiters and text filters are defense-in-depth, not prompt-injection prevention guarantees. Use controlled benign and adversarial tests and code-enforced tool permission checks. This file does not replace governance, model selection, RAG, AI red-team, or full regression-evaluation reviews.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

  • Test focus: prompt versions, formatting, instructions, refusals, and injection resistance; include model and dataset behaviors where applicable.
  • Cross-stack: promptfoo.
  • Python: DeepEval + pytest; for pytest-based suites, run python -m pytest -q.
  • TypeScript: promptfoo + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
  • JavaScript: promptfoo + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
  • Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

How to check: In a project-specific copy, have the agent audit the AI-specific scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.

Reference, not a certification. Check applicability, verify important claims against current primary sources, and seek expert review where necessary. Suggest an improvement.