awesomechecklists
llms.txtGitHub ↗
/

15 — Prompt Engineering and Versioning Checklist#

Review instructions, templates, prompt revisions, and evaluation gates used in model-integrated software. Treat prompt wording as a behavioral control and engineering artifact, not a guarantee of security or deterministic correctness. Protect trust boundaries and external actions with independent code-level authorization.

Scope: Backend / API-specific. This file is one part of the 15_Prompt_Engineering_and_Versioning.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

  • Project / product:
  • Release / version / environment:
  • Scope and applicable jurisdictions / markets:
  • Prompt purpose / workflow / output contract:
  • Model / provider / snapshot / prompt revision:
  • Safety, evaluation, and release owners:
  • Reviewer / owner:
  • Review date:
  • Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

  1. Copy this shared template into project-specific review records; leave the repository's checkboxes unchecked.
  2. Mark [x] only after checking versioned source, actual rendered messages, execution-boundary controls, reproducible evaluation results, deployment evidence, or an accountable risk decision. Keep failed checks, N/A reasons, and exceptions in project-specific notes/issues keyed by PROMPT ID, not in an evidence table here.
  3. Choose prompt strategies based on the target model and its current API. Clear role separation, examples, and formatting may improve output consistency but do not guarantee safety, confidentiality, or obedience.
  4. Review ## API Checks and ## Frontend Checks separately if those surfaces exist; document inapplicability outside the shared template.
  5. Re-run relevant evaluations when prompt text, model revision, retrieval, tools, parsing, policy, traffic, or user interface changes. Detailed RAG, agent orchestration, adversarial red-teaming, evaluation regression, and model-monitoring checks belong in the relevant dedicated AI domain checklists.

API Checks#

  • PROMPT-093 Confirm the runtime selects the approved prompt revision and model configuration for each endpoint.
  • PROMPT-094 Validate required prompt variables against typed schemas before rendering or sending to a model.
  • PROMPT-095 Prevent user-controlled strings from becoming executable templates, privileged instructions, or unreviewed tool definitions.
  • PROMPT-096 Enforce context length and output budget limits and return explicit errors or safe truncation behavior.
  • PROMPT-097 Ensure request-level user, tenant, and resource permissions apply to retrieval and all model-triggered tool operations.
  • PROMPT-098 Validate structured model outputs and function-call arguments before storing data or executing downstream actions.
  • PROMPT-099 Test missing variable values, malformed input, unsupported modalities, and invalid encodings with predictable responses.
  • PROMPT-100 Verify any API caching includes prompt, model, tenant, permissions, and context version when they affect response correctness.
  • PROMPT-101 Guard against cross-tenant leakage in prompt histories, retrieval snippets, provider cache, and diagnostic payloads.
  • PROMPT-102 Limit and monitor token usage, tool calls, retries, and spending to protect the service during prompt regressions.
  • PROMPT-103 Ensure sensitive prompts and response content are not exposed in default HTTP logs, traces, or provider debugging.
  • PROMPT-104 Return explicit refusal, abstention, incomplete-result, and validation-error states where required by API contracts.
  • PROMPT-105 Test streaming output parsing and guardrails without committing partial, unvalidated data to irreversible actions.
  • PROMPT-106 Measure latency, token cost, schema validity, and policy-block rates by approved prompt revision.
  • PROMPT-107 Correlate incident events with model, prompt, policy, and tool schema revisions using safe identifiers.
  • PROMPT-108 Verify fallback to a prior prompt or alternate model preserves permissions, output format, and safety requirements.
  • PROMPT-109 Check prompt injection from external API results, documents, and tool outputs with nonproduction fixtures.
  • PROMPT-110 Run CI or deployment contract tests proving prompt changes do not silently break supported API consumers.
  • PROMPT-111 Ensure prompt release toggles, rollout cohort selection, and emergency disabling are server-controlled.
  • PROMPT-112 Record the prompt-version change and its approval evidence in the API release notes or equivalent change log.

Release gate#

  • The production prompt's intended task, trusted instructions, allowed inputs, output contract, limits, and accountable owners are documented.
  • Prompt templates safely handle variable inputs and untrusted content; model-generated tool calls are checked by independent permissions and schema validators.
  • A reviewed, identifiable prompt revision is tied to an approved model/configuration snapshot, relevant schemas, and the release artifact.
  • Representative held-out and adversarial evaluations meet predefined task-quality, policy, and format thresholds, with severe failures documented.
  • The new prompt is compared against the currently approved baseline for critical regressions, false refusals, task completion, latency, and resource cost.
  • API and frontend integration tests cover prompt-version-specific behavior, unsafe output handling, and useful recovery states where applicable.
  • Rollout, monitoring, incident escalation, prompt rollback, and documented change authorization are ready for the intended release.
  • Material residual privacy, security, quality, or user-harm risks have explicit owner, mitigation, retest deadline, and authorized release disposition.

Domain-specific approval rule: Block release when a prompt change can bypass required authorization, misroute sensitive content, break critical output contracts, or introduce significant unaccepted quality or safety regressions. No amount of instruction wording substitutes for enforced permission boundaries, input/output validation, or meaningful evaluation.

Trusted references and scope#

These primary or authoritative sources inform the verification checks. Apply vendor-specific suggestions only to models and platforms where they have been tested; they are not universal guarantees of reliable or safe model behavior.

Applicability: OpenAI's November 2026 prompt guidance prefers reviewed, code-managed prompts and notes deprecation of its reusable prompt-object API; check the official migration and deadline before relying on that feature. Prompt advice differs by model generation and vendor. OWASP string delimiters and text filters are defense-in-depth, not prompt-injection prevention guarantees. Use controlled benign and adversarial tests and code-enforced tool permission checks. This file does not replace governance, model selection, RAG, AI red-team, or full regression-evaluation reviews.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

  • Test focus: prompt versions, formatting, instructions, refusals, and injection resistance; include API and service behavior where applicable.
  • Cross-stack: promptfoo.
  • Python: DeepEval + pytest; for pytest-based suites, run python -m pytest -q.
  • TypeScript: promptfoo + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
  • JavaScript: promptfoo + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
  • Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

How to check: In a project-specific copy, have the agent audit the backend / API scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.

Reference, not a certification. Check applicability, verify important claims against current primary sources, and seek expert review where necessary. Suggest an improvement.