16 — RAG and Knowledge Bases Checklist#

Verify that retrieval-augmented generation (RAG) uses authorized, attributable, sufficiently fresh evidence; retrieves relevant passages; and produces answers whose factual claims can be checked against the source. Search relevance, answer quality, and access controls are separate requirements: improving one does not establish the others.

Scope: Backend / API-specific. This file is one part of the 16_RAG_and_Knowledge_Bases.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

• Project / product:
• Release / version / environment:
• Scope and applicable jurisdictions / markets:
• Knowledge sources, document owners, and access policy:
• Index / embedding / retrieval / reranker / model versions:
• Source freshness expectations and evaluation owners:
• Reviewer / owner:
• Review date:
• Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

1. Make a project-specific copy of this shared template; leave repository checkboxes unchecked.
2. Verify each applicable item using approved source inventories, ingestion logs, index metadata, permission-boundary tests, retrieval judgments, grounded-answer evaluations, and user-facing citation checks. Record failures, approved exceptions, and justified N/A outside the shared template, keyed by RAG ID.
3. Define what content is authoritative and what the current user is permitted to see before optimizing embeddings or model prompts. A retrieval score is not an access-control decision.
4. Evaluate retrieval relevance separately from generation groundedness. Test missing, stale, conflicting, malicious, and inaccessible sources, not only answerable examples.
5. Complete ## API Checks and ## Frontend Checks independently where present. For retrieval-only or non-interactive systems, document why an interface is inapplicable.
6. Reassess the entire retrieval-to-answer chain after source permissions, documents, ingestion, embeddings, index schema, ranking, prompts, model, or client presentation change. Deep AI agent, red-team, monitoring, and regression requirements belong in their dedicated templates.

API Checks#

• RAG-089 Enforce identity, role, tenant, and document permissions on every retrieval request before exposing text or embeddings.
• RAG-090 Validate query text, metadata filters, collection IDs, source selectors, and requested result limits against server-side rules.
• RAG-091 Prevent arbitrary caller-supplied index names, filters, or paths from bypassing allowed corpus boundaries.
• RAG-092 Check vector, keyword, hybrid, and multi-query retrieval modes for equivalent authorization guarantees.
• RAG-093 Ensure model generation never receives passages the current caller is not authorized to read.
• RAG-094 Version and record embedding, indexing, ranking, query-rewrite, and generation configurations for each release.
• RAG-095 Return stable source identifiers and passage-level citation metadata without disclosing protected source locations.
• RAG-096 Document contract behavior for zero results, low relevance, unavailable sources, and conflicting evidence.
• RAG-097 Provide structured answer confidence or groundedness indicators only when calibrated and meaningful.
• RAG-098 Apply strict limits to query expansion, retrieval size, token budget, embedding requests, latency, and spend.
• RAG-099 Validate and sanitize retrieved text before downstream rendering or tool use; do not treat retrieved instructions as authorized actions.
• RAG-100 Test malicious document content for prompt-injection resistance and independent authorization of all model-proposed tools.
• RAG-101 Protect embedding provider and search API credentials in trusted server components.
• RAG-102 Test source deletion, revocation, and permission updates against retrievable chunks, responses, caches, and conversation context.
• RAG-103 Prevent cross-tenant data leakage via embedding caches, search-result caches, response caches, and conversation memory.
• RAG-104 Define retry/backoff and partial-degradation behavior for unavailable index, source system, or embedding service.
• RAG-105 Measure API latency, search relevance, ingestion lag, failures, and cost using appropriate privacy-safe telemetry.
• RAG-106 Test pagination, score thresholds, and metadata filters against their documented search API behavior.
• RAG-107 Keep sensitive queries, retrieved passages, provider payloads, and citations out of broad-access logs and traces.
• RAG-108 Run contract and regression tests for retrieval result formats and citation identifiers used by clients.

Release gate#

• Knowledge sources have accountable owners, approved permitted uses, provenance records, and documented inclusion/exclusion and retention policies.
• Ingestion, extraction, chunking, embeddings, indexing, and synchronization have been tested with representative source formats and update/delete scenarios.
• Retrieval enforces document, user, group, and tenant authorization before any protected passage is sent to the model or client; cross-tenant tests pass.
• Held-out relevance checks meet agreed recall, precision/ranking, identifier search, and latency thresholds for critical question types.
• Grounded answers meet predeclared factual support, citation accuracy, completeness, uncertainty, and abstention criteria; severe unsupported claims are resolved or accepted through an authorized process.
• Stale, revoked, conflicting, insufficient, and malicious source material triggers approved warnings, no-answer behavior, containment, or escalation rather than unauthorized disclosure.
• API and frontend integrations preserve safe source identifiers, access boundaries, evidence presentation, and usable recovery paths where applicable.
• A named owner approves the exact corpus/index/model configurations, remaining limitations, monitoring of freshness/quality, and incident response or rollback plan.

Domain-specific approval rule: Block sign-off if unauthorized content can be retrieved or disclosed, critical source revocations fail to propagate within approved bounds, harmful prompt injection enables unapproved actions, or high-impact answers present materially unsupported or incorrect citations without accepted mitigation. A plausible generated answer or clickable citation alone is not proof of source fidelity or legal compliance.

Trusted references and scope#

The primary technical references below describe common RAG patterns and specific vendor implementations. Their mechanisms and defaults are conditional, not universal requirements.

• Microsoft Azure AI Search — RAG overview — retrieval architecture, chunking, security boundaries, multi-source retrieval, citation context.
• Microsoft Azure Architecture Center — RAG solution design and evaluation — end-to-end architecture and distinct retrieval and answer-quality evaluation stages.
• Microsoft Azure Architecture Center — Retrieval phase — hybrid search, relevant passage ranking, reranking, recall/precision tradeoffs, latency.
• Microsoft Azure Architecture Center — LLM end-to-end evaluation — groundedness, completeness, relevance, correctness, and evaluation limitations.
• Microsoft Azure Architecture Center — RAG prompt engineering — source-grounding instructions, abstention, citations, handling conflicting documents.
• Amazon Bedrock — Knowledge-base chunking — structure-aware chunking, metadata, and treatment of tables and multimodal content.
• Amazon Bedrock — Knowledge-base metadata — document and per-chunk metadata for search filters and provenance.
• Amazon Bedrock — Data-source synchronization — source ingestion, synchronization, and operational failure handling.
• OpenAI API — Retrieval guide — vector stores, file chunking, metadata filters, indexing status, and eventual deletion consistency.
• OWASP — Vector and Embedding Weaknesses — injection, poisoning, tenant isolation, and sensitive information risks in RAG.
• OWASP — LLM Prompt Injection Prevention Cheat Sheet — trust boundaries for retrieved text and limits of prompt-only defenses.

Applicability: Azure AI Search, Amazon Bedrock, and OpenAI Retrieval support different indexing, chunking, metadata, and deletion behaviors; validate deployed service versions and settings rather than transferring defaults between vendors. Some retrieval operations and deletions can be eventually consistent. Embeddings and vector similarity are not a substitute for authorization. A high retrieval score, citation, or instruction to ground responses does not guarantee factual accuracy, privacy protection, or resistance to prompt injection. Never ingest protected content without the appropriate processing rights and scope.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

• Test focus: retrieval relevance, groundedness, citations, freshness, and document access; include API and service behavior where applicable.
• Cross-stack: promptfoo.
• Python: Ragas + pytest; for pytest-based suites, run python -m pytest -q.
• TypeScript: promptfoo + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
• JavaScript: promptfoo + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
• Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

• rag-implementation — Check ingestion, retrieval, and grounding architecture (wshobson/agents, 42,437 repo stars).
• llm-evaluation — Check retrieval/citation evaluation design (wshobson/agents, 42,437 repo stars).
• pinecone-rag — Inspect index/retrieval behavior where Pinecone is used (github/awesome-copilot, 41,073 repo stars).

How to check: In a project-specific copy, have the agent audit the backend / API scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.