12 — Monitoring and Observability Checklist#

Verify that critical services and user journeys can be understood through trustworthy telemetry, actionable alerts, service-level measurements, and effective incident investigation. Monitoring systems must also be reliable, proportionate, and privacy-conscious.

Scope: Shared / cross-cutting. This file is one part of the 12_Monitoring_and_Observability.md checklist. Apply the domain-wide release gate across all applicable scope files; each checklist ID remains unique.

Review details#

• Project / product:
• Release / version / environment:
• Scope and applicable jurisdictions / markets:
• Critical services / user journeys:
• Monitoring and incident-response owners:
• SLI / SLO / alerting policy reference:
• Reviewer / owner:
• Review date:
• Decision: Pending / Approved / Approved with documented exceptions / Blocked

How to use#

1. Copy the shared template into project-specific review records; leave all checkboxes in this repository unchecked.
2. Mark [x] only after verifying the item through instrumentation configuration, actual dashboards and queries, sample logs/traces, controlled alert tests, incident drills, or other documented evidence. Record failed checks and N/A reasons in project-specific notes or issue tracking keyed by OBS ID.
3. Identify user-critical journeys and their real failure modes before selecting metrics, alert thresholds, or telemetry vendors. Adjust monitoring depth to expected operational risk and staff coverage.
4. Review ## API Checks and ## Frontend Checks separately when relevant; record genuinely inapplicable checks outside the shared template.
5. Reassess instrumentation and alerts after deployments, new data flows, new critical features, incidents, and changes to support arrangements. Monitoring review is not a substitute for separate Security, Privacy, Reliability, or Performance reviews.

Telemetry and instrumentation#

• OBS-003 Instrument user-relevant latency, traffic, error, and saturation signals for critical operations.

• OBS-004 Define consistent metric names, descriptions, units, event types, and attributes so data from related components is comparable.

• OBS-017 Set retention, access, encryption, and deletion rules for logs, metrics, traces, and incident records based on risk and applicable obligations.

Alerting and on-call#

• OBS-025 Define on-call coverage, escalation contacts, and response ownership for every business-critical service.
• OBS-027 Define alert severity using expected customer impact, urgency, and required response time.
• OBS-028 Create useful alert descriptions containing affected service, symptom, threshold, time window, and supporting evidence.
• OBS-029 Link alerts to current dashboards, service ownership, and an appropriate remediation or investigation runbook.
• OBS-030 Separate immediate paging alerts from non-urgent tickets, informational notifications, and trend reports.
• OBS-031 Use suitable pending periods, hysteresis, or grouping to reduce transient alert flapping without hiding sustained incidents.
• OBS-032 Deduplicate and correlate related alerts so a single dependency failure does not flood responders with redundant pages.
• OBS-033 Configure silencing, maintenance windows, and inhibition rules with explicit ownership and expiry where supported.
• OBS-034 Define alert routing for business hours, overnight coverage, regional ownership, and emergency fallback.
• OBS-035 Test that alerts actually reach the intended responder through the configured notification path.
• OBS-036 Exercise escalation when the primary responder is unavailable or an alert remains unacknowledged.
• OBS-039 Test representative failure scenarios and verify the associated alerts fire at a useful time and severity.
• OBS-040 Review false positives, missed incidents, repeated noisy rules, and response workload on a regular cadence.
• OBS-041 Ensure critical alert thresholds are related to approved service objectives or demonstrated risk, rather than arbitrary defaults.
• OBS-042 Document how responders can manually declare an incident when monitoring has not yet detected the problem.
• OBS-043 Check that alert routing and contact information remain correct after team or ownership changes.
• OBS-044 Retire obsolete alerts and dashboards when services, dependencies, or metrics are removed.

Service-level indicators and objectives#

• OBS-045 Identify user-visible services and critical transactions that deserve explicit reliability indicators.

• OBS-046 Define each service-level indicator (SLI) with a precise numerator, denominator, scope, exclusions, and measurement source.

• OBS-049 Include traffic, errors, latency, and saturation views for services where those signals are meaningful.

• OBS-050 Set service-level objectives (SLOs) and evaluation windows that match risk, user expectations, and operational capacity.

• OBS-051 Document whether planned maintenance and dependency failures count against SLOs and customer commitments.

• OBS-052 Ensure availability measurements include sufficiently representative traffic and account for low-volume services.

• OBS-054 Track percentile latency and error behavior by relevant service, operation, region, or user segment where justified.

• OBS-055 Make error-budget remaining and consumption trends visible to service owners when error budgets are used.

• OBS-057 Use low-traffic-appropriate detection methods rather than applying high-volume percentage alerts blindly.

• OBS-059 Distinguish service objectives from individual release performance tests and contractual service-level agreements.

• OBS-060 Ensure dashboards communicate metric units, time ranges, target thresholds, freshness, and known caveats.

• OBS-061 Review SLI and SLO relevance after changes to user journeys, traffic mix, product boundaries, or business commitments.

• OBS-062 Assign owners and review schedules for service objective definitions, measurement quality, and unresolved gaps.

Incident investigation and learning#

• OBS-063 Document incident severity definitions, response roles, communications channels, and escalation conditions.
• OBS-064 Make relevant dashboards, trace search, structured logs, runbooks, and dependency maps discoverable from incident entry points.
• OBS-065 Include a consistent incident identifier or timeline record across alerts, communications, changes, and remediation.
• OBS-068 Validate that responders can traverse from a failed user action to its service requests, dependencies, and error context where feasible.
• OBS-069 Preserve enough diagnostic history to investigate intermittent problems that are discovered after the event.
• OBS-070 Record incident detection time, user impact, affected scope, mitigation actions, and recovery verification.
• OBS-071 Distinguish triggering events, contributing conditions, and systemic weaknesses when analyzing causes.
• OBS-072 Prefer blameless, evidence-based incident reviews over unsupported attribution to individuals.
• OBS-073 Define which incidents require a written postmortem and when the review should occur.
• OBS-074 Capture actions that reduced impact as well as detection gaps, slow investigations, and failed mitigations.
• OBS-075 Assign postmortem follow-up actions owners, priorities, tracking references, and observable completion criteria.
• OBS-076 Verify incident follow-ups are completed and that relevant alerts, tests, dashboards, and runbooks are updated.
• OBS-077 Periodically analyze recurring incident patterns and alert fatigue to identify broader reliability risks.
• OBS-078 Exercise a simulated incident to verify dashboard access, escalation, communication, and diagnosis without harming production.
• OBS-079 Make status updates and customer-facing communications reflect confirmed impact rather than inferred internal health.
• OBS-080 Review incident evidence retention and access controls for confidentiality and applicable policy requirements.
• OBS-081 Keep documented operational handoffs current across on-call changes, teams, and vendor responsibilities.

Release gate#

• Critical user journeys and dependent services have identifiable monitoring owners, useful health indicators, and working observability access.
• Service-level objectives or other accepted reliability expectations have clearly defined measurement and incident-detection methods where applicable.
• Meaningful user-impact alerts reach appropriate responders; alert routing, escalation, and at least one representative trigger have been tested.
• Logs, metrics, traces, and dashboard views needed to investigate a critical failure are usable, correlated where necessary, and tagged with the deployed version or environment.
• Telemetry pipelines report their own health or absence of data, and logging/monitoring does not expose prohibited sensitive information.
• API and frontend error, latency, and critical-journey indicators are reviewed where those interfaces exist, including known collection and sampling limits.
• Incident response roles, escalation procedures, relevant runbooks, and unresolved observability gaps have accountable owners and authorized dispositions in project-specific records.

Domain-specific approval rule: Block monitoring sign-off if a critical user-impacting failure cannot reasonably be detected or investigated, required alerts do not reach responders, monitoring is materially blind without accepted mitigation, or sensitive telemetry exposure remains unresolved. Dashboard existence alone does not establish production observability.

Trusted references and scope#

These checks translate authoritative observability and operations guidance into practical verification tasks. They do not mandate a specific vendor, instrumentation SDK, or SLO target.

• OpenTelemetry — Signals — core telemetry concepts and selecting logs, metrics, and traces.
• OpenTelemetry — Semantic conventions — consistent operation, resource, metric, log, and trace attributes.
• OpenTelemetry — Logs — structured logging, stable schema, and correlation.
• OpenTelemetry — Handling sensitive data — data minimization, redaction, and telemetry privacy.
• OpenTelemetry — Collector resiliency — telemetry queueing, exporter failures, and data-loss controls.
• Google SRE — Monitoring Distributed Systems — four golden signals and symptom-driven monitoring.
• Google SRE Workbook — Monitoring — service and dependency signals and operational dashboards.
• Google SRE Workbook — Alerting on SLOs — error-budget and multi-window burn-rate alerting.
• Google SRE Workbook — Postmortem Culture — blameless incident learning and follow-up ownership.
• Prometheus — Alerting best practices — actionable, user-impact-focused alerts and monitoring of monitoring.
• Prometheus — Alerting rules — alerting thresholds, pending durations, labels, and annotations.
• Grafana — Alerting best practices — routing, severity, useful alerts, and alert noise management.
• web.dev — Web Vitals — client-side loading, interactivity, and layout stability metrics.

Applicability: Monitoring requirements depend on the service's criticality, traffic, architecture, user population, jurisdiction, and available operational coverage. Tracing across distributed services, browser Web Vitals, SLO burn-rate paging, and OpenTelemetry Collector configuration are conditional, not universal. Sample size and low-traffic behavior must be considered before configuring percentage-based alerts. Keep sensitive personal, financial, health, and credential data out of broadly visible telemetry.

Source review date: 2026-11-12.

Suggested testing tools#

Choose tools appropriate to the project's stack and risks; automation supports evidence but does not approve the checklist.

• Test focus: trace propagation, metrics/logs, alert conditions, and SLO visibility; include shared/cross-team controls where applicable.
• Cross-stack: Prometheus promtool + OpenTelemetry.
• Python: OpenTelemetry + pytest; for pytest-based suites, run python -m pytest -q.
• TypeScript: OpenTelemetry + Vitest; for Vitest-based suites, run npx vitest run (or npx playwright test for E2E).
• JavaScript: OpenTelemetry + Jest; for Jest-based suites, run npx jest --ci (or npx playwright test for E2E).
• Evidence: Save tool versions, configuration, test environment, fixtures, results, and links in project-specific review records using the existing checklist IDs. Expert review remains necessary where relevant.

Suggested AI skills for verification#

These are optional agent review aids, not automated passes or release approval. Each linked repository had at least 1,000 GitHub stars when checked on 2026-11-12; counts are per repository, not per skill, and do not indicate security or correctness.

• observability-and-instrumentation — Check telemetry and diagnosis coverage (addyosmani/agent-skills, 116,378 repo stars).
• distributed-tracing — Check tracing across service boundaries (wshobson/agents, 42,437 repo stars).
• slo-implementation — Review alerting and objective measurements (wshobson/agents, 42,437 repo stars).

How to check: In a project-specific copy, have the agent audit the shared / cross-cutting scope and map findings to this file's existing checklist IDs, with code/configuration references, reproducible test results, failed or inapplicable cases, and uncertainties. Independently verify findings; keep shared template boxes unchecked and retain human approval requirements.

Before use: Read the skill's SKILL.md, referenced scripts, license, and requested tool permissions. Only execute vetted skills with appropriate access in a controlled environment.

Important warning: reference only#

⚠️ Reference only — not a definitive or exhaustive checklist. This template is a starting point for review, not the ultimate authority or a guarantee of project safety, quality, regulatory compliance, accessibility, security, or launch readiness.

Use project-specific judgment and independent verification. Requirements and risks evolve and differ by project, technology, and jurisdiction. Adapt the checks, consult current authoritative sources, perform relevant tests, and obtain qualified specialist review or approval when appropriate. Checking every box does not replace these steps.