PADI Travel Monitor Runbooks

Monitor: Datadog Agent unavailable

Meaning and impact

One production host has stopped reporting datadog.agent.up. Host metrics, logs, traces, process data, and systemd checks from that host may be missing. The application may still be serving customers normally.

Respond

1. Record the affected host group and determine whether it is the web host (i-00c3563dac78c97cc) or Celery host (i-00fa9eb40442ebf69).
2. Check Public site and API healthcheck independently.
3. Correlate with the estate-wide telemetry, EC2 status, and systemd monitors. If customer checks are healthy, classify this as lost visibility rather than an application outage.

Diagnose

On only the affected host, capture:

systemctl is-active datadog-agent.service
systemctl status datadog-agent.service --no-pager
datadog-agent health
journalctl -u datadog-agent.service --since '-30 minutes' --no-pager
ss -lntp | grep -E ':(8126|8125)\b'
df -h
df -i

Check for a recent Agent/package/config deployment, invalid YAML, intake/DNS errors, exhausted disk/inodes, or an EC2/network event. Do not expose Agent keys or dump the complete rendered configuration into the incident channel.

Recover

• If the service is healthy but intake is failing, fix the shared DNS/network or Datadog intake problem before restarting the Agent.
• If the Agent is stopped and the cause is understood, obtain incident-lead approval to restart only datadog-agent.service.
• Repair persistent configuration through server/datadog-agent.sh and the managed deployment path; do not hand-edit /etc/datadog-agent as the fix.

Verify and close

Verify Agent health, port 8126, current host metrics, a new journal event or trace, systemd checks, and monitor recovery. Keep the incident open if customer checks fail after telemetry returns.

Back to monitor index ↑

Monitor: Public site unavailable

Meaning and impact

The host-based HTTPS check cannot complete an end-to-end request to https://travel.padi.com/. The failing boundary may be DNS/TLS, Cloudflare, Nginx, Django/SSR, or the static asset path.

Respond

1. Confirm the failure from a separate network and record status, latency, response headers, certificate result, and a Cloudflare request ID if present.
2. Compare the homepage with /api/v2/healthcheck/. If the API succeeds, focus on frontend/SSR/static delivery. If both fail, continue edge-to-origin.
3. Check recent Cloudflare, deployment, certificate, DNS, and AWS changes.

curl -sS -o /dev/null -D - -w 'status=%{http_code} total=%{time_total}\n' https://travel.padi.com/
curl -sS -o /dev/null -D - -w 'status=%{http_code} total=%{time_total}\n' https://travel.padi.com/api/v2/healthcheck/

Diagnose the first failing boundary

1. DNS/TLS/Cloudflare: verify resolution, certificate validity, Cloudflare status/events, and whether the failure is edge-generated or origin-generated. For a 403, compare the edge response headers/request ID with a permitted origin-side check before deciding whether the cause is a Cloudflare rule or Nginx/origin configuration.
2. Nginx on the web host: capture systemctl status nginx, nginx -t, the recent Nginx journal, and bounded access/error-log evidence.
3. Application: check padi.service, Travel SSR/Next units, listeners 7000 and the deployed version. Do not restart every service together.
4. Static deployment example from the call: if HTML is 2xx but browser JS or CSS chunks return CloudFront/S3 403, treat it as missing/incomplete collectstatic or artifact publication. Restarting Django will not repair it.
5. Search-only symptoms: local Elasticsearch can fail while the homepage and health route remain up; scope that separately.

Contain and recover

• Edge-only regression: revert the exact Cloudflare/DNS/certificate change or bypass only through an approved existing rollback.
• Incomplete deployment: restore the last complete application/static release through the release pipeline. Do not run an untracked ad-hoc production build.
• One failed service: after preserving the journal and identifying the cause, restart only that unit with incident-lead approval.

Verify and close

Verify homepage and healthcheck through edge, representative static chunks, SSR/navigation, current deployment version, Nginx/application stability, and no new 5xx spike. Record whether the fault was edge, origin, app, or static assets.

Back to monitor index ↑

Monitor: Required systemd service inactive

Meaning and impact

A monitored production unit has repeatedly reported a non-active state. Impact depends on the Datadog host and unit group. Do not treat Flower, a queue worker, Nginx, and padi.service as equivalent failures.

Respond

1. Capture the exact host and unit tags from the alert.
2. Map the unit to its exact responsibility: padi_celery_high consumes high; padi_celery_mid consumes mid,high; padi_celery_low consumes low,mid; padi_celery_crowdin consumes crowdin; padi_celery_beat schedules periodic work; and padi_celery_flower provides visibility on port 5555 but does not consume a queue.
3. Confirm the affected customer or operator path. For Celery, check whether an overlapping worker is still consuming the queue before declaring queue loss.

Diagnose

systemctl is-active UNIT.service
systemctl status UNIT.service --no-pager
journalctl -u UNIT.service --since '-45 minutes' --no-pager
systemctl show UNIT.service -p ActiveState -p SubState -p Result -p ExecMainStatus -p NRestarts
systemctl list-units '*celery*' --all

Check the unit's last deployment/config change, dependency availability, exit status, restart loop, listener, disk/inodes, memory pressure, and EC2 events. For Celery, use Flower/APM/logs to inspect queue throughput, failing task names, retries, and oldest work. Do not infer a safe restart from queue depth alone.

Contain and recover

• A poison/retry loop: contain the task producer or exact queue through the approved application path; preserve task identifiers and errors.
• A single stopped unit with healthy dependencies: after cause and rollback are known, restart only that unit with incident-lead approval.
• A deployment mismatch: restore the unit and code/config as one reviewed release. The call notes warn that partial deployments can leave services and static assets on different versions.
• Never run a broad restart '*celery*' or restart web and Celery together.

Verify and close

Verify active/running, listener or queue throughput, no restart loop, recovered customer/operator flow, and Datadog recovery for the original host,unit group.

Back to monitor index ↑

Monitor: AWS EC2 status check failed

Meaning and impact

AWS reports an instance-level or system-level status-check failure on a tagged PADI Travel instance. The web host and Celery host have different customer impact; identify the instance before application remediation.

Respond and diagnose

1. Capture instance_id from Datadog. The production web host is i-00c3563dac78c97cc; the production Celery host is i-00fa9eb40442ebf69.
2. Check AWS instance state, both status checks, scheduled events, recent EC2 changes, and the AWS Health dashboard.
3. Confirm customer impact: web instance failures affect edge/origin traffic; Celery instance failures affect asynchronous work while Redis retains queued messages subject to task/visibility semantics.

aws ec2 describe-instance-status \
  --include-all-instances \
  --instance-ids INSTANCE_ID \
  --query 'InstanceStatuses[0].{state:InstanceState.Name,instance:InstanceStatus,system:SystemStatus,events:Events}'

• System status failed: investigate AWS host/network/power and AWS events.
• Instance status failed: investigate guest OS boot, network, CPU/memory, filesystem, and kernel issues using approved console/SSM diagnostics.
• Correlate with Datadog Agent/systemd/telemetry alerts, but remember a failed Agent can hide application state.

Contain and recover

• Preserve system logs and instance metadata before a reboot or replacement.
• Reboot, stop/start, fail over, replace, or detach storage only with incident lead and infrastructure-owner approval plus a rollback/recovery plan.
• For a Celery-host failure, inspect broker depth and task age after host recovery; do not purge queues.
• For a web-host failure, use only an existing approved maintenance or rollback path; do not point DNS at an unverified host.

Verify and close

Require both AWS checks healthy, instance reachability, required units stable, Datadog telemetry restored, customer/queue smoke checks, and no scheduled event.

Back to monitor index ↑

Monitor: Public API healthcheck unavailable

Meaning and impact

The external check cannot receive a successful response from https://travel.padi.com/api/v2/healthcheck/. The endpoint is shallow Django liveness only; its implementation returns {"status":"ok"} without touching PostgreSQL, Redis, Elasticsearch, or external APIs.

Respond

1. Capture response status, body class, latency, headers, and Cloudflare request ID. Do not paste cookies or tokens.
2. Compare homepage and healthcheck. A homepage-only failure is not this path; a healthcheck-only failure points toward routing or Django URL/application.
3. Correlate with public site, padi.service, EC2, Agent, and estate telemetry monitors. Do not declare a database incident from this signal alone.

curl -sS -D - https://travel.padi.com/api/v2/healthcheck/ -o /tmp/travel-healthcheck-body

Diagnose

1. Edge: DNS/TLS/Cloudflare and whether the response is generated at edge.
2. Nginx: vhost/routing, listener, config test, recent errors, and upstream connection failures on the web host.
3. Django: padi.service, port 7000, recent journal errors, deployed version, and a bounded local request through Nginx with the production host header.
4. Dependencies: investigate PostgreSQL/Redis only when application logs or real customer/API requests show dependency failures. The healthcheck itself does not prove them.

Contain and recover

• Revert the exact edge/routing/deployment regression when identified.
• If padi.service alone is stopped, preserve evidence and restart only that unit after approval and dependency checks.
• If a partial deployment is present, restore a complete known-good release; do not mix manual code, static, and service changes.

Verify and close

Require the exact JSON response through edge and origin, stable padi.service, representative read-only API/browser requests, no related 5xx spike, and monitor recovery. Record that this proves liveness, not full dependency readiness.

Back to monitor index ↑

Monitor: Primary database storage critical

Known monitor defect

The monitor currently queries aws.rds.free_storage_space for dbinstanceidentifier:diviac-logbook. Live AWS inventory confirms that this is a separate 100 GiB PostgreSQL instance. The Django production configuration and infrastructure notes identify Aurora cluster padi-travel-covid19-aurora-cluster as the primary application database.

Therefore this alert does not currently prove that primary application database storage is critical. Correct the monitor separately; do not perform an Aurora storage change solely because this monitor fires.

Respond

1. Confirm the alert's dbinstanceidentifier and distinguish No Data from an actual threshold breach.
2. Check the live status, storage, pending changes, recent events, and CloudWatch trend for the exact diviac-logbook instance.
3. Identify its current owner/workload before assigning customer impact.
4. Separately check the Aurora cluster and real Travel database symptoms.

aws rds describe-db-instances \
  --db-instance-identifier diviac-logbook \
  --query 'DBInstances[0].{status:DBInstanceStatus,storage:AllocatedStorage,pending:PendingModifiedValues}'

aws rds describe-db-clusters \
  --db-cluster-identifier padi-travel-covid19-aurora-cluster \
  --query 'DBClusters[0].{status:Status,members:DBClusterMembers}'

Contain and recover

• Do not delete logs/data, vacuum aggressively, resize storage, or fail over a database as the first step.
• If diviac-logbook genuinely needs capacity, obtain its owner and DBA approval for the supported RDS storage change and rollback/risk plan.
• If Aurora is actually affected, open a separate primary-database incident based on Aurora metrics/events and application errors.

Verify and close

Verify the exact instance metric recovers with safe headroom, pending changes finish, owning workload is healthy, and the monitor is corrected or renamed so future responders are not directed to the wrong database.

Back to monitor index ↑

Monitor: Primary database unavailable

Known monitor defects

The monitor targets dbinstanceidentifier:diviac-logbook, not the Aurora cluster used by Django default and long. It also treats fewer than one connection for 15 minutes as unavailability; an idle database can legitimately have zero connections. This signal alone is insufficient for a P1 primary-database outage.

Respond

1. Confirm the exact target and whether this is threshold or No Data.
2. Check real customer/API failures, Django database exceptions, Aurora status, failover/events, writer availability, and current connection/error trends.
3. Check diviac-logbook separately and determine its owning workload.

aws rds describe-db-instances \
  --db-instance-identifier diviac-logbook \
  --query 'DBInstances[0].{status:DBInstanceStatus,pending:PendingModifiedValues}'

aws rds describe-db-clusters \
  --db-cluster-identifier padi-travel-covid19-aurora-cluster \
  --query 'DBClusters[0].{status:Status,writer:DBClusterMembers[?IsClusterWriter==`true`].DBInstanceIdentifier,events:PendingModifiedValues}'

Diagnose a confirmed Aurora incident

Inspect RDS events, failover state, writer/reader roles, connections, latency, locks, CPU/memory, storage/I/O, security-group/network changes, and recent schema or deployment changes. Correlate with padi.service logs and representative read-only application requests. Do not use the shallow healthcheck as proof of database health.

Contain and recover

• Do not reboot/fail over Aurora, terminate sessions, change parameter groups, or promote a reader without DBA/infrastructure approval and a rollback plan.
• If a deployment caused connection exhaustion or invalid credentials, contain or roll back that release through the supported pipeline.
• Preserve query/session evidence; never paste SQL containing customer data.

Verify and close

Verify Aurora writer/reader status, application queries, error/latency trend, booking-safe read paths, and monitor correction. Do not close merely because diviac-logbook reports a connection.

Back to monitor index ↑

Monitor: All application instances unavailable

Meaning and impact

No production host is reporting systemd uptime for padi.service, the main Django Gunicorn application on the web host. This may mean the unit is down or the metric is missing; correlate with Agent/telemetry before declaring outage.

Respond

1. Check public site and API healthcheck from outside the VPC.
2. Correlate with Datadog Agent, estate telemetry, EC2 status, and required systemd monitors.
3. On the web host, capture unit state, exit result, restarts, journal, port 7000, Nginx upstream errors, and recent deployment state.

systemctl is-active padi.service
systemctl status padi.service --no-pager
systemctl show padi.service -p ActiveState -p SubState -p Result -p ExecMainStatus -p NRestarts
journalctl -u padi.service --since '-45 minutes' --no-pager
ss -lntp | grep -E ':7000\b'

Check memory pressure/OOM, configuration or secret lookup failures, database connectivity, port conflicts, migrations, disk/inodes, and release mismatch. The call notes document a partial OOM deployment; compare code, static assets, and service version rather than assuming a clean release.

Contain and recover

• If only telemetry is absent but port 7000 and customer paths are healthy, follow the Agent runbook instead of restarting Django.
• If padi.service is stopped, preserve evidence and confirm dependencies. Then obtain incident-lead approval to restart only padi.service.
• If it fails again, stop the restart loop and restore the last complete known-good release. Do not restart Celery or every frontend service.

Verify and close

Require stable unit state and restart count, listener 7000, local and edge healthcheck, representative customer/API flows, current telemetry, and no new Gunicorn/Nginx/database error spike.

Back to monitor index ↑

Monitor: Estate-wide telemetry unavailable

Meaning and impact

Neither production host is reporting datadog.agent.running. Datadog may be blind across the web and Celery estates. This can be two host failures, a shared Agent/config/intake/network failure, or a tag/query problem. Customer traffic and queues may still be healthy.

Respond

1. Independently test the public site and API healthcheck.
2. Check AWS EC2 status for both production instances.
3. Check Agent service/health locally on both hosts through the approved access path. Compare failure timestamps: simultaneous loss suggests a shared change or intake path; separate loss suggests host-local faults.
4. Preserve CloudWatch/Nginx/application evidence because Datadog history may be incomplete during the incident.

On each host:

systemctl is-active datadog-agent.service
systemctl status datadog-agent.service --no-pager
datadog-agent health
journalctl -u datadog-agent.service --since '-45 minutes' --no-pager

Diagnose

• Agents stopped: check package/config deployment, keys without printing them, disk/inodes, OOM, and service dependency errors.
• Agents healthy but data absent: check Datadog intake status, DNS/TLS/egress, time synchronization, account/integration state, and required tags.
• Only datadog.agent.running absent while other telemetry is present: treat as a monitor/query/instrumentation defect, not an estate outage.
• Use CloudWatch and AWS console to retain visibility while Datadog is impaired.

Contain and recover

• Fix a shared network/intake/config problem once, then canary one host before applying to the second.
• Restart only the Agent on one host after evidence and approval; verify intake before repeating. Do not restart application or Celery units to repair telemetry.
• Reconcile persistent configuration through the managed Agent deployment.

Verify and close

Require current Agent-running metrics from both hosts, logs/traces/systemd data, healthy public checks, observable Celery workers/queues, and monitor recovery. Record the telemetry gap so later incident analysis does not treat it as zero traffic or zero errors.

Back to monitor index ↑