Create an anomaly investigation checklist for analytics teams

Run a disciplined diagnostic process before investigating metric shifts. Use this checklist whenever unexpected data spikes or drops appear in reports.

You getPhase 1, phase 2 and phase 3

Prompt

Prompt
Act as a staff analytics engineer. Create an operational troubleshooting checklist for investigating unexpected KPI spikes or drops.

- Prioritize simple data collection and pipeline errors before assuming genuine user behavior changes.
- Include specific technical checks covering tracking code, data warehouse pipelines, bot traffic, and external factors.
- Keep every step practical, fast to verify, and checkable by junior analysts.

Output format:
1. Phase 1: Telemetry and Tracking Integrity (4 checklist items)
2. Phase 2: Pipeline and ETL Freshness (3 checklist items)
3. Phase 3: Traffic Composition and Slicing (4 checklist items)
4. Phase 4: Business and External Validation (3 checklist items)
5. Escalation Decision Rule (when to notify leadership vs. resolve silently)

Example output

  1. Phase 1: Telemetry and Tracking Integrity
  • [ ] Compare event count to unique user count: Verify that the ratio has not suddenly changed, which indicates duplicate firing loops.
  • [ ] Check deployment logs: Confirm whether a web or app frontend deployment occurred within 24 hours of the shift.
  • [ ] Inspect tag manager changes: Verify whether tracking tags, triggers, or container versions were updated or republished.
  • [ ] Test client-side console: Open production environments in browser dev tools to confirm the event fires once per expected user action.
  1. Phase 2: Pipeline and ETL Freshness
  • [ ] Check sync completion timestamps: Ensure raw extraction pipelines (e.g., Fivetran, Airbyte) finished without silent partial loads.
  • [ ] Verify transformation run logs: Review dbt or SQL scheduler runs for failed upstream dependency models or timeouts.
  • [ ] Inspect deduplication logic: Confirm primary key uniqueness checks passed on the reporting table.
  1. Phase 3: Traffic Composition and Slicing
  • [ ] Slice by platform and OS: Identify whether the change is isolated to a single client type or app build.
  • [ ] Check geographic and IP distribution: Look for sudden clustering from a single IP range, cloud host, or foreign data center indicative of web scrapers.
  • [ ] Evaluate acquisition source: Break down the metric by UTM source/medium to check for broken landing page redirects or untagged campaigns.
  • [ ] Check new vs. returning users: Isolate whether top-of-funnel traffic or existing user behavior accounts for the shift.
  1. Phase 4: Business and External Validation
  • [ ] Match with payment ledger: For revenue metrics, verify matching transaction records in payment gateways or bank accounts.
  • [ ] Check calendar events: Confirm whether national holidays, seasonal sales events, or partner promotions occurred on that date.
  • [ ] Review customer support channels: Scan inbound tickets and outage boards for spikes in user-reported bugs.
  1. Escalation Decision Rule
    Escalate immediately to stakeholders if the anomaly affects financial reporting, core conversion funnels, or has persisted for more than 4 business hours. Resolve internally without executive alerts if the discrepancy is confirmed to be an isolated pipeline delay catching up on backfill.
Written by AI from this prompt. Your results will vary.

Copy and paste this prompt directly into ChatGPT, Claude, or Gemini without any edits. As a follow-up, ask the model to adapt Phase 1 specifically for mobile SDK event tracking.

Make it repeatable

Keep using this prompt without pasting it in every time.

Keep exploring

More in Data Analysis