Stress test data conclusions with counterarguments

Challenge data findings and business recommendations before presenting them. Use this to catch false correlations, survivorship bias, and hidden assumptions.

You getCore Vulnerabilities, skeptical Audit Questions and data Validation Checklist

Prompt

Prompt
Act as a skeptical principal data scientist and strategy auditor. Stress-test the analysis and conclusions provided below to uncover hidden biases and weak assumptions.

Rules:
- Challenge assertions based on correlation versus causation.
- Identify data blind spots, such as selection bias, survivor bias, seasonality, or missing cohorts.
- Formulate tough, specific counterarguments that a critical executive or competitor would raise.
- Do not affirm the findings; prioritize finding structural vulnerabilities in the logic.

Output format:
1. Core Vulnerabilities (table: Claim Made, Logical Weakness, Alternative Explanation)
2. Skeptical Audit Questions (5 challenging questions the analyst must answer)
3. Data Validation Checklist (specific checks needed to defend the thesis)

Data conclusions and context are below:

Example output

Example for: report concluding that a new onboarding tutorial increased 30-day user retention

1. Core Vulnerabilities
Claim MadeLogical WeaknessAlternative Explanation
Tutorial lifts 30-day retention by 22%Self-selection biasUsers who voluntarily finish a 10-minute tutorial are already more motivated than average users.
Drop-off at step 3 reflects feature complexityMissing cohort contextStep 3 coincides with mandatory billing detail entry, not product difficulty.
Adoption proves feature valueSurvivorship biasAnalysis only measures retained users; churned users may have left precisely because of forced onboarding flows.
2. Skeptical Audit Questions
  1. Did a true randomized holdout group skip the tutorial entirely during the exact same calendar window?
  2. How did paid acquisition channel mix shift during this test period compared to the preceding baseline?
  3. What was the net day-1 bounce rate for accounts subjected to the mandatory walkthrough versus the control?
  4. Did the cohort that completed the tutorial generate higher lifetime value, or merely log more empty sessions?
  5. Have you controlled for seasonality, such as quarter-end business signups that historically show higher retention?
3. Data Validation Checklist
  • Re-run retention curves comparing tutorial finishers against non-finishers within an identical organic search acquisition cohort.
  • Verify whether user activation milestones (e.g., inviting a teammate) occurred before or after tutorial completion.
  • Check whether users who closed the tutorial midway retained at lower rates than those who never received it.
Written by AI from this prompt. Your results will vary.

Paste your draft insights, presentation claims, or memo conclusions into ChatGPT, Claude or Gemini. Follow up by asking the model to suggest control groups that could definitively prove or disprove your primary hypothesis.

Make it repeatable

Keep using this prompt without pasting it in every time.

Keep exploring

More in Data Analysis