Interpret A/B Test Results and Decide Whether to Ship, Iterate or Kill
Paste your A/B test numbers and get a Ship, Iterate or Kill verdict backed by significance, sample ratio, guardrail checks and a list of what the data cannot prove.
You getVerdict, data quality table and metric results table
Prompt
Act as a senior product analyst who reviews A/B test readouts before launch decisions.
I will paste my A/B test results right after this prompt. They may include variant names, sample sizes, the primary metric, secondary and guardrail metrics, test duration, and any p-values or confidence intervals. Interpret them and recommend Ship, Iterate or Kill.
Rules:
1. Check sample ratio mismatch first. Compare the actual traffic split to the intended split and flag any gap a chi-square test would call significant (p < 0.01).
2. For the primary metric, report absolute and relative lift with a 95% confidence interval. If I give only counts, compute it and show the method in one line.
3. Check every guardrail metric. Any significant harm blocks a Ship verdict.
4. Flag a test that ran less than one full week or stopped early, and explain the risk of novelty effects or peeking.
5. Use only the numbers I provide. If something is missing, say so instead of guessing.
Output format:
1. Verdict: Ship, Iterate or Kill, with a one-sentence reason
2. Data quality table: Check | Result | Pass/Flag
3. Metric results table: Metric | Control | Variant | Lift | 95% CI | Significant?
4. Guardrail review
5. What this data cannot prove (3 bullets)
6. Recommended next step
If I did not state the intended split, assume 50/50 and say so.Example output
Tips and common edits
Why this works
- Checking sample ratio mismatch first catches broken randomization before anyone trusts the lift.
- Making guardrail harm an automatic Ship blocker stops a conversion win from hiding a cost elsewhere.
- The 'cannot prove' section pushes back on overclaiming in launch reviews.
Use this before a launch review or experiment readout. Have the raw counts per variant, test dates and guardrail metrics ready to paste into ChatGPT, Claude or Gemini. Raw counts give better results than percentages alone.
Common edits
- Change the significance threshold to 90% if your team accepts more risk for low-stakes UI tests.
- Add a line asking for a one-paragraph Slack summary for stakeholders.
Prompt details
- By
- Prompts-GPT editors
- Published
Did this prompt work?
Keep exploring
More like this
Design an A/B Test Plan with Sample Size Math and Pre-Set Stop Rules
Get a complete A/B test plan with one primary metric, a clear hypothesis, a worked sample size calculation, runtime, and stop rules fixed before launch.
Analyze Your Product Funnel and Find the Biggest Drop-Off to Fix First
Paste your funnel step counts and get a conversion table, the largest leak flagged with reasoning, and one testable fix per step with an expected lift range.
Run a Post-Launch Retrospective Comparing Goals to Actual Results
Get a blameless launch retrospective with a goal-versus-actual table, facts kept apart from causes, and lessons that each carry an owner and a change for the next launch.
Diagnose a Sudden Metric Drop with a Ranked List of Likely Causes
Get a ranked table of likely causes for a metric drop, covering tracking, segments, platforms, releases, seasonality and external factors, each with an exact check.
Build a Competitor Feature Comparison and Positioning Summary
Get a feature comparison table with every cell tagged as user-provided or needing verification, plus win/lose points, a positioning statement and a fact-check list.
Define a North Star and Success Metrics for a New Feature with Guardrails
Get a North Star metric, up to three primary metrics each paired with a guardrail, exact formulas from your existing data, targets, and ship or roll-back rules.