Interpret A/B Test Results and Decide Whether to Ship, Iterate or Kill

Paste your A/B test numbers and get a Ship, Iterate or Kill verdict backed by significance, sample ratio, guardrail checks and a list of what the data cannot prove.

You getVerdict, data quality table and metric results table

Prompt

Prompt
Act as a senior product analyst who reviews A/B test readouts before launch decisions.

I will paste my A/B test results right after this prompt. They may include variant names, sample sizes, the primary metric, secondary and guardrail metrics, test duration, and any p-values or confidence intervals. Interpret them and recommend Ship, Iterate or Kill.

Rules:
1. Check sample ratio mismatch first. Compare the actual traffic split to the intended split and flag any gap a chi-square test would call significant (p < 0.01).
2. For the primary metric, report absolute and relative lift with a 95% confidence interval. If I give only counts, compute it and show the method in one line.
3. Check every guardrail metric. Any significant harm blocks a Ship verdict.
4. Flag a test that ran less than one full week or stopped early, and explain the risk of novelty effects or peeking.
5. Use only the numbers I provide. If something is missing, say so instead of guessing.

Output format:
1. Verdict: Ship, Iterate or Kill, with a one-sentence reason
2. Data quality table: Check | Result | Pass/Flag
3. Metric results table: Metric | Control | Variant | Lift | 95% CI | Significant?
4. Guardrail review
5. What this data cannot prove (3 bullets)
6. Recommended next step

If I did not state the intended split, assume 50/50 and say so.

Example output

At a glanceCheckout redesign test: control vs variantVerdict: Iterate. Conversion lift is significant, but page load regressed.
ItemControlVariant
Checkout conversion6.00%6.60%
Avg order value$54.20$53.10
Revenue per visitor$3.25$3.50
Page load p751.8s2.4s
Refund rate2.1%2.3%
Support contacts per 1k orders1112

Example for: a checkout redesign test at an online homewares store, 14 days, intended 50/50 split.

1. Verdict: Iterate

The new checkout lifts conversion significantly, but a significant page-load regression on a guardrail metric blocks shipping as-is.

2. Data quality
CheckResultPass/Flag
Sample ratio (intended 50/50)24,812 vs 25,104; chi-square 1.71, p ≈ 0.19Pass
Duration14 days, two full weekly cyclesPass
Sample size~25k per arm, enough to detect a 0.5pp changePass
Early stoppingYou did not say whether results were checked mid-testFlag
3. Metric results
MetricControlVariantLift95% CISignificant?
Checkout conversion6.00% (1,489)6.60% (1,657)+0.60pp (+10.0%)+0.17pp to +1.03ppYes, p ≈ 0.006
Avg order value$54.20$53.10-2.0%-4.1% to +0.1% (your figure)No
Revenue per visitor$3.25$3.50+7.7%Not computable, no variance givenUnknown

Method: two-proportion z-test with an unpooled standard error of 0.218pp.

4. Guardrail review
  • Page load p75: 1.8s to 2.4s, which you reported as significant. This is harm and blocks Ship.
  • Refund rate: 2.1% to 2.3%, not significant. Watch it, since refunds lag orders by up to 30 days.
  • Support contacts per 1k orders: 11 vs 12, not significant.
5. What this data cannot prove
  • That the lift lasts. Fourteen days cannot rule out a novelty effect for returning customers.
  • Which change caused the lift. The redesign bundled a new layout and Apple Pay, so the effects are mixed together.
  • That revenue rose. The slightly lower order value means revenue per visitor needs its own confidence interval.
6. Recommended next step

Fix the page-load regression. The likely cause is the payment widget script. Then rerun for 14 days with page load as a pre-registered guardrail. Ask your data team for the revenue-per-visitor variance so revenue impact can be tested directly.

Written by AI from this prompt. Your results will vary.

Tips and common edits

Why this works

  • Checking sample ratio mismatch first catches broken randomization before anyone trusts the lift.
  • Making guardrail harm an automatic Ship blocker stops a conversion win from hiding a cost elsewhere.
  • The 'cannot prove' section pushes back on overclaiming in launch reviews.

Use this before a launch review or experiment readout. Have the raw counts per variant, test dates and guardrail metrics ready to paste into ChatGPT, Claude or Gemini. Raw counts give better results than percentages alone.

Common edits

  • Change the significance threshold to 90% if your team accepts more risk for low-stakes UI tests.
  • Add a line asking for a one-paragraph Slack summary for stakeholders.

Prompt details

By
Prompts-GPT editors
Published

Did this prompt work?

Keep exploring

More in Product Management