Design an A/B Test Plan with Sample Size Math and Pre-Set Stop Rules

Get a complete A/B test plan with one primary metric, a clear hypothesis, a worked sample size calculation, runtime, and stop rules fixed before launch.

You getHypothesis, metrics table and sample size calculation

Prompt

Prompt
Act as a senior product manager who designs experiments with a data scientist.

Help me design an A/B test plan. First ask me these questions in one message, then wait for my answers:
1. What change are you testing, and on which page or flow?
2. What is the main metric and its current baseline rate or average?
3. How many users reach this point per day?
4. What is the smallest improvement worth shipping, and which metrics must not get worse?

Rules:
- Choose exactly one primary metric. Everything else is labeled secondary or guardrail.
- Write the hypothesis as one sentence naming the change, the expected direction and size of effect, and the reason.
- Show the sample size math step by step: formula, inputs, result per variant. Use 95% confidence and 80% power unless I say otherwise. Round runtime up to full weeks.
- Set stop rules before launch: a fixed end date, no stopping early for a win, and early stops only when a guardrail crosses a stated threshold.
- If traffic is too low to detect the effect, say so and offer two options.

Output format:
1. Hypothesis
2. Metrics table (Metric | Type | Baseline | Target or threshold)
3. Sample size calculation
4. Duration and traffic split
5. Stop rules
6. Decision rules (ship, iterate, drop)

End with one line naming the biggest assumption or risk in this plan.

Example output

At a glanceA/B test plan at a glanceOne-page checkout test for a meal-kit app, 4.0% baseline conversion
Target conversion
4.4%
From 4.0% baseline, 10% relative lift
Sample size per variant
39,430
95% confidence, 80% power
Total visitors
78,900
50/50 split by user ID
Runtime
14 days
3 June to 16 June

Example for: a meal-kit subscription app testing a one-page checkout, 4.0% baseline conversion, 6,000 eligible visitors per day.

1. Hypothesis

Replacing the three-step checkout with a single page will raise checkout conversion from 4.0% to at least 4.4% (a 10% relative lift), because session recordings show 38% of drop-offs happen on step 2 (delivery details).

2. Metrics
MetricTypeBaselineTarget or threshold
Checkout conversion (visit to paid order)Primary4.0%4.4% or higher
Average order valueSecondary$62.10Report only
Payment error rateGuardrail1.2%Stop if above 2.0%
First-week cancellation rateGuardrail6.5%Flag if above 7.5%
3. Sample size calculation

Formula (two proportions): n = (Z_α/2 + Z_β)² × [p1(1−p1) + p2(1−p2)] / (p2 − p1)²

  • p1 = 0.040, p2 = 0.044
  • Z_α/2 = 1.96 (95% confidence), Z_β = 0.84 (80% power)
  • (1.96 + 0.84)² = 7.84
  • p1(1−p1) + p2(1−p2) = 0.0384 + 0.0421 = 0.0805
  • (p2 − p1)² = 0.004² = 0.000016
  • n = 7.84 × 0.0805 / 0.000016 ≈ 39,430 per variant
  • Total: about 78,900 visitors
4. Duration and traffic split
  • 50/50 split, randomized by user ID
  • 78,900 ÷ 6,000 per day = 13.2 days, rounded up to 14 days (two full weeks) to cover weekday and weekend patterns
  • Start Monday 3 June, end Sunday 16 June
5. Stop rules
  • No decision on the primary metric before 16 June, even if the dashboard shows significance.
  • Stop early only if payment error rate exceeds 2.0% for 24 hours in the variant.
  • If a tracking bug affects more than 5% of sessions, pause, fix, and restart the count.
6. Decision rules
  • Ship: lift is significant, at least 4.4%, and no guardrail breached.
  • Iterate: positive but not significant, or lift below 4.4%; review step-level drop-off and redesign.
  • Drop: flat or negative result, or any guardrail breach.

Biggest risk: the 4.0% baseline comes from one month that included a holiday promotion, so true conversion may be lower and the test underpowered.

Written by AI from this prompt. Your results will vary.

Tips and common edits

Why this works

  • Locking one primary metric stops teams from picking whichever number looks best after the test ends.
  • Requiring the formula and inputs shown lets you check the sample size instead of trusting a guessed number.
  • Writing stop and decision rules before launch removes the temptation to end the test early on a lucky spike.

Use before building any experiment. Have your baseline conversion rate, daily traffic, and smallest worthwhile lift ready. Works in ChatGPT, Claude or Gemini; check the math with your analytics tool.

Common edits

  • Change confidence to 90% or power to 90% to fit your team's risk tolerance.
  • Ask for a three-variant version with a correction for multiple comparisons.

Prompt details

By
Prompts-GPT editors
Published

Did this prompt work?

Keep exploring

More in Product Management