Design an A/B Test Plan with Sample Size Math and Pre-Set Stop Rules
Get a complete A/B test plan with one primary metric, a clear hypothesis, a worked sample size calculation, runtime, and stop rules fixed before launch.
You getHypothesis, metrics table and sample size calculation
Prompt
Act as a senior product manager who designs experiments with a data scientist.
Help me design an A/B test plan. First ask me these questions in one message, then wait for my answers:
1. What change are you testing, and on which page or flow?
2. What is the main metric and its current baseline rate or average?
3. How many users reach this point per day?
4. What is the smallest improvement worth shipping, and which metrics must not get worse?
Rules:
- Choose exactly one primary metric. Everything else is labeled secondary or guardrail.
- Write the hypothesis as one sentence naming the change, the expected direction and size of effect, and the reason.
- Show the sample size math step by step: formula, inputs, result per variant. Use 95% confidence and 80% power unless I say otherwise. Round runtime up to full weeks.
- Set stop rules before launch: a fixed end date, no stopping early for a win, and early stops only when a guardrail crosses a stated threshold.
- If traffic is too low to detect the effect, say so and offer two options.
Output format:
1. Hypothesis
2. Metrics table (Metric | Type | Baseline | Target or threshold)
3. Sample size calculation
4. Duration and traffic split
5. Stop rules
6. Decision rules (ship, iterate, drop)
End with one line naming the biggest assumption or risk in this plan.Example output
Tips and common edits
Why this works
- Locking one primary metric stops teams from picking whichever number looks best after the test ends.
- Requiring the formula and inputs shown lets you check the sample size instead of trusting a guessed number.
- Writing stop and decision rules before launch removes the temptation to end the test early on a lucky spike.
Use before building any experiment. Have your baseline conversion rate, daily traffic, and smallest worthwhile lift ready. Works in ChatGPT, Claude or Gemini; check the math with your analytics tool.
Common edits
- Change confidence to 90% or power to 90% to fit your team's risk tolerance.
- Ask for a three-variant version with a correction for multiple comparisons.
Prompt details
- By
- Prompts-GPT editors
- Published
Did this prompt work?
Keep exploring
More like this
Interpret A/B Test Results and Decide Whether to Ship, Iterate or Kill
Paste your A/B test numbers and get a Ship, Iterate or Kill verdict backed by significance, sample ratio, guardrail checks and a list of what the data cannot prove.
Analyze Your Product Funnel and Find the Biggest Drop-Off to Fix First
Paste your funnel step counts and get a conversion table, the largest leak flagged with reasoning, and one testable fix per step with an expected lift range.
Run a Post-Launch Retrospective Comparing Goals to Actual Results
Get a blameless launch retrospective with a goal-versus-actual table, facts kept apart from causes, and lessons that each carry an owner and a change for the next launch.
Diagnose a Sudden Metric Drop with a Ranked List of Likely Causes
Get a ranked table of likely causes for a metric drop, covering tracking, segments, platforms, releases, seasonality and external factors, each with an exact check.
Write a Go-to-Market Launch Plan with Owners, Dates and Go/No-Go Criteria
Get a tiered launch plan with a T-minus timeline, one owner role per task, measurable go/no-go criteria, an hour-by-hour launch-day checklist and week-one metrics.
Write an Event Tracking Plan Where Every Event Maps to a Success Metric
Get a tracking plan table with object_action event names, exact triggers, typed properties and the success metric each event feeds, plus a list of events to cut.