Free tool · 03 / WRITE
A/B test calculator.Is the winning email really better? In seconds, no signup.
Says if one cold email really out-replies another, and how many sends a fair test needs.
Runs in your browser. Nothing you type is sent to us.
Email A
Email B
- A's reply rate
- 1%
- B's reply rate
- 4.5%
- B against A
- +350%
- Chance of a gap this big
- 6.2%
B replied at 4.5% against A's 1%. If the two emails were equally good, a gap this big would still happen 6.2% of the time, so it isn't significant at 95% confidence yet.
A fair test of a gap this size needs about 339 sends per variant at 95% confidence and 80% power. Decide the number first and stop there, not when it first looks significant.
A website A/B calculator's z-test would give 3.2% and call a winner. Fisher's exact test doesn't rest on an approximation, so trust it here.
Planning a test? Sends needed per variant
1,502 sends per variant, 3,004 in all.
An example: two made-up emails, 200 sends each. Everything runs in your browser.
When the test is done, send the winner to the right people. Valley: 15-45% reply rates, against a category average of about 2%. See how Valley does it ↗
NEXT STEP
Fed up with tests that never find a winner.Valley reads up on every person first.
Book a demo ↗$500 a month for your first seat, $200 for each seat after that.
Sends you need
How many emails do you need for an A/B test?Sends per variant, at 95% confidence and 80% power.
It depends on your reply rate and the gap you want to find. To tell a 3% reply rate from 5% at 95% confidence and 80% power takes about 1,502 sends per variant; smaller gaps and lower rates take more.
| Reply rate now | Reply rate to detect | Sends per variant | Sends in all |
|---|---|---|---|
| 1% | 2% | 2,313 | 4,626 |
| 2% | 4% | 1,137 | 2,274 |
| 3% | 5% | 1,502 | 3,004 |
| 5% | 8% | 1,055 | 2,110 |
| 10% | 15% | 683 | 1,366 |
n = (z₁₋α/₂ + z₁₋β)² × [p₁(1 − p₁) + p₂(1 − p₂)] ÷ (p₁ − p₂)², with z = 1.96 for 95% confidence and 0.84 for 80% power, rounded up. It reproduces the source's worked example: 20% against 30% needs 291 per group.
From Kim HY (2016), Restorative Dentistry & Endodontics: sample size for two independent proportions ↗, read 20 Nov 2026
How it works
How this tool works.And what it can't see.
Type each email's sends and replies, and choose the confidence level: 95% is the usual one. Everything runs in your browser.
The calculator runs Fisher's exact test, two-sided: with the totals fixed, it adds up the probability of every split of the replies no more likely than yours. That's the chance of a gap this big if the two emails were equally good, and at 5% or less it's significant at 95% confidence. Website A/B calculators use the pooled z-test, an approximation that needs at least 5 expected replies and non-replies in each variant; it's shown beside the exact test, with a warning when that fails or the two disagree.
The sends needed come from the standard formula for comparing two proportions at 80% power, which assumes the normal approximation, so treat it as a floor. Decide the number before you start and stop there: stopping at the first significant moment makes the reported significance meaningless. A significant result says the gap is unlikely to be chance, not that it's big.
Built by Valley.
Sources
- NIST/SEMATECH e-Handbook of Statistical Methods §7.3.3: comparing two proportions ↗read 20 Nov 2026
- R documentation: fisher.test, Fisher's exact test for count data ↗read 20 Nov 2026
- Penn State STAT 500 §8.2: the 2×2 table, a test of two independent proportions ↗read 20 Nov 2026
- Kim HY (2016), Restorative Dentistry & Endodontics: sample size for two independent proportions ↗read 20 Nov 2026
- Penn State STAT 200 §5.1: introduction to hypothesis testing ↗read 20 Nov 2026
- Evan Miller: How Not To Run an A/B Test (2010) ↗read 20 Nov 2026
- American Statistical Association: statement on p-values, press release (7 March 2016) ↗read 20 Nov 2026
Testing cold email
Know when a winner is a winner.Small numbers need the exact test.
Cold email tests are small: a few hundred sends and a handful of replies per variant. That's exactly where the usual calculator's shortcut stops working.
- 01
Why Fisher's exact test
NIST's handbook calls Fisher's exact test an excellent technique when the two samples are small: it works out the probability of the table exactly, with the totals fixed. The z-test is the large-sample version, and Penn State's notes give its condition: all four expected cells of the 2×2 table at least 5. A test with 200 sends a variant and a 2% reply rate expects 4 replies in each, under it.
From NIST/SEMATECH e-Handbook of Statistical Methods §7.3.3: comparing two proportions ↗ · Penn State STAT 500 §8.2: the 2×2 table, a test of two independent proportions ↗, read 20 Nov 2026
- 02
Fix the sample size first
Evan Miller's well-known warning: significance assumes the sample size was fixed in advance. Run a test until you see a significant difference and the reported significance levels become meaningless; peek ten times and what looks like 1% significance is really 5%. Work out the sends per variant first, with the formula in the table above, and read the result once.
From Evan Miller: How Not To Run an A/B Test (2010) ↗ · Kim HY (2016), Restorative Dentistry & Endodontics: sample size for two independent proportions ↗, read 20 Nov 2026
- 03
What significant means
The American Statistical Association's 2016 principles on p-values: a p-value doesn't measure the size of an effect or the importance of a result, and business decisions shouldn't rest only on whether it passes a threshold. A real one-point lift can still be too small to matter.
Tests tune the words. Who gets the email moves reply rates the most. See how Valley does it ↗
From American Statistical Association: statement on p-values, press release (7 March 2016) ↗ · Penn State STAT 200 §5.1: introduction to hypothesis testing ↗, read 20 Nov 2026
Questions
Questions people ask.
- How do I know if my A/B test is statistically significant?
- Type each variant's sends and replies. If the chance of a gap that big, with equally good emails, is under 5%, it's significant at 95% confidence.
- Which test should I use for low reply rates?
- Fisher's exact test. The z-test most calculators use needs at least 5 expected replies and non-replies per variant; small cold email tests often have fewer.
- How many emails do I need for an A/B test?
- About 1,502 per variant to tell a 3% reply rate from 5% at 95% confidence and 80% power. Smaller gaps take more.
- What does significant at 95% confidence mean?
- That a gap as big as yours would happen 5% of the time or less if the two emails were equally good: a p-value at or under 0.05.
- Can I stop the test as soon as it's significant?
- No. Checking as you go and stopping at the first significant result inflates false winners. Decide the sends first.
- What is a p-value?
- The probability of a result as extreme as yours or more, if there were no real difference. It doesn't measure how big the difference is.
- Is this A/B test calculator free?
- Yes. It's free, needs no signup and runs in your browser.
Keep going
Try one of these next.
- Cold email calculatorOutbound planningWorks back from the meetings you want to the people to contact and emails to send.Open the tool
- Email spam checkerEmail outreachScores a cold email out of 100 and flags links, money signs, em dashes and merge tags.Open the tool
- Spintax checkerMessagingChecks spintax, counts every version and converts it between six cold email tools.Open the tool
- Readability checkerMessagingScores any text's Flesch Reading Ease and grade level, and marks the long sentences.Open the tool
- Buyer persona builderMessagingA B2B buyer persona for each role in the deal, side by side, to copy, print or share.Open the tool