# A/B test calculator

> Says if one cold email really out-replies another, and how many sends a fair test needs. Free, no signup. Runs in your browser. Nothing you type is sent to us.

**A/B test calculator. Is the winning email really better? In seconds, no signup.** Use it at https://tools.joinvalley.co/ab-test-calculator

## How many emails do you need for an A/B test? Sends per variant, at 95% confidence and 80% power.

It depends on your reply rate and the gap you want to find. To tell a 3% reply rate from 5% at 95% confidence and 80% power takes about 1,502 sends per variant; smaller gaps and lower rates take more.

| Reply rate now | Reply rate to detect | Sends per variant | Sends in all |
| --- | --- | --- | --- |
| 1% | 2% | 2,313 | 4,626 |
| 2% | 4% | 1,137 | 2,274 |
| 3% | 5% | 1,502 | 3,004 |
| 5% | 8% | 1,055 | 2,110 |
| 10% | 15% | 683 | 1,366 |

n = (z₁₋α/₂ + z₁₋β)² × [p₁(1 − p₁) + p₂(1 − p₂)] ÷ (p₁ − p₂)², with z = 1.96 for 95% confidence and 0.84 for 80% power, rounded up. It reproduces the source's worked example: 20% against 30% needs 291 per group.

Sources: [Kim HY (2016), Restorative Dentistry & Endodontics: sample size for two independent proportions](https://pmc.ncbi.nlm.nih.gov/articles/PMC4868880/) (read 2026-11-20)

## How it works

Type each email's sends and replies, and choose the confidence level: 95% is the usual one. Everything runs in your browser.

The calculator runs Fisher's exact test, two-sided: with the totals fixed, it adds up the probability of every split of the replies no more likely than yours. That's the chance of a gap this big if the two emails were equally good, and at 5% or less it's significant at 95% confidence. Website A/B calculators use the pooled z-test, an approximation that needs at least 5 expected replies and non-replies in each variant; it's shown beside the exact test, with a warning when that fails or the two disagree.

The sends needed come from the standard formula for comparing two proportions at 80% power, which assumes the normal approximation, so treat it as a floor. Decide the number before you start and stop there: stopping at the first significant moment makes the reported significance meaningless. A significant result says the gap is unlikely to be chance, not that it's big.

## Know when a winner is a winner. Small numbers need the exact test.

Cold email tests are small: a few hundred sends and a handful of replies per variant. That's exactly where the usual calculator's shortcut stops working.

### Why Fisher's exact test

NIST's handbook calls Fisher's exact test an excellent technique when the two samples are small: it works out the probability of the table exactly, with the totals fixed. The z-test is the large-sample version, and Penn State's notes give its condition: all four expected cells of the 2×2 table at least 5. A test with 200 sends a variant and a 2% reply rate expects 4 replies in each, under it.

Sources: [NIST/SEMATECH e-Handbook of Statistical Methods §7.3.3: comparing two proportions](https://www.itl.nist.gov/div898/handbook/prc/section3/prc33.htm) (read 2026-11-20), [Penn State STAT 500 §8.2: the 2×2 table, a test of two independent proportions](https://online.stat.psu.edu/stat500/Lesson08#the-2x2-table-test-of-2-independent-proportions) (read 2026-11-20)

### Fix the sample size first

Evan Miller's well-known warning: significance assumes the sample size was fixed in advance. Run a test until you see a significant difference and the reported significance levels become meaningless; peek ten times and what looks like 1% significance is really 5%. Work out the sends per variant first, with the formula in the table above, and read the result once.

Sources: [Evan Miller: How Not To Run an A/B Test (2010)](https://www.evanmiller.org/how-not-to-run-an-ab-test.html) (read 2026-11-20), [Kim HY (2016), Restorative Dentistry & Endodontics: sample size for two independent proportions](https://pmc.ncbi.nlm.nih.gov/articles/PMC4868880/) (read 2026-11-20)

### What significant means

The American Statistical Association's 2016 principles on p-values: a p-value doesn't measure the size of an effect or the importance of a result, and business decisions shouldn't rest only on whether it passes a threshold. A real one-point lift can still be too small to matter.

Tests tune the words. Who gets the email moves reply rates the most. [Valley](https://www.joinvalley.co/?utm_source=tools_suite)

Sources: [American Statistical Association: statement on p-values, press release (7 March 2016)](https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf) (read 2026-11-20), [Penn State STAT 200 §5.1: introduction to hypothesis testing](https://online.stat.psu.edu/stat200/Lesson05#introduction-to-hypothesis-testing) (read 2026-11-20)

## Sources

- [NIST/SEMATECH e-Handbook of Statistical Methods §7.3.3: comparing two proportions](https://www.itl.nist.gov/div898/handbook/prc/section3/prc33.htm) (read 2026-11-20)
- [R documentation: fisher.test, Fisher's exact test for count data](https://stat.ethz.ch/R-manual/R-devel/library/stats/html/fisher.test.html) (read 2026-11-20)
- [Penn State STAT 500 §8.2: the 2×2 table, a test of two independent proportions](https://online.stat.psu.edu/stat500/Lesson08#the-2x2-table-test-of-2-independent-proportions) (read 2026-11-20)
- [Kim HY (2016), Restorative Dentistry & Endodontics: sample size for two independent proportions](https://pmc.ncbi.nlm.nih.gov/articles/PMC4868880/) (read 2026-11-20)
- [Penn State STAT 200 §5.1: introduction to hypothesis testing](https://online.stat.psu.edu/stat200/Lesson05#introduction-to-hypothesis-testing) (read 2026-11-20)
- [Evan Miller: How Not To Run an A/B Test (2010)](https://www.evanmiller.org/how-not-to-run-an-ab-test.html) (read 2026-11-20)
- [American Statistical Association: statement on p-values, press release (7 March 2016)](https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf) (read 2026-11-20)

## Questions people ask

### How do I know if my A/B test is statistically significant?

Type each variant's sends and replies. If the chance of a gap that big, with equally good emails, is under 5%, it's significant at 95% confidence.

### Which test should I use for low reply rates?

Fisher's exact test. The z-test most calculators use needs at least 5 expected replies and non-replies per variant; small cold email tests often have fewer.

### How many emails do I need for an A/B test?

About 1,502 per variant to tell a 3% reply rate from 5% at 95% confidence and 80% power. Smaller gaps take more.

### What does significant at 95% confidence mean?

That a gap as big as yours would happen 5% of the time or less if the two emails were equally good: a p-value at or under 0.05.

### Can I stop the test as soon as it's significant?

No. Checking as you go and stopping at the first significant result inflates false winners. Decide the sends first.

### What is a p-value?

The probability of a result as extreme as yours or more, if there were no real difference. It doesn't measure how big the difference is.

### Is this A/B test calculator free?

Yes. It's free, needs no signup and runs in your browser.

## Related tools

- [Cold email calculator](https://tools.joinvalley.co/cold-email-calculator): Works back from the meetings you want to the people to contact and emails to send.
- [Email spam checker](https://tools.joinvalley.co/email-spam-checker): Scores a cold email out of 100 and flags links, money signs, em dashes and merge tags.
- [Spintax checker](https://tools.joinvalley.co/spintax-checker): Checks spintax, counts every version and converts it between six cold email tools.
- [Readability checker](https://tools.joinvalley.co/readability-checker): Scores any text's Flesch Reading Ease and grade level, and marks the long sentences.
- [Buyer persona builder](https://tools.joinvalley.co/buyer-persona-builder): A B2B buyer persona for each role in the deal, side by side, to copy, print or share.

## Next step

Want replies, not just clean copy? Valley finds buyers showing real buying signals and messages them across email and LinkedIn. [Valley](https://www.joinvalley.co/?utm_source=tools_suite)

Shelf: [03 Messaging](https://tools.joinvalley.co/messaging-and-personalization)

---

Tools by Valley, made by [Valley](https://www.joinvalley.co/?utm_source=tools_suite). HTML version: https://tools.joinvalley.co/ab-test-calculator
