Skip to content

Tools is made by Valley.

See how ↗

Tools is made by Valley.

See how ↗

Tools is made by Valley.

See how ↗

Free tool · 03 / WRITE

A/B test calculator.Is the winning email really better? In seconds, no signup.

Says if one cold email really out-replies another, and how many sends a fair test needs.

Runs in your browser. Nothing you type is sent to us.

Email A

Email B

A's reply rate
1%
B's reply rate
4.5%
B against A
+350%
Chance of a gap this big
6.2%

B replied at 4.5% against A's 1%. If the two emails were equally good, a gap this big would still happen 6.2% of the time, so it isn't significant at 95% confidence yet.

A fair test of a gap this size needs about 339 sends per variant at 95% confidence and 80% power. Decide the number first and stop there, not when it first looks significant.

A website A/B calculator's z-test would give 3.2% and call a winner. Fisher's exact test doesn't rest on an approximation, so trust it here.

Planning a test? Sends needed per variant

1,502 sends per variant, 3,004 in all.

An example: two made-up emails, 200 sends each. Everything runs in your browser.

When the test is done, send the winner to the right people. Valley: 15-45% reply rates, against a category average of about 2%. See how Valley does it ↗

Built by Valley ↗

NEXT STEP

Fed up with tests that never find a winner.Valley reads up on every person first.

Book a demo ↗

$500 a month for your first seat, $200 for each seat after that.

Sends you need

How many emails do you need for an A/B test?Sends per variant, at 95% confidence and 80% power.

It depends on your reply rate and the gap you want to find. To tell a 3% reply rate from 5% at 95% confidence and 80% power takes about 1,502 sends per variant; smaller gaps and lower rates take more.

Sends needed per variant to detect each change in reply rate, at 95% confidence and 80% power
Reply rate nowReply rate to detectSends per variantSends in all
1%2%2,3134,626
2%4%1,1372,274
3%5%1,5023,004
5%8%1,0552,110
10%15%6831,366

n = (z₁₋α/₂ + z₁₋β)² × [p₁(1 − p₁) + p₂(1 − p₂)] ÷ (p₁ − p₂)², with z = 1.96 for 95% confidence and 0.84 for 80% power, rounded up. It reproduces the source's worked example: 20% against 30% needs 291 per group.

From Kim HY (2016), Restorative Dentistry & Endodontics: sample size for two independent proportions ↗, read 20 Nov 2026

How it works

How this tool works.And what it can't see.

Type each email's sends and replies, and choose the confidence level: 95% is the usual one. Everything runs in your browser.

The calculator runs Fisher's exact test, two-sided: with the totals fixed, it adds up the probability of every split of the replies no more likely than yours. That's the chance of a gap this big if the two emails were equally good, and at 5% or less it's significant at 95% confidence. Website A/B calculators use the pooled z-test, an approximation that needs at least 5 expected replies and non-replies in each variant; it's shown beside the exact test, with a warning when that fails or the two disagree.

The sends needed come from the standard formula for comparing two proportions at 80% power, which assumes the normal approximation, so treat it as a floor. Decide the number before you start and stop there: stopping at the first significant moment makes the reported significance meaningless. A significant result says the gap is unlikely to be chance, not that it's big.

Built by Valley.

Sources

Testing cold email

Know when a winner is a winner.Small numbers need the exact test.

Cold email tests are small: a few hundred sends and a handful of replies per variant. That's exactly where the usual calculator's shortcut stops working.

  1. 01

    Why Fisher's exact test

    NIST's handbook calls Fisher's exact test an excellent technique when the two samples are small: it works out the probability of the table exactly, with the totals fixed. The z-test is the large-sample version, and Penn State's notes give its condition: all four expected cells of the 2×2 table at least 5. A test with 200 sends a variant and a 2% reply rate expects 4 replies in each, under it.

    From NIST/SEMATECH e-Handbook of Statistical Methods §7.3.3: comparing two proportions ↗ · Penn State STAT 500 §8.2: the 2×2 table, a test of two independent proportions ↗, read 20 Nov 2026

  2. 02

    Fix the sample size first

    Evan Miller's well-known warning: significance assumes the sample size was fixed in advance. Run a test until you see a significant difference and the reported significance levels become meaningless; peek ten times and what looks like 1% significance is really 5%. Work out the sends per variant first, with the formula in the table above, and read the result once.

    From Evan Miller: How Not To Run an A/B Test (2010) ↗ · Kim HY (2016), Restorative Dentistry & Endodontics: sample size for two independent proportions ↗, read 20 Nov 2026

  3. 03

    What significant means

    The American Statistical Association's 2016 principles on p-values: a p-value doesn't measure the size of an effect or the importance of a result, and business decisions shouldn't rest only on whether it passes a threshold. A real one-point lift can still be too small to matter.

    Tests tune the words. Who gets the email moves reply rates the most. See how Valley does it ↗

    From American Statistical Association: statement on p-values, press release (7 March 2016) ↗ · Penn State STAT 200 §5.1: introduction to hypothesis testing ↗, read 20 Nov 2026

Questions

Questions people ask.

How do I know if my A/B test is statistically significant?
Type each variant's sends and replies. If the chance of a gap that big, with equally good emails, is under 5%, it's significant at 95% confidence.
Which test should I use for low reply rates?
Fisher's exact test. The z-test most calculators use needs at least 5 expected replies and non-replies per variant; small cold email tests often have fewer.
How many emails do I need for an A/B test?
About 1,502 per variant to tell a 3% reply rate from 5% at 95% confidence and 80% power. Smaller gaps take more.
What does significant at 95% confidence mean?
That a gap as big as yours would happen 5% of the time or less if the two emails were equally good: a p-value at or under 0.05.
Can I stop the test as soon as it's significant?
No. Checking as you go and stopping at the first significant result inflates false winners. Decide the sends first.
What is a p-value?
The probability of a result as extreme as yours or more, if there were no real difference. It doesn't measure how big the difference is.
Is this A/B test calculator free?
Yes. It's free, needs no signup and runs in your browser.