The statistical power calculator above handles the three-way trade-off at the heart of study design. Power, sample size and effect size are locked together with the significance level: fix any two and the third is determined. Choose which one you want solved, set the others, and the tool returns it along with the sample sizes that 80% and 90% power would require and a bar chart of how power responds to changing n.
Arb Digital runs this calculation before every conversion test, because the alternative is worse than useless. An underpowered test that returns no significant result tells you nothing — not that the change failed, only that the study was too small to see it either way. Knowing that in advance lets you either commit the traffic or abandon the test, instead of burning six weeks to learn nothing.
What This Statistical Power Calculator Does
Statistical power is the probability that a study detects an effect, given that the effect is genuinely there. A test with 80% power applied to a real effect of the size you specified will produce a statistically significant result 80 times out of 100 and miss it the other 20. That miss is a Type II error, and its probability is beta, so power is simply 1 − beta.
The tool covers three common designs. Two independent means is the classic A/B comparison of a continuous outcome such as order value or session duration. One mean against a fixed value tests whether a single group differs from a target or a historical benchmark. Two proportions covers conversion rates, click-through rates, defect rates and anything else counted as a share of a total; it converts your two percentages into Cohen's h internally, which is the standardised effect measure that behaves properly for proportions.
Two boundaries are worth stating plainly. Our sample size calculator sizes a survey from a target margin of error, which is a precision question rather than a detection question. Our minimum detectable effect calculator holds the sample size fixed and asks how large an effect would have to be to show up. This page is the one that solves for power itself, and it will solve for the other two as a convenience.
How to Use It
- Pick what you want solved. Planning a study from scratch? Solve for sample size. Already committed to a fixed traffic budget? Solve for power, or for the effect size you could detect.
- Choose the test type. Continuous outcome measured on two groups, one group against a target, or two rates. The rate option asks for percentages rather than an abstract effect size, which is far easier to reason about.
- Enter the effect you care about. This is the judgement call, and it should come from what would change your decision, not from what you hope to see. A 0.5% lift that is real but not worth implementing is not an effect worth powering for.
- Set alpha and tails. Alpha 0.05 and two tails is the default in most fields. Use one tail only when a change in the opposite direction would lead to exactly the same action as no change at all.
- Adjust the allocation ratio if your groups will be uneven. Unequal groups always need more total subjects than equal ones for the same power, and the tool shows you how much more.
The Formula / How It's Calculated
Everything runs through a normal approximation. Define the non-centrality parameter as the effect size divided by the standard error of the estimate in standardised units. For two independent groups that is ncp = d ÷ √(1/n₁ + 1/n₂), and for a one-sample test it is ncp = d × √n. Power is then Φ(ncp − z1−α/2) + Φ(−ncp − z1−α/2), where Φ is the standard normal cumulative distribution and the second term is the negligible probability of being significant in the wrong direction.
Work the default example. With d = 0.5, 64 per group, alpha 0.05 two-tailed: ncp = 0.5 ÷ √(1/64 + 1/64) = 0.5 ÷ √0.03125 = 2.8284. Subtract the critical value 1.9600 to get 0.8684, and Φ(0.8684) = 0.807. So 64 per group gives about 81% power to detect a medium effect — which is why 64 per group is the number so many textbooks quote.
Solving in reverse gives the sample-size formula n₁ = (1 + 1/k)(z1−α/2 + zpower)² ÷ d², with k the allocation ratio. For d = 0.5, 80% power and equal groups that is 2 × (1.9600 + 0.8416)² ÷ 0.25 = 62.8, rounded up to 63. Exact calculations based on the non-central t-distribution return 64, one higher, because they account for estimating the standard deviation from the data. The NIST/SEMATECH e-Handbook section on sample sizes required to detect a shift sets out the same formula and recommends exactly this kind of iteration when the standard deviation is unknown. For proportions the effect size is Cohen's h = 2·arcsin(√p₁) − 2·arcsin(√p₂), which stabilises the variance so the same machinery applies.
Choosing an Effect Size Without Guessing
The effect size is the input people fudge, and fudging it invalidates everything downstream. Cohen's conventional labels — 0.2 small, 0.5 medium, 0.8 large — were offered as a last resort for fields with no prior data, and Cohen himself warned against leaning on them. Using 0.5 because it sounds reasonable produces a sample size that is reasonable-sounding and arbitrary.
Three better routes exist. Use the smallest effect of practical interest: work out the lift that would actually justify shipping the change, given its cost, and power for that. Use pilot or historical data: if a similar previous test moved the metric by 8%, that is a defensible planning figure, though pilot estimates are noisy and tend to be optimistic. Or use a published estimate from your own field, discounted downward, since effects reported in the literature are inflated by the same publication filter that favours significant results.
What you must not do is run the study, observe the effect you got, and compute power from it afterwards. Observed power is a deterministic function of the p-value and adds no information at all: a non-significant result always yields low observed power, so the exercise cannot tell you anything you did not already know. If a study came back null and you want to know what it ruled out, compute a confidence interval instead with our confidence interval calculator.
Why Underpowered Studies Are Worse Than No Study
The obvious cost of low power is missing real effects. The less obvious cost is more damaging: among the results that do reach significance in an underpowered study, the effect sizes are systematically exaggerated.
The mechanism is simple. If your study can only cross the significance threshold when the sampled difference happens to land unusually large, then every significant result you get is, by construction, an overestimate. Run a test with 20% power on a true 5% lift and the significant outcomes will tend to report lifts of 15% or 20%, because nothing smaller would have survived the filter. The team implements the change expecting 20% and gets 5%, and nobody understands why the win did not materialise in the quarterly numbers.
There is a sign-error risk too. At very low power, a meaningful share of significant results can point the wrong way entirely — showing a decrease where the true effect is an increase. This is why chasing significance with a small sample is not a cheaper version of doing the study properly; it is a different activity with a different, worse output. Pair every result with the raw uncertainty from the standard error calculator before deciding what it means.
What Moves Power the Most
Four levers control power, and they are not equally useful. Effect size is the strongest: because it enters the formula squared, doubling the effect you are chasing cuts the required sample by a factor of four. You cannot change reality, but you can often design a stronger intervention rather than a marginal one.
Sample size is the reliable lever, subject to the same square-root economics that govern precision generally — four times the data for twice the sensitivity. Alpha trades one error against the other: moving from 0.05 to 0.10 raises power but doubles your false-positive rate, and it is rarely the right move. Variance reduction is the underused lever. Anything that shrinks the noise — a paired or within-subject design, tighter measurement, controlling for a strong covariate, stratifying the randomisation — raises the standardised effect size without recruiting anybody, and it is usually cheaper than any of the alternatives.
One structural point on allocation. Equal groups are the most efficient use of a fixed total sample. If you split 2:1 instead of 1:1, you need roughly 12% more subjects overall for the same power. That can still be the right choice — when one arm is far more expensive, or when you need a bigger control group for secondary analysis — but it is a cost, and the calculator prices it for you.
Sequential Testing and the Peeking Problem
A power calculation assumes you will collect the planned sample and analyse once. Checking the p-value daily and stopping the moment it dips below 0.05 breaks that assumption badly. Under repeated peeking the true false-positive rate climbs well above the nominal alpha, because you are giving random fluctuation many chances to cross the line.
If you genuinely need to stop early — for cost, for ethics, or because a change is doing harm — use a method built for it, such as group sequential designs with alpha-spending boundaries or a Bayesian approach with a pre-declared stopping rule. Both cost a little power relative to a single fixed analysis, and both keep the error rate honest. What does not work is running a fixed-sample power calculation and then peeking anyway. Whatever you decide, write the stopping rule down before the first observation arrives, and run the final test with the t-test calculator or the p-value calculator once the sample is complete.
Arb Digital builds free, no-signup calculators for statistics, finance, health and construction. Browse the full library, or get in touch if there is a calculation your team keeps doing by hand.
Browse All Free Tools Contact Arb DigitalCommon Mistakes to Avoid
- Computing power after the fact from the observed effect — observed power is a restatement of the p-value and carries no independent information.
- Picking an effect size because it produces an affordable sample — that reverses the logic and guarantees the study is powered for something you do not care about.
- Using one-tailed tests to shrink the sample — only justified when an effect in the opposite direction would lead to identical action, which is rare.
- Powering for the total sample when the formula wants per group — this tool takes n for group one, and applies the allocation ratio to get group two.
- Ignoring dropout and invalid records — recruit above the calculated figure so the analysable sample, not the recruited sample, meets the target.
Related Free Tools From Arb Digital
Size a survey around precision with the sample size calculator, find the smallest effect a fixed sample can see with the minimum detectable effect calculator, run the final comparison with the t-test calculator, and check a categorical outcome with the chi-square calculator. For the underlying uncertainty of a single estimate, use the standard error calculator. The full free online tools hub holds the rest of the statistics set, and NIST's overview of statistical tests and the two kinds of error is a good primer on where alpha and beta come from.
Frequently Asked Questions
Power is the probability that a study detects an effect of a specified size, given that the effect is real. A test with 80% power will find such an effect in 80 of 100 repetitions and miss it in the other 20. The miss is a Type II error, and its probability is beta, so power equals one minus beta.
It is a convention, not a law. It reflects an informal judgement that a false negative is about four times more tolerable than a false positive at alpha 0.05. Where a missed effect is costly — safety, medicine, an expensive irreversible change — 90% or 95% is the more defensible target.
Use the smallest difference that would actually change your decision, or an estimate from pilot data or comparable past studies. Cohen's small, medium and large labels are a last resort for fields with no prior information, and should not be used simply because they are convenient.
Not usefully. Power computed from the effect you observed is a direct function of the p-value, so a non-significant result always produces low observed power. If you want to know what a null study ruled out, report the confidence interval instead.
For Cohen's d of 0.5, 80% power, alpha 0.05 and a two-tailed two-sample test, the normal approximation gives about 63 per group and exact non-central t methods give 64. That is 128 subjects in total, split evenly between the two arms.
Yes. Equal allocation is the most efficient use of a fixed total sample. A 2:1 split needs roughly 12% more subjects overall to reach the same power, which can still be worth it if one arm is much more expensive to run.
Cohen's h is the difference between the arcsine square-root transforms of two proportions. The transform stabilises the variance, which is otherwise tied to the proportion itself, so the same power formula that works for means can be applied to rates.
Not without inflating the false-positive rate. Repeatedly checking and stopping at the first significant reading gives random variation many chances to cross the threshold. If early stopping matters, plan a group sequential design with alpha-spending boundaries before collecting any data.
This calculator is provided for education and planning only. Statistical output should be read alongside a plot of your data and, for consequential decisions, reviewed by someone qualified in your field.
