A hypothesis test is a decision procedure, not a single formula. You state a null hypothesis, choose a test that matches your data, compute a statistic, compare it to a threshold, and end with one of exactly two verdicts: reject the null, or fail to reject it. Most pages online do one link in that chain and leave you to assemble the rest. This hypothesis testing calculator runs the whole chain, writes your hypotheses out in words so you can check they say what you meant, and reports the decision alongside the numbers that produced it.
Arb Digital built it because the assembly step is where the errors happen. In our own conversion-rate and campaign work the arithmetic is rarely the problem — people can compute a t statistic. The failures are choosing a one-tailed test after peeking at the direction of the effect, reading a large p-value as proof that two variants perform identically, or running a z test on a sample of nine. Those are workflow errors, so the fix has to be a workflow.
What This Hypothesis Testing Calculator Does
It supports four of the tests that cover most first-year and applied work: a one-sample t test when you are comparing a sample mean against a fixed target and estimating the spread from the data; a one-sample z test for the rarer case where the population standard deviation is genuinely known; a two-sample t test assuming equal variances, for comparing two group means; and a one-proportion z test for rates and percentages. For each one it reports the standard error, the test statistic, the degrees of freedom, the critical value at your chosen alpha, the p-value, and the reject or fail-to-reject decision in a full sentence.
This is deliberately a different job from two adjacent pages. Our t-test calculator computes a t statistic for a given set of sample inputs, and our p-value calculator converts a statistic you already hold into a tail probability; this page is the end-to-end decision workflow that picks the test, states the hypotheses, and delivers the verdict rather than performing one step of it. If you only need one of those steps, the dedicated page is faster.
How to Use It
- Choose the test that matches your data. One mean against a target, two means against each other, or a proportion against a target rate. If you estimated the spread from the sample itself, the answer is a t test.
- Set the direction before you look at the result. Two-tailed asks whether there is any difference. One-tailed asks whether the effect goes in one specific direction, and commits you to ignoring a large effect the other way.
- Enter your summary statistics. Sample mean, hypothesised value, standard deviation and sample size. For a proportion test, enter the observed rate as a decimal and leave the standard deviation field alone.
- Pick alpha. This is the false-positive rate you accept in advance, not something to adjust after seeing the p-value.
- Read the plain-English box first, then the numbers. If the written hypotheses do not describe the question you actually care about, the p-value below them is answering something else.
The Formulas and How It's Calculated
Every test on this page has the same shape: a difference divided by the standard error of that difference. For a one-sample test the statistic is (x̄ − μ₀) ÷ (s ÷ √n), with n − 1 degrees of freedom when s comes from the sample. For two samples with equal variances the calculator pools the two standard deviations, s²ₚ = [(n₁−1)s₁² + (n₂−1)s₂²] ÷ (n₁+n₂−2), and divides the difference in means by sₚ√(1/n₁ + 1/n₂), on n₁ + n₂ − 2 degrees of freedom. For a proportion the standard error uses the hypothesised rate, √[p₀(1−p₀)/n], because under the null that is the true rate.
Work through the default values, which come from a wafer-inspection example in the NIST handbook. A sample of 10 has a mean of 53.7 against an assumed process mean of 50, with a sample standard deviation of 6.567. The standard error is 6.567 ÷ √10 = 2.0767. The statistic is (53.7 − 50) ÷ 2.0767 = 1.7819 on 9 degrees of freedom. The two-tailed critical value at alpha 0.05 is 2.2622, and 1.7819 does not reach it, so the test fails to reject the null. The p-value is 0.1086 — the probability of seeing a difference at least this large in either direction if the process mean really were 50. You can follow the same example through the NIST/SEMATECH e-Handbook section on testing an assumed process mean.
The p-value itself comes from the tail area of the relevant distribution. For z tests the calculator uses the normal cumulative distribution; for t tests it evaluates the regularised incomplete beta function, which is the standard way of getting exact Student-t tail probabilities without a printed table. Critical values are found by inverting the same functions numerically, so the threshold you see is the exact one for your degrees of freedom rather than the nearest row of a textbook table.
Failing to Reject Is Not Proof of the Null
This is the single most important sentence on the page, so it is worth stating without hedging: a large p-value is not evidence that the null hypothesis is true. The test is built to control one error only — the chance of declaring an effect that is not there. It says nothing symmetric about the chance of missing an effect that is there. A study with eight observations and a noisy measurement will fail to reject almost any null you hand it, including nulls that are badly wrong.
That is why the calculator words the verdict as "fail to reject" rather than "accept". The correct reading of a non-significant result is: this sample did not contain enough evidence to rule out the null at the level of proof I demanded. Whether that is because there is no effect, or because the sample was too small to see one, is a question the p-value cannot answer. To distinguish the two you need the power of the test and the width of the confidence interval, which is why our statistical power calculator and a confidence interval belong alongside this page rather than after it.
One Tail or Two, and Why the Choice Must Come First
A one-tailed test puts your whole alpha budget in one direction, so the critical value is closer and the same data is more likely to reach significance. That is a legitimate gain when the direction is genuinely fixed in advance — a manufacturing tolerance where only excess contamination matters, or a change that can only be an improvement by construction. It is not legitimate when you looked at the data, saw which way the difference went, and then chose the tail that suits it.
The reason is arithmetic rather than etiquette. If you are willing to run either one-tailed test depending on what you see, your real false-positive rate is the sum of both tails: alpha in each direction, so 10% when you thought you were working at 5%. The switch is invisible in the write-up and doubles the error rate you claimed to control. Default to two-tailed unless you can say, before seeing the numbers, what you would do if the effect went the other way.
Significance and Size Are Different Questions
A p-value answers one narrow question: how surprising is this data if the null is true. It does not tell you how big the effect is, and it is strongly affected by sample size. Push n high enough and a difference of no practical consequence will clear any alpha you like. Keep n small and a large, genuinely important difference will sail past unnoticed.
The habit that fixes this is to report the difference and its confidence interval next to the p-value, always. A result of "0.4 percentage points higher, 95% interval −0.1 to 0.9, p = 0.11" is informative. "Not significant" on its own is not, because it hides whether the plausible range was tight around zero or wide enough to contain an effect worth chasing. When you are deciding whether a change is worth shipping, the interval is the number you act on, and the standard error behind it is available from our standard error calculator.
Assumptions That Actually Bite
The t test assumes the sample mean is approximately normally distributed. With a symmetric population that holds at almost any sample size; with a strongly skewed one it takes more data, and the usual rule of thumb of thirty observations is a rough guide, not a guarantee. The test also assumes independent observations, and this is the assumption that fails most often in practice: repeated measurements on the same subject, sessions from the same user, or items produced in the same batch are not independent, and treating them as if they were understates the standard error and inflates significance.
Outliers deserve a separate check. A single extreme value moves the mean and inflates the standard deviation at the same time, and the two effects do not cancel — usually the statistic shrinks and a real effect is buried. Screen the data for extreme values before testing rather than after, and decide what to do with any flagged points on the basis of what they are, not on whether keeping them helps your result. The NIST/SEMATECH e-Handbook introduction to statistical tests sets out the same assumption checklist for process comparisons.
Multiple Tests Multiply the False Positives
One test at alpha 0.05 has a one-in-twenty chance of a false positive. Twenty independent tests have a 64% chance that at least one of them comes back significant when nothing is going on. This is not a subtlety; it is the dominant risk in any analysis that slices a dataset several ways, tests several metrics, or checks a running experiment repeatedly.
There are two honest responses. Decide the single primary comparison before you collect data and treat everything else as exploratory, or correct the threshold — dividing alpha by the number of tests is crude but safe. What is not acceptable is running many comparisons, reporting the one that cleared 0.05, and describing it as a 5% risk. It never was.
Arb Digital's analysts set the hypothesis, the sample size and the stopping rule before a test starts — so the result at the end means what it appears to mean.
Browse All Free Tools Talk To Our TeamCommon Mistakes to Avoid
- Reading a large p-value as proof of no effect — it means the evidence was insufficient, which is a different claim and often just a sample-size problem.
- Choosing the tail after seeing the data — this quietly doubles the false-positive rate you claimed to be working at.
- Using a z test with an estimated standard deviation — if the spread came from your sample, the extra uncertainty needs the wider t distribution, especially below about thirty observations.
- Stopping the moment the p-value dips under 0.05 — repeatedly checking a running test and halting on success will reach significance eventually on pure noise.
- Reporting significance without the effect size — a tiny difference measured on a huge sample is significant and often useless; the interval tells you which case you are in.
Related Free Tools From Arb Digital
Find the threshold for any alpha and degrees of freedom with the critical value calculator, standardise a single observation with the z-score calculator, or size the study in advance with the statistical power calculator. The full free online tools hub lists every statistics tool we publish.
Frequently Asked Questions
It is the probability of getting a result at least as extreme as the one you observed, assuming the null hypothesis is true. It is not the probability that the null is true, and it is not the probability that your finding is a fluke.
No. Failing to reject means the sample did not provide enough evidence to rule the null out at your chosen level. A small or noisy sample fails to reject almost any null, including ones that are clearly wrong.
Whenever the standard deviation is estimated from your own sample, which is nearly always. The z test assumes the population standard deviation is known in advance, and using it otherwise understates the uncertainty.
Decide before you see the data. Use one-tailed only when a difference in the opposite direction would lead to the same action as no difference at all. Otherwise use two-tailed.
Alpha is the false-positive rate you set in advance as your standard of proof. The p-value is what the data produced. You compare the second against the first, and you fix alpha before looking.
They control how heavy the tails of the t distribution are. With few observations the tails are fatter, so the critical value is further out and a larger statistic is needed to reach the same significance.
Yes, choose the one-proportion z test and enter the observed rate as a decimal. The standard error under the null uses the hypothesised proportion rather than the observed one.
The pooled two-sample test on this page assumes similar variances. If one group is far more variable than the other, or the group sizes are very unequal, a Welch test with adjusted degrees of freedom is the safer choice.
This page explains a statistical procedure for study and for checking your own working. It is not a substitute for showing your method, and it is not medical, legal, or financial advice.
