What a One-Way ANOVA Actually Tests
One-way ANOVA answers a single question: could all of these groups plausibly share the same population mean? The NIST/SEMATECH e-Handbook of Statistical Methods states the null hypothesis as "there is no difference in the population means of the different levels of factor A," with the alternative that the means are not all the same (NIST, section 7.4.3). Notice what that leaves out. It never says which mean is different, or by how much.
The name is a little misleading, because the test compares means by analyzing variances. It splits the total variation in your data into two piles: the part explained by group membership and the part left over inside the groups. When the first pile is large compared with the second, group membership is doing real work. That is the whole idea, and everything in the table above is bookkeeping around it.
Why Not Just Run Several T-Tests?
Because every test you add is another lottery ticket for a false positive. Three groups give three pairwise comparisons. If each is run at α = 0.05 and we pretend the tests are independent, the chance of at least one false alarm is 1 − 0.95³, about 14%.
Ten groups give 45 pairs, and the same arithmetic gives 1 − 0.95⁴⁵, above 90%. The pairs share data, so the real figures differ a little, but the direction is not in doubt.
ANOVA sidesteps the problem by asking one question once. Only if that omnibus test comes back significant does it make sense to dig into individual pairs, and then with a procedure built to hold the overall error rate down.
How the F-Statistic Is Built
With k groups and N observations in total, the calculator computes three sums of squares and the degrees of freedom that go with them:
- SSB = Σ nᵢ(x̄ᵢ − x̄)², with df₁ = k − 1
- SSW = ΣΣ (x − x̄ᵢ)², with df₂ = N − k
- SST = SSB + SSW, with N − 1 degrees of freedom
- MSB = SSB / df₁, MSW = SSW / df₂, F = MSB / MSW
MSW is an average of the variances inside the groups, so it measures pure scatter. MSB measures how far the group means sit from the grand mean, scaled by group size.
If the null hypothesis is true, both estimate the same quantity and F lands near 1. The p-value is the upper-tail area of the F-distribution with (df₁, df₂) degrees of freedom, the same distribution that powers the F-distribution table.
A Worked Example, Start to Finish
The calculator loads with three groups of ten scores each, so k = 3 and N = 30. Here is what comes out, in order.
The group means are 83.4, 89.3 and 84.7. The grand mean is 85.8.
Each group contributes n(x̄ᵢ − x̄)² to SSB: 57.6, 122.5 and 12.1. Together they give SSB = 192.2.
Inside the groups, the squared deviations add up to 640.4, 208.1 and 252.1. So SSW = 1100.6, and SST = 1292.8.
The degrees of freedom are 2 and 27. MSB = 192.2 / 2 = 96.1, and MSW = 1100.6 / 27 = 40.763.
Then F = 96.1 / 40.763 = 2.3575, with p = 0.1138. The 5% critical value for F(2, 27) is 3.354, and 2.3575 falls short of it.
The honest reading is that these data do not show a reliable difference between the three means. Group 2 averages about six points above group 1, which looks like something, but group 1 alone runs from 71 to 98, and ten observations per group is not enough to separate a six-point gap from scatter that wide.
Eta squared is 0.1487, a large effect by the usual benchmarks, yet it is not significant. Both facts are true at once, and that is why you should report the effect size next to the p-value rather than instead of it.
Assumptions Worth Checking
- Independent observations. Each value should come from a different subject, and one group's values should not influence another's. Repeated measurements on the same people need a repeated-measures design, which this tool does not cover.
- Roughly normal data within each group. The NIST handbook describes ANOVA as applied under the assumption that the sampled populations are normally distributed. With a handful of values per group you cannot check this well, so lean on a plot and on common sense about how the measurement behaves.
- Similar spread across groups. The classic F-test pools the within-group variances. Look at the standard deviations in the group table: if one is several times larger than another and the group sizes differ, treat a borderline p-value with suspicion.
Eta Squared: How Big Is the Difference?
Eta squared is SSB / SST, the share of total variation that sits between the groups. It runs from 0 to 1 and it is easy to explain to a non-statistician: 0.15 means about 15% of the variation in scores goes with group membership. The common rule-of-thumb benchmarks, 0.01 small, 0.06 medium and 0.14 large, come from Cohen. Treat them as a rough yardstick, not a verdict, because what counts as a meaningful effect depends on the field. Eta squared also runs a little high for small samples, since it describes the sample rather than the population.
After a Significant Result
A small p-value says some mean is different, nothing more. The usual next step is a post hoc test such as Tukey's HSD, which compares every pair of groups while controlling the overall error rate. Plan it before you look at the data if you can.
If you only ever care about one specific contrast, such as treatment versus control, a planned comparison is both simpler and more powerful. To size a study so that the F-test has a fair chance of catching the difference you care about, start with the power analysis calculator.
Frequently Asked Questions
What does a one-way ANOVA test?
It tests whether three or more group means could all come from populations with the same mean. The null hypothesis says they do; the alternative says at least one mean is different. The test compares how far the group means spread out against how much the individual values spread out inside each group. If the first dwarfs the second, the F-statistic gets large and the p-value gets small.
Why not just run several t-tests?
Every extra test is another chance to see a fluke. With three groups you would run three pairwise t-tests, and if each uses α = 0.05 and they were independent, the chance of at least one false positive climbs to about 14%. With ten groups there are 45 pairs, and that chance passes 90%. ANOVA asks one question with one α instead.
How do I read the F-statistic?
F is the mean square between groups divided by the mean square within groups. An F near 1 means the group means differ about as much as you would expect from noise alone. An F well above 1 means the means are spread out more than the within-group scatter can explain. Compare the p-value with your α, or look up the critical value in the F-distribution table.
My ANOVA is significant. Which groups differ?
The ANOVA cannot tell you. A significant F only says that at least one mean is different. To find the pairs, run a post hoc procedure such as Tukey's HSD, which compares every pair while keeping the overall error rate under control. This calculator stops at the omnibus test.
Can I use ANOVA with only two groups?
Yes. It matches a pooled two-sample t-test: F equals t squared and the p-values are identical. In the example data, groups 1 and 2 give F = 3.692 and t = −1.9215, whose square is 3.692. Still, the t-test is the better pick here, since it also shows a signed difference.
What if my groups have different variances or sizes?
Equal group sizes make the classic F-test more forgiving about unequal variances. When sizes differ and the spread looks very different, Welch's ANOVA is the safer route. This calculator runs the classic equal-variance test, so check the group standard deviations before trusting a borderline result.