What Is a P-Value?
The single most common misunderstanding in applied statistics goes something like this: a p-value of 0.03 means there is a 97% chance my hypothesis is correct. It does not. A p-value of 0.03 actually means something narrower — if no real effect exists, you would see data this extreme only 3% of the time through sampling variability alone, which is a statement about the data under the null, not a statement about your theory being right. Ronald Fisher formalized this logic in his 1925 text Statistical Methods for Researchers, and the American Statistical Association's 2016 statement reaffirmed that the p-value measures evidence against the null hypothesis, never evidence for your alternative.
The clearest way to think about p-values is as surprise scores — the smaller the p-value, the more surprised you should be if nothing is actually happening. That reframing matters because it shifts the focus from a binary yes/no gate to a continuous measure of how unusual your data look. A growing number of statisticians have pushed this same point, arguing that dichotomous labels like "significant" obscure the continuous nature of statistical evidence. Major medical journals now require authors to present exact p-values alongside effect sizes and confidence intervals, moving the field away from binary cutoffs.
Significance Levels
| P-Value | Significance | Notation |
|---|---|---|
| p < 0.001 | Highly significant | *** |
| p < 0.01 | Very significant | ** |
| p < 0.05 | Significant | * |
| p < 0.1 | Marginally significant | . |
| p ≥ 0.1 | Not significant | ns |
Frequently Asked Questions
What does p < 0.05 mean?
A p-value below 0.05 means there is less than a 5% probability that sampling noise alone produced results this extreme, assuming no real effect. The uncomfortable truth is that the difference between p = 0.048 and p = 0.052 is statistically meaningless, yet one gets published and the other lands in the file drawer — which is exactly the problem the field is trying to fix. Most introductory courses still anchor instruction around 0.05, though the ASA's 2016 statement cautioned against treating any single threshold as a universal gate for scientific conclusions.
One-tailed vs two-tailed: which should I use?
Two-tailed tests check for effects in both directions and produce more conservative p-values, which is why they are far more common in published research. The rule is simple: pre-register your direction or default to two-tailed. One-tailed tests are only justified when your directional hypothesis was locked in before data collection — otherwise reviewers will flag the choice as post-hoc p-hacking, and they would be right to do so.
How do I calculate p-value in Excel?
For z-test: =2*(1-NORM.S.DIST(ABS(z),TRUE)). For t-test: =T.DIST.2T(ABS(t),df). For chi-square: =1-CHISQ.DIST(x,df,TRUE).