1 A Motivating Example
A school district identifies the 50 worst-performing schools on this year's standardised test and enrols them in an expensive remediation programme. Next year, almost all of them improve. The superintendent declares victory. Should we believe the programme worked? Not yet.
Even if the programme did nothing at all, the worst-scoring schools would tend to score higher the following year simply because some of their low scores were bad luck (a flu outbreak on test day, an unlucky cohort, a mismarked batch) that does not repeat. This is regression to the mean, and it is one of the most pervasive and under-recognised threats to causal claims. It has fooled doctors evaluating treatments, coaches evaluating training, and governments evaluating policies. The phenomenon was first described by Francis Galton in the 1880s, who noticed that tall fathers tend to have sons shorter than themselves (and short fathers, taller sons) heights "regressed" toward the population average [Stigler, 1997].
2 The Fundamental Mechanism
Regression to the mean arises whenever a measured outcome is the sum of a stable component and a transitory, random component. Write a unit's score in period t as
where μᵢ is the unit's stable "true" level and ϵᵢₜ is mean-zero noise, independent across periods, with variance σ_ϵ² Now suppose we select units because they had extreme values in period 1 say, the lowest scores. A low observed Yᵢ₁ can happen two ways: the unit truly has a low μᵢ, or it had an unusually negative shock ϵᵢ₁ Selection on a low Yᵢ₁ therefore picks up units with genuinely low μᵢ and units that merely had bad luck. In period 2 the luck does not persist: the expected shock returns to zero. So the selected group's average moves back toward the population mean, with no causal force at work.
The strength of the effect is governed by the reliability of the measure. If Y is highly reliable (little noise), there is little regression to the mean; if it is noisy, the bounce-back is large. Quantitatively, for a group selected on period-1 scores, the expected period-2 score moves a fraction toward the grand mean equal to one minus the correlation between the two periods:
where ρ=Corr(Yᵢ₁,Yᵢ₂). If ρ=0.7, a group selected because it scored 10 points below average will, on average, score 7 points below average next period a 3-point "improvement" that is pure statistical artefact.
3 A Graphical Picture
Figure 1 shows the mechanism. Units selected for being far below average in period 1 (left) drift upward toward the mean in period 2 (right), while units selected for being far above drift downward. Nothing happened to them; the extremes were partly noise that did not recur.
Figure 1: Regression to the mean. Groups selected for extreme period-1 scores move toward the population mean in period 2 even with no treatment, because part of the extremity was transitory noise.
4 Where It Bites in Causal Research
Pre-post comparisons. Any evaluation that selects units because they were doing badly, then measures whether they improved, is contaminated. Job-training programmes enrol workers after a spell of low earnings; remedial classes enrol students after low scores; clinics treat patients when symptoms peak. All will show "improvement" from regression to the mean alone.
The Ashenfelter dip. In the training-evaluation literature, participants' earnings typically dip just before enrolment (often the dip is why they enrolled) and recover afterward regardless of the programme [Ashenfelter, 1978]. A naive before-after estimate credits the programme with a recovery that would have happened anyway. The dip is regression to the mean operating on earnings.
5 How to Defend Against It
The remedy is the same one that defends against most threats to causal inference: a comparison group that was selected the same way but did not receive treatment.
- Randomisation. If, among the low-scoring schools, we randomly assign half to the programme and half to a control, both groups regress to the mean equally. The difference in their period-2 outcomes nets out the regression artefact and isolates the treatment effect.
- Difference-in-differences. With a comparison group selected on the same criterion, the regression-to-the-mean component is common to both groups and is differenced away [Angrist and Pischke, 2009].
- Avoid selecting on the noisy baseline. Where possible, select treatment units on a variable measured independently of the outcome's transitory noise, or use a multi-period average to reduce σ_ϵ²
6 Common Mistakes and Where to Learn More
The cardinal error is interpreting a single group's before-after change as a treatment effect when the group was selected for being extreme. A related subtlety is Lord's paradox: adjusting for a noisy baseline by regression does not fully remove the bias, because the baseline itself contains the noise that drives the regression to the mean. The only reliable fix is a control group subjected to the same selection.
Galton's insight is nearly 150 years old, yet it remains a leading reason that uncontrolled evaluations overstate or invent effects. Whenever you read that "the worst performers improved after the intervention," ask the Galtonian question first: compared to what? For the history and statistics, see Stigler [1997]; for its role in programme evaluation, Angrist and Pischke [2009].
References
- Angrist, J. D., and Pischke, J.-S. (2009). Mostly Harmless Econometrics: An Empiricist's Companion. Princeton University Press.
- Ashenfelter, O. (1978). Estimating the effect of training programs on earnings. Review of Economics and Statistics, 60(1), 47-57.
- Stigler, S. M. (1997). Regression towards the mean, historically considered. Statistical Methods in Medical Research, 6(2), 103-114.