1 A Policy Experiment and Fifteen Outcomes
Imagine a study of a new job-training programme. The programme is randomised across 200 participants. Because the researchers are thorough, they measure 15 labour market outcomes: employment at 6 months, employment at 12 months, wages at 6 months, wages at 12 months, hours worked, industry of employment, whether the participant obtained a certification, and so on.
If the programme has no effect whatsoever on any outcome, what is the probability that at least one of the 15 tests will yield a p-value below 0.05? Each test has a 5% chance of a false rejection. Assuming independence across tests, the probability that at least one test rejects is:
More than a fifty-fifty chance of finding a "significant" effect even when the programme does nothing. If the researcher then reports the significant outcome without mentioning the 14 null outcomes, the published literature will contain a false positive.
This is the multiple testing problem, and it is ubiquitous in causal research: event studies with 20 pre- and post-treatment coefficients, subgroup analyses across demographic cells, multi-arm randomised trials, and specification searches all generate the same problem in slightly different forms.
2 Two Types of Error
The core framework distinguishes two types of error when testing a collection of hypotheses H₁, H₂, ..., Hₖ:
- Type I error (false positive): rejecting a null hypothesis that is actually true.
- Type II error (false negative): failing to reject a null hypothesis that is actually false.
The classical significance level α = 0.05 controls the Type I error rate for a single test. When running k tests simultaneously, the Type I error rate for the entire family of tests—the family-wise error rate (FWER)—can be much larger, as the calculation above illustrates.
Two different targets can be controlled:
- FWER: the probability of making at least one false rejection. This is the most conservative target, appropriate when any false discovery is very costly.
- FDR: the expected proportion of discoveries that are false. This is more lenient, appropriate when we are willing to tolerate some false positives as long as they are a small fraction of total discoveries.
3 Controlling FWER: Bonferroni and Holm
3.1 Bonferroni Correction
The simplest FWER correction is Bonferroni: reject Hⱼ if pⱼ < α/k where k is the number of hypotheses. This controls the FWER at level α regardless of the dependence structure among the tests.
Why it works: By the union bound, P(at least one false rejection) ≤ Σ P(reject Hⱼ | Hⱼ true) ≤ k · (α/k) = α.
The cost: Bonferroni is conservative. When tests are positively correlated (as is typical in causal studies—all outcomes may be affected by the same treatment), the actual FWER under Bonferroni is below α, meaning that the test loses power unnecessarily.
3.2 Holm's Stepdown Procedure
Holm [1979] proposed a stepdown correction that is uniformly more powerful than Bonferroni while still controlling FWER exactly.
Algorithm: Order the k p-values from smallest to largest: p₍₁₎ ≤ p₍₂₎ ≤ ... ≤ p₍ₖ₎.
- If p₍₁₎ ≥ α/k: fail to reject all hypotheses. Stop.
- Otherwise: reject H₍₁₎. If p₍₂₎ ≥ α/(k-1): stop.
- Continue: at step j, reject H₍ⱼ₎ if p₍ⱼ₎ < α/(k-j+1).
Holm’s procedure always rejects at least as many hypotheses as Bonferroni, and often many more when there are many true effects. It is uniformly preferred over Bonferroni in practice.
4 Controlling FDR: Benjamini-Hochberg
The Benjamini and Hochberg [1995] (BH) procedure controls the FDR at level q:
Algorithm: Order p-values from smallest to largest: p₍₁₎ ≤ ... ≤ p₍ₖ₎.
- Find the largest j such that p₍ⱼ₎ ≤ q · j/k.
- Reject all H₍₁₎, ..., H₍ⱼ₎.
Under independence of tests, BH controls FDR at exactly q. Under positive dependence (which is typical), it controls FDR at q · m₀/k ≤ q, where m₀ is the number of true nulls. Under general dependence, a modified procedure (Benjamini-Yekutieli) is needed.
Numerical example: With k = 15 tests, significance level q = 0.05, and ordered p-values [0.001, 0.006, 0.008, 0.018, 0.031, 0.060, 0.082, ...]:
The largest j with p₍ⱼ₎ ≤ 0.05j/15 is j = 3. We reject the three hypotheses corresponding to the smallest p-values.
5 Multiple Testing in Causal Research: Specific Contexts
5.1 Event Studies
An event study DiD estimates βₖ for each pre- and post-treatment period k. With 5 pre-periods and 10 post-periods, testing whether all pre-period coefficients are jointly zero (a pre-trend test) involves multiple comparisons. The standard approach is a joint F-test or χ² test, which correctly accounts for correlations among the βₖ estimates. Individual t-tests on each pre-period coefficient, corrected separately, are less powerful.
5.2 Subgroup Analysis
If a treatment effect is estimated for 10 demographic subgroups, any apparent heterogeneity must be interpreted with caution. The appropriate test is a joint test of treatment effect homogeneity across subgroups, not a comparison of which groups have significant effects and which do not. Pre-specifying the subgroups of interest before looking at the data eliminates most of the multiple testing problem in practice [Casey et al., 2012].
5.3 Multi-Arm Trials
When a randomised experiment has J treatment arms and one control group, J pairwise comparisons generate a family of tests. List et al. [2019] discuss the appropriate corrections and argue that the correct family depends on the research question: if interest lies in ranking treatments relative to control, FWER corrections apply; if interest lies in identifying any effective treatment, FDR corrections may be appropriate.
6 Practical Implementation in R
7 Pre-Specification as the Gold Standard
The best solution to the multiple testing problem is not a statistical correction but a research design choice: pre-specifying the hypotheses, outcomes, and analysis plan before observing the data. A pre-analysis plan (PAP) registered before data collection or analysis forces researchers to commit to their primary outcomes, subgroup analyses, and hypothesis tests [Casey et al., 2012, Olken, 2015].
Pre-registration shifts the researcher from a position of choosing which tests to report (which creates bias) to one where the test list is fixed (which does not). It also allows readers to distinguish confirmatory tests (specified in advance) from exploratory analyses (conducted post hoc), which should be treated as hypothesis-generating rather than hypothesis-confirming.
8 Common Mistakes
- Reporting only the significant outcomes: The most common form of multiple testing bias. Always report all outcomes tested, with their raw p-values and any corrections applied.
- Applying individual-test corrections when a joint test suffices: For pre-trend testing in event studies, a single joint F-test is more powerful than multiple individual t-tests with Bonferroni correction.
- Confusing p-value adjustment with effect size adjustment: Correcting for multiple comparisons changes the significance threshold for rejecting null hypotheses, not the point estimates themselves. A subgroup effect that is "not significant after Bonferroni" may still be a large and important effect.
- Over-correcting when outcomes are strongly correlated: Bonferroni assumes tests are independent. When 15 outcomes all measure labour market success and are strongly correlated, Bonferroni is far too conservative. Permutation-based corrections that account for the actual correlation structure are preferred.
9 Conclusion
Multiple testing is not a technical nuisance but a fundamental feature of empirical research design. Whenever a study examines multiple outcomes, subgroups, or time periods, the probability of false discoveries rises. Bonferroni, Holm, and Benjamini-Hochberg provide statistical tools to control this rate at the cost of reduced power on individual tests. Pre-registration provides a complementary research design approach that eliminates the problem at the source. Responsible empirical practice uses both.
References
- Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B, 57(1):289-300.
- Casey, K., Glennerster, R., and Miguel, E. (2012). Reshaping institutions: Evidence on aid impacts using a preanalysis plan. Quarterly Journal of Economics, 127(4):1755-1812.
- Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65-70.
- List, J. A., Shaikh, A. M., and Xu, Y. (2019). Multiple hypothesis testing in experimental economics. Experimental Economics, 22(4):773-793.
- Olken, B. A. (2015). Promises and perils of pre-analysis plans. Journal of Economic Perspectives, 29(3):61-80.
- Romano, J. P. and Wolf, M. (2005). Stepwise multiple testing as formalized data snooping. Econometrica, 73(4):1237-1282.