1 The Promise of Evidence Aggregation

Individual causal studies are expensive to produce and often imprecisely estimated. A DiD study of the minimum wage using counties in a single US state may generate a standard error too large to draw firm conclusions. A series of 50 such studies, each using a different state or time period, generates far more information—but how do we aggregate it?

Meta-analysis has a long tradition in medicine and psychology: combine estimates from multiple randomised trials using inverse-variance weighting to produce a more precise summary estimate. The logic is compelling. The practice is, in economics, deeply contested.

A recent working paper by de Chaisemartin and D'Haultfœuille [2024] directly addresses this question for the DiD and event-study literature, arguing that "harvesting" the hundreds of event-study coefficients published in leading economics journals is possible and informative but requires attention to how the estimates were produced, what estimands they target, and how publication bias affects the resulting pool of evidence.

This article examines the case for and against systematic meta-analysis of causal estimates from quasi-experimental studies.

2 The Case For: Information Is Being Wasted

2.1 Published Studies Contain More Information Than the Abstract

A typical event-study DiD paper reports an overall ATT (perhaps 1-2 sentences in the abstract) but produces a rich set of estimated coefficients: ATTs for each cohort, each post-period, and each demographic subgroup. These estimates are rarely systematically synthesised even within the paper, let alone across the literature.

The "harvesting" approach [de Chaisemartin and D'Haultfœuille, 2024] argues that this information should be aggregated. If 30 papers each study the effect of a policy on wages, and each produces estimates for multiple cohorts and periods, the underlying pool of coefficient-level evidence is far larger than 30. A systematic extraction and aggregation of this evidence—controlling for the characteristics of the underlying studies—could produce estimates with much smaller standard errors than any individual paper.

2.2 Within-Study Aggregation Is Already Standard

Within a single study, researchers routinely aggregate from group-time ATTs to overall ATTs using weighted averages. Callaway and Sant'Anna [2021] show how to aggregate ATT(g, t) estimates across cohorts and periods to produce a single summary treatment effect. This is accepted practice.

Between-study aggregation follows the same logic, but across different samples, time periods, and institutional settings. If the within-study aggregation is valid, why not the between-study version?

2.3 More Variation, Better External Validity

A single study identifying from variation in, say, a state-level minimum wage policy estimates a LATE for the specific states, firms, and workers involved. Aggregating across many minimum wage studies—different states, years, labour market conditions—provides information about the average effect across a wider population and a broader range of institutional settings. This is precisely the external validity argument for meta-analysis.

3 The Case Against: The LATEs Are Different

3.1 Different Instruments, Different Compliers

The most fundamental objection to meta-analysing causal estimates is that different studies estimate different parameters. An IV estimate of the return to schooling using college proximity as an instrument identifies a LATE for workers who are marginal in terms of college attendance relative to distance. An IV estimate using quarter-of-birth as an instrument identifies a LATE for workers who are marginal in terms of compulsory schooling. These parameters are defined over different populations and are not directly comparable.

Combining them in a meta-analysis produces a weighted average of different LATEs, which may not correspond to any well-defined causal parameter. The weights depend on sample sizes and standard errors, not on the policy relevance of the respective complier populations.

3.2 Between-Study Heterogeneity Cannot Be Fully Explained

In medicine, meta-analyses use random-effects models to accommodate between-study heterogeneity in treatment effects. But in economics, the sources of between-study heterogeneity are often structural: different minimum wage levels, different labour market tightness, different firm size distributions, different product markets. A random-effects model attributes this heterogeneity to a random component without modelling its sources, producing credible intervals that are wide and difficult to interpret.

When between-study heterogeneity is large and structured, the meta-analytic summary is potentially misleading: a minimum wage study from a tight labour market and a minimum wage study from a depressed labour market are not draws from the same distribution of effects.

3.3 Publication Bias Infects the Pool of Evidence

Perhaps the most serious objection is publication bias. The studies available for meta-analysis are not a random sample of the true distribution of treatment effects. Studies that find statistically significant effects (whether positive or negative) are more likely to be published than null results. The pool of published DiD studies is therefore biased toward larger effects.

Andrews and Kasy [2019] showed formally that if publication occurs only when the t-statistic exceeds a threshold, the distribution of published estimates is truncated and the meta-analytic mean is upward-biased. Publication bias corrections ("trim-and-fill", Egger tests, selection models) exist but require assumptions about the publication process that cannot be verified.

3.4 Extracting Event-Study Coefficients Requires Heroic Assumptions

Even setting aside the conceptual issues, the practical challenge of extracting comparable estimates from published event-study DiD papers is formidable. Papers differ in their reference periods (some use k = −1, others use k = {−2, −1} as the reference group), their sample periods, their control variables, their treatment definitions, and whether they use TWFE or a heterogeneity-robust estimator. Extracting a single "ATT at k = 2" estimate from papers with these differences and treating them as comparable observations in a meta-analysis requires assumptions about comparability that are rarely tested.

4 Where Does This Leave Us?

The debate highlights a genuine tension between two legitimate goals: learning from the accumulated stock of causal evidence versus respecting the heterogeneity and context-dependence of individual estimates.

A defensible middle ground involves the following principles:

• Within-study aggregation is generally valid. Aggregating from group-time ATTs to overall ATTs within a single well-specified study is sound, provided the aggregation weights are policy-relevant.
• Cross-study aggregation requires a structural framework. If the goal is to extrapolate from the existing evidence to a new policy context, the extrapolation requires a model of how the treatment effect varies with context. A structural model of the minimum wage market can predict what a wage elasticity found in New Jersey in 1994 implies for California in 2026—but the prediction rests on the model, not on the raw average of published estimates.
• Systematic reviews are valuable even without meta-analysis. A systematic review that catalogues the existing evidence, describes the range of estimates, and identifies the conditions under which effects are large or small provides useful information without requiring numerical aggregation. Vote-counting across studies ("most studies find negative employment effects") is informative about the direction of effects even when a precise pooled estimate is not credible.
• Publication bias must be addressed explicitly. Any meta-analysis of the economics literature must account for publication bias. Models that explicitly incorporate the selection process for publication—as in Andrews and Kasy [2019]—are more credible than those that ignore it.

5 Conclusion

The question of whether and how to aggregate causal estimates from quasi-experimental studies does not have a clean answer. The "harvesting" programme represents a serious attempt to extract more value from the existing literature, but it faces fundamental challenges from heterogeneous LATEs, between-study heterogeneity, and publication bias. The most defensible synthesis: aggregate within studies, interpret between-study patterns as indicative rather than definitive, and always model the sources of heterogeneity rather than averaging it away.

References

1. Andrews, I. and Kasy, M. (2019). Identification of and correction for publication bias. American Economic Review, 109(8):2766-2794.
2. Callaway, B. and Sant'Anna, P. H. (2021). Difference-in-differences with multiple time periods. Journal of Econometrics, 225(2):200-230.
3. de Chaisemartin, C. and D'Haultfœuille, X. (2024). Credible answers to hard questions: Differences-in-differences for natural experiments. NBER Working Paper, w34550.
4. Goodman-Bacon, A. (2021). Difference-in-differences with variation in treatment timing. Journal of Econometrics, 225(2):254-277.
5. Imbens, G. W. (2010). Better LATE than nothing: Some comments on Deaton (2009) and Heckman and Urzua (2009). Journal of Economic Literature, 48(2):399-423.
6. Brodeur, A., Le, M., Sangnier, M., and Zylberberg, Y. (2016). Star wars: The empirics strike back. American Economic Journal: Applied Economics, 8(1):1-32.
7. Card, D. and Krueger, A. B. (1995). Myth or Measurement: The New Economics of the Minimum Wage. Princeton University Press.
8. Roth, J., Sant'Anna, P. H., Bilinski, A., and Poe, J. (2023). What's trending in difference-in-differences? A synthesis of the recent econometrics literature. Journal of Econometrics, 235(2):2218-2244.