1 Motivating Example

You have just estimated the average treatment effect of a job training programme using propensity score matching on a sample of 500 treated and 500 control workers. Your point estimate is a 12% increase in earnings. Now you need a confidence interval. The analytical standard error formula for matching estimators is complex, depends on unknown quantities, and assumes a large sample with well-behaved propensity scores. A colleague suggests using the bootstrap. Is that appropriate, and how does it work?

2 The Core Idea: Let the Data Speak for Itself

The bootstrap, introduced by Efron [1979], is a resampling method for approximating the sampling distribution of a statistic using only the observed data. The key insight is conceptually simple:

• The sampling distribution of an estimator θ̂ describes how θ̂ varies across all possible samples of size n drawn from the population.
• We only have one sample, so we cannot directly observe this distribution.
• But we can approximate it by treating our observed sample as if it were the population, and repeatedly resampling from it with replacement.

Each bootstrap resample ℬᵇ (b = 1, ..., B) is drawn by sampling n observations from the original data with replacement. For each resample, we compute the estimate θ̂ᵇ. The distribution of {θ̂ᵇ}ᵦ₌₁ᴮ across B bootstrap resamples approximates the true sampling distribution of θ̂.

The bootstrap standard error is:

where θ̄* = (1 / B) Σᵦ θ̂ᵇ is the mean across bootstrap resamples.

3 The Paired Bootstrap

The paired bootstrap (also called the nonparametric bootstrap) resamples observations (Yᵢ, Dᵢ, Xᵢ) as units. This is the most common form and is applicable to virtually any estimator. Algorithm:

1. Draw a resample of size n from {(Yᵢ, Dᵢ, Xᵢ)}ᵢ₌₁ⁿ with replacement.
2. Compute the estimate τ̂ᵇ on the resample.
3. Repeat steps 1-2 B times (typically B = 999 or B = 1999).
4. Construct the confidence interval from the bootstrap distribution.

The paired bootstrap works well for standard estimators (OLS, matching, IPW) in large samples with i.i.d. observations. It inherits the asymptotic properties of the original estimator.

4 Confidence Intervals from the Bootstrap

Three main approaches for constructing bootstrap confidence intervals:

Percentile interval: Use the α/2 and 1 - α/2 quantiles of the bootstrap distribution:

Simple, but can be biased in small samples or when the estimator distribution is skewed.

Percentile-t (studentised) interval: Compute the t-statistic tᵇ = (θ̂ᵇ - θ̂) / SÊᵇ in each resample, and use its quantiles to construct the interval:

More accurate than the percentile interval, especially with skewed distributions, but requires computing a standard error in each bootstrap resample.

BCa (bias-corrected and accelerated) interval: Adjusts for both bias and skewness in the sampling distribution. Generally the most reliable nonparametric bootstrap interval, recommended by Efron and Tibshirani [1993] for practical use.

5 The Cluster Bootstrap

Causal inference studies often involve clustered data: students within schools, workers within firms, counties within states. When treatment is assigned at the cluster level (e.g., a school receives a programme), observations within a cluster are correlated. Standard errors must account for this clustering.

When the number of clusters is small (fewer than 30-50), asymptotic cluster-robust standard errors (using the standard sandwich formula) may be unreliable. The cluster bootstrap provides a valid alternative:

Algorithm (cluster bootstrap):

1. Identify the G clusters {g₁, ..., g_G}.
2. Draw a resample of G clusters with replacement (i.e., resample clusters, not individual observations).
3. Include all observations from each selected cluster in the resample.
4. Estimate τ̂ᵇ on the resample.
5. Repeat B times and compute the bootstrap standard error or confidence interval.

Cameron et al. [2008] show that the cluster bootstrap performs well even with a modest number of clusters (G ≥ 10) and outperforms asymptotic cluster-robust standard errors with fewer than 20-30 clusters.

6 The Wild Bootstrap

The wild bootstrap, introduced by Wu [1986] and extended by Cameron et al. [2008], is particularly important for difference-in-differences designs with few treated clusters. In a DiD with G₁ treated and G₀ control clusters, with G₁ small (e.g., 5-10 states adopt a new policy), the cluster bootstrap can fail because resampling clusters with replacement cannot generate treated clusters that were "not treated"—the bootstrap distribution of the difference-in-means estimator is constrained.

The wild bootstrap imposes the null hypothesis and draws inference by:

1. Compute the restricted residuals ê_g under the null hypothesis H₀: τ = 0 (regress Yᵢₜ on time and unit fixed effects only, omitting the treatment indicator).
2. In each bootstrap resample, multiply the cluster residual ê_g by a randomly drawn Rademacher weight w_g ∈ {-1, +1} with equal probability.
3. Reconstruct a bootstrap outcome Yᵢₜᵇ = Ŷᵢₜ + w_g ê_g and re-estimate the treatment effect.

The wild bootstrap p-value for a two-sided test is the fraction of |τ̂ᵇ| that exceed |τ̂| in absolute value. MacKinnon and Webb [2017] provide refinements that improve accuracy further.

Figure 1: Bootstrap distribution of the centred statistic τ̂ᵇ - τ̂. The 2.5th and 97.5th percentiles define the 95% confidence interval by the percentile-t method.

7 Numerical Example in R

The following R code illustrates paired and cluster bootstrap for a DiD estimator:

Listing 1: Bootstrap for a DiD estimator in R

8 When the Bootstrap Fails

The bootstrap does not always work. Key failure modes:

• Few clusters. With G < 10 clusters, the cluster bootstrap lacks sufficient variation across resamples and is unreliable. The wild bootstrap is preferred.
• Weak instruments. Bootstrap confidence intervals for IV estimators with weak instruments (small first-stage F) are unreliable because the sampling distribution of 2SLS is non-normal and can be bimodal. Conditional likelihood ratio (CLR) tests [Moreira, 2003] are preferred.
• Non-standard estimators. Some estimators are not bootstrap-consistent—most notably, the maximum of a sample (e.g., the min or max of individual treatment effects in certain bounds analyses) and quantile estimators near the boundary of the support.

9 Common Mistakes

• Not resampling the right unit. If treatment is clustered at the school level, resample schools, not students.
• Too few resamples. Use B ≥ 999 for standard errors, B ≥ 4999 for percentile CIs, and B ≥ 9999 for hypothesis tests at the 1% level.
• Using the bootstrap for bias correction. The bootstrap bias correction θ̂ - (θ̄* - θ̂) often amplifies noise rather than reducing bias.
• Ignoring strata in stratified samples. Bootstrap within strata to preserve the sampling design.

Conclusion

The bootstrap is one of the most versatile and widely applicable tools in the applied econometrician's toolkit. By approximating the sampling distribution of any estimator through resampling, it provides standard errors and confidence intervals without requiring closed-form analytical expressions. The paired bootstrap works for most estimators with i.i.d. data. For clustered causal inference designs—the norm in economics—the cluster bootstrap and wild bootstrap provide valid inference even when the number of clusters is too small for asymptotic approximations. Knowing when each variant applies, and when the bootstrap fails, is essential for credible empirical work.

References

1. Cameron, A. C., Gelbach, J. B., and Miller, D. L. (2008). Bootstrap-based improvements for inference with clustered errors. Review of Economics and Statistics, 90(3):414-427.
2. Efron, B. (1979). Bootstrap methods: Another look at the jackknife. Annals of Statistics, 7(1):1-26.
3. Efron, B. and Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall.
4. MacKinnon, J. G. and Webb, M. D. (2017). Wild bootstrap inference for wildly different cluster sizes. Journal of Applied Econometrics, 32(2):233-254.
5. Moreira, M. J. (2003). A conditional likelihood ratio test for structural models. Econometrica, 71(4):1027-1048.
6. Wu, C. F. J. (1986). Jackknife, bootstrap and other resampling methods in regression analysis. Annals of Statistics, 14(4):1261-1295.