Striking the Right Balance: Why Standard Balance Tests Over-Reject the Null, and How to Fix it

58 Pages Posted: 19 Aug 2024

See all articles by Jason Kerwin

Jason Kerwin

University of Washington - Economics; IZA; Massachusetts Institute of Technology (MIT) - Abdul Latif Jameel Poverty Action Lab (J-PAL)

Nada Rostom

University of Antwerp

Olivier Sterck

University of Antwerp - Institute of Development Policy; University of Oxford - Department of International Development; University of Oxford - Centre for the Study of African Economies (CSAE)

Abstract

Economists often use balance tests to demonstrate that the treatment and control groups are comparable prior to an intervention. We show that typical implementations of balance tests have poor statistical properties. Pairwise t-tests leave it unclear how many rejections indicate overall imbalance. Omnibus tests of joint orthogonality, in which the treatment is regressed on all the baseline covariates, address this ambiguity but substantially over-reject the null hypothesis using the sampling-based p-values that are typical in the literature. This problem is exacerbated when the number of covariates is high compared to the number of observations. We examine the performance of alternative tests, and show that omnibus F-tests of joint orthogonality with randomization inference p-values have the correct size and reasonable power. We apply these tests to data from two prominent recent articles, where standard F-tests indicate imbalance, and show that the study arms are actually balanced when appropriate tests are used.

Keywords: balance tests, power, size, randomization inference

JEL Classification: C1, C9, O12

Suggested Citation

Kerwin, Jason and Rostom, Nada and Sterck, Olivier, Striking the Right Balance: Why Standard Balance Tests Over-Reject the Null, and How to Fix it. IZA Discussion Paper No. 17217, Available at SSRN: https://ssrn.com/abstract=4926535

Jason Kerwin (Contact Author)

University of Washington - Economics ( email )

Seattle, WA
United States

IZA ( email )

P.O. Box 7240
Bonn, D-53072
Germany

Massachusetts Institute of Technology (MIT) - Abdul Latif Jameel Poverty Action Lab (J-PAL) ( email )

30 Wadsworth Street, E53-320
77 Massachusetts Avenue
Cambridge, MA 02142
United States

Nada Rostom

University of Antwerp ( email )

Prinsstraat 13
Antwerp, 2000
Belgium

Olivier Sterck

University of Antwerp - Institute of Development Policy ( email )

Belgium

University of Oxford - Department of International Development ( email )

3 Mansfield Road
Oxford, OX1 3TB
United Kingdom

University of Oxford - Centre for the Study of African Economies (CSAE) ( email )

Oxford OX1 3UL
United Kingdom

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
29
Abstract Views
144
PlumX Metrics