Sample Attrition Bias in Randomized Experiments: A Tale of Two Surveys

31 Pages Posted: 19 May 2009

See all articles by Luc Behaghel

Luc Behaghel

Paris School of Economics; National Institute of Statistics and Economic Studies (INSEE) - Center for Research in Economics and Statistics (CREST)

Bruno Crepon

National Institute of Statistics and Economic Studies (INSEE) - National School for Statistical and Economic Administration (ENSAE); IZA Institute of Labor Economics

Marc Gurgand

National Center for Scientific Research (CNRS); National Institute of Statistics and Economic Studies (INSEE) - Center for Research in Economics and Statistics (CREST)

Thomas Le Barbanchon

Bocconi university; Centre for Economic Policy Research (CEPR)

Abstract

The randomized trial literature has helped to renew the fields of microeconometric policy evaluation by emphasizing identification issues raised by endogenous program participation. Measurement and attrition issues have perhaps received less attention. This paper analyzes the dramatic impact of sample attrition in a large job search experiment. We take advantage of two independent surveys on the same initial sample of 8, 000 persons. The first one is a long telephone survey that had a strikingly low and unbalanced response rate of about 50%. The second one is a combination of administrative data and a short telephone survey targeted at those leaving the unemployment registers; this enriched data source has a balanced and much higher response rate (about 80%). With naive estimates that neglect non responses, these two sources yield puzzlingly different results. Using the enriched administrative data as benchmark, we find evidence that estimates from the long telephone survey lack external and internal validity. We turn to existing methods to bound the effects in the presence of sample selection; we extend them to the context of randomization with imperfect compliance. The bounds obtained from the two surveys are compatible but those from the long telephone survey are somewhat uninformative. We conclude on the consequences for data collection strategies.

Keywords: randomized evaluation, survey non response, bounds

JEL Classification: C31, C93, J68

Suggested Citation

Behaghel, Luc and Crepon, Bruno and Gurgand, Marc and Le Barbanchon, Thomas, Sample Attrition Bias in Randomized Experiments: A Tale of Two Surveys. IZA Discussion Paper No. 4162. Available at SSRN: https://ssrn.com/abstract=1405930

Luc Behaghel (Contact Author)

Paris School of Economics ( email )

48 Boulevard Jourdan
Paris, 75014 75014
France

National Institute of Statistics and Economic Studies (INSEE) - Center for Research in Economics and Statistics (CREST) ( email )

15 Boulevard Gabriel Peri
Malakoff Cedex, 1 92245
France

Bruno Crepon

National Institute of Statistics and Economic Studies (INSEE) - National School for Statistical and Economic Administration (ENSAE) ( email )

92245 Malakoff Cedex
France

IZA Institute of Labor Economics

P.O. Box 7240
Bonn, D-53072
Germany

Marc Gurgand

National Center for Scientific Research (CNRS) ( email )

3, rue Michel-Ange
Paris cedex 16, 75794
France

National Institute of Statistics and Economic Studies (INSEE) - Center for Research in Economics and Statistics (CREST)

15 Boulevard Gabriel Peri
Malakoff Cedex, 1 92245
France

Thomas Le Barbanchon

Bocconi university ( email )

Via Sarfatti, 25
Milan, MI 20136
Italy

Centre for Economic Policy Research (CEPR) ( email )

London
United Kingdom

Register to save articles to
your library

Register

Paper statistics

Downloads
102
Abstract Views
984
rank
267,416
PlumX Metrics