Measuring the Measurement Error: A Method to Qualitatively Validate Survey Data

47 Pages Posted: 18 Aug 2015

See all articles by Christopher Blattman

Christopher Blattman

University of Chicago, Harris School of Public Policy; National Bureau of Economic Research (NBER)

Julian C. Jamison

University of Exeter Business School - Department of Economics; World Bank eMBeD (Mind, Behavior, and Development); Massachusetts Institute of Technology (MIT) - Abdul Latif Jameel Poverty Action Lab (J-PAL); Innovations for Poverty Action

Tricia Koroknay-Palicz

The World Bank

Katherine Rodrigues

Columbia University SIPA; International Rescue Committee

Margaret Sheridan

Harvard Medical School/Boston Children's Hospital; University of North Carolina (UNC) at Chapel Hill - Department of Psychology and Neuroscience

Multiple version iconThere are 2 versions of this paper

Date Written: August 2015

Abstract

Field experiments rely heavily on self-reported data, but subjects may misreport behaviors, especially sensitive ones such as crime. If treatment influences survey responses, it biases experimental estimates. We develop a validation technique that uses intensive qualitative work to assess survey measurement error. Subjects were assigned to receive cash, therapy, both, or neither. According to survey responses, receiving both treatments dramatically reduced crime and other sensitive behaviors. Local researchers spent several days with a random subsample of subjects following their endline surveys, building trust and seeking verbal confirmation of six behaviors: theft, drug use, homelessness, gambling, and two expenditures. This validation suggests that subjects in the control and cash only groups underreported sensitive behaviors and expenditures in the survey relative to the other treatment arms. We bound survey-based treatment effects estimates, and find the impacts of cash and therapy on crime may be larger than suggested by surveys alone.

Suggested Citation

Blattman, Christopher and Jamison, Julian C. and Koroknay-Palicz, Tricia and Rodrigues, Katherine and Sheridan, Margaret, Measuring the Measurement Error: A Method to Qualitatively Validate Survey Data (August 2015). NBER Working Paper No. w21447. Available at SSRN: https://ssrn.com/abstract=2645523

Christopher Blattman (Contact Author)

University of Chicago, Harris School of Public Policy ( email )

1101 East 58th Street
Chicago, IL 60637
United States

National Bureau of Economic Research (NBER) ( email )

1050 Massachusetts Avenue
Cambridge, MA 02138
United States

Julian C. Jamison

University of Exeter Business School - Department of Economics ( email )

Streatham Court
Exeter, EX4 4RJ
United Kingdom

World Bank eMBeD (Mind, Behavior, and Development) ( email )

1818 H Street, N.W.
Washington, DC 20433
United States

Massachusetts Institute of Technology (MIT) - Abdul Latif Jameel Poverty Action Lab (J-PAL) ( email )

30 Wadsworth Street, E53-320
77 Massachusetts Avenue
Cambridge, MA 02142
United States

Innovations for Poverty Action ( email )

1731 Connecticut Ave, 4th floor
New Haven, CT 20009
United States

Tricia Koroknay-Palicz

The World Bank ( email )

1818 H Street, NW
Washington, DC 20433
United States

Katherine Rodrigues

Columbia University SIPA ( email )

420 West 118th Street
New York, NY 10027
United States

International Rescue Committee ( email )

New York, NY
United States

Margaret Sheridan

Harvard Medical School/Boston Children's Hospital ( email )

25 Shattuck St
Boston, MA 02115
United States

University of North Carolina (UNC) at Chapel Hill - Department of Psychology and Neuroscience ( email )

Chapel Hill, NC 27599
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
7
Abstract Views
285
PlumX Metrics