Measuring the Measurement Error: A Method to Qualitatively Validate Survey Data

46 Pages Posted: 13 Jun 2014 Last revised: 28 Jul 2015

Christopher Blattman

University of Chicago, Harris School of Public Policy; National Bureau of Economic Research (NBER)

Julian C. Jamison

World Bank Global Insights Initiative; Innovations for Poverty Action

Tricia G. Koroknay-Palicz

The World Bank

Katherine Rodrigues

Columbia University SIPA; International Rescue Committee

Margaret Sheridan

Harvard Medical School/Boston Children's Hospital; University of North Carolina (UNC) at Chapel Hill - Department of Psychology and Neuroscience

Multiple version iconThere are 2 versions of this paper

Date Written: July 27, 2015

Abstract

Field experiments rely heavily on self-reported data, but subjects may misreport behaviors, especially sensitive ones such as crime. If treatment influences survey responses, it biases experimental estimates. We develop a validation technique that uses intensive qualitative work to assess survey measurement error. Subjects were assigned to receive cash, therapy, both, or neither. According to survey responses, receiving both treatments dramatically reduced crime and other sensitive behaviors. Local researchers spent several days with a random subsample of subjects following their endline surveys, building trust and seeking verbal confirmation of six behaviors: theft, drug use, homelessness, gambling, and two expenditures. This validation suggests that subjects in the control and cash only groups underreported sensitive behaviors and expenditures in the survey relative to the other treatment arms. We bound survey-based treatment effects estimates, and find the impacts of cash and therapy on crime may be larger than suggested by surveys alone.

Keywords: survey methodology, field experiments, measurement error, crime, drug use, Liberia

JEL Classification: O1, C93, I32, C81, K4

Suggested Citation

Blattman, Christopher and Jamison, Julian C. and Koroknay-Palicz, Tricia G. and Rodrigues, Katherine and Sheridan, Margaret, Measuring the Measurement Error: A Method to Qualitatively Validate Survey Data (July 27, 2015). Available at SSRN: https://ssrn.com/abstract=2449048 or http://dx.doi.org/10.2139/ssrn.2449048

Christopher Blattman (Contact Author)

University of Chicago, Harris School of Public Policy ( email )

1101 East 58th Street
Chicago, IL 60637
United States

National Bureau of Economic Research (NBER) ( email )

1050 Massachusetts Avenue
Cambridge, MA 02138
United States

Julian C. Jamison

World Bank Global Insights Initiative ( email )

1818 H Street, N.W.
Washington, DC 20433
United States
202-817-4942 (Phone)

Innovations for Poverty Action ( email )

1731 Connecticut Ave, 4th floor
New Haven, CT 20009
United States

Tricia G. Koroknay-Palicz

The World Bank ( email )

1818 H Street, NW
Washington, DC 20433
United States

Katherine Rodrigues

Columbia University SIPA ( email )

420 West 118th Street
New York, NY 10027
United States

International Rescue Committee ( email )

New York, NY
United States

Margaret Sheridan

Harvard Medical School/Boston Children's Hospital ( email )

250 Longwood Avenue
Boston, MA 02115
United States

University of North Carolina (UNC) at Chapel Hill - Department of Psychology and Neuroscience ( email )

Chapel Hill, NC 27599
United States

Paper statistics

Downloads
622
Rank
33,103
Abstract Views
4,276