In Praise of Moderation: Suggestions for the Scope and Use of Pre-Analysis Plans for Rcts in Economics

16 Pages Posted: 20 Apr 2020 Last revised: 27 Apr 2020

See all articles by Esther Duflo

Esther Duflo

Massachusetts Institute of Technology (MIT) - Department of Economics; Abdul Latif Jameel Poverty Action Lab (J-PAL); National Bureau of Economic Research (NBER); Centre for Economic Policy Research (CEPR); Bureau for Research and Economic Analysis of Development (BREAD)

Abhijit V. Banerjee

Massachusetts Institute of Technology (MIT) - Department of Economics

Amy Finkelstein

Massachusetts Institute of Technology (MIT) - Department of Economics; National Bureau of Economic Research (NBER)

Lawrence F. Katz

Harvard University - Department of Economics; National Bureau of Economic Research (NBER)

Benjamin Olken

Massachusetts Institute of Technology (MIT) - Department of Economics

Anja Sautmann

Massachusetts Institute of Technology; Brown University - Department of Economics

Date Written: April 2020

Abstract

Pre-Analysis Plans (PAPs) for randomized evaluations are becoming increasingly common in Economics, but their definition remains unclear and their practical applications therefore vary widely. Based on our collective experiences as researchers and editors, we articulate a set of principles for the ex-ante scope and ex-post use of PAPs. We argue that the key benefits of a PAP can usually be realized by completing the registration fields in the AEA RCT Registry. Specific cases where more detail may be warranted include when subgroup analysis is expected to be particularly important, or a party to the study has a vested interest. However, a strong norm for more detailed pre-specification can be detrimental to knowledge creation when implementing field experiments in the real world. An ex-post requirement of strict adherence to pre-specified plans, or the discounting of non-pre-specified work, may mean that some experiments do not take place, or that interesting observations and new theories are not explored and reported. Rather, we recommend that the final research paper be written and judged as a distinct object from the “results of the PAP”; to emphasize this distinction, researchers could consider producing a short, publicly available report (the “populated PAP”) that populates the PAP to the extent possible and briefly discusses any barriers to doing so.

Institutional subscribers to the NBER working paper series, and residents of developing countries may download this paper without additional charge at www.nber.org.

Suggested Citation

Duflo, Esther and Banerjee, Abhijit V. and Finkelstein, Amy and Katz, Lawrence F. and Olken, Benjamin and Sautmann, Anja, In Praise of Moderation: Suggestions for the Scope and Use of Pre-Analysis Plans for Rcts in Economics (April 2020). NBER Working Paper No. w26993, Available at SSRN: https://ssrn.com/abstract=3580551

Esther Duflo (Contact Author)

Massachusetts Institute of Technology (MIT) - Department of Economics ( email )

50 Memorial Drive
Room E52-544
Cambridge, MA 02139
United States
617-258-7013 (Phone)
617-253-6915 (Fax)

Abdul Latif Jameel Poverty Action Lab (J-PAL) ( email )

Cambridge, MA
United States

HOME PAGE: http://www.povertyactionlab.org/

National Bureau of Economic Research (NBER)

1050 Massachusetts Avenue
Cambridge, MA 02138
United States

Centre for Economic Policy Research (CEPR)

London
United Kingdom

Bureau for Research and Economic Analysis of Development (BREAD) ( email )

Duke University
Durham, NC 90097
United States

Abhijit V. Banerjee

Massachusetts Institute of Technology (MIT) - Department of Economics ( email )

50 Memorial Drive
Room E52-252D
Cambridge, MA 02142
United States
617-253-8855 (Phone)
617-253-6915 (Fax)

Amy Finkelstein

Massachusetts Institute of Technology (MIT) - Department of Economics ( email )

50 Memorial Drive
E52-391
Cambridge, MA 02142
United States
617-588-0361 (Phone)
617-868-7242 (Fax)

National Bureau of Economic Research (NBER) ( email )

1050 Massachusetts Avenue
Cambridge, MA 02138
United States

Lawrence F. Katz

Harvard University - Department of Economics ( email )

Littauer Center
Room 215
Cambridge, MA 02138
United States
617-495-5148 (Phone)
617-868-2742 (Fax)

HOME PAGE: http://www.economics.harvard.edu/faculty/katz/katz

National Bureau of Economic Research (NBER) ( email )

1050 Massachusetts Avenue
Cambridge, MA 02138
United States

Benjamin Olken

Massachusetts Institute of Technology (MIT) - Department of Economics ( email )

50 Memorial Drive
E52-391
Cambridge, MA 02142
United States

Anja Sautmann

Massachusetts Institute of Technology ( email )

77 Massachusetts Avenue
50 Memorial Drive
Cambridge, MA 02139-4307
United States

Brown University - Department of Economics ( email )

64 Waterman Street
Providence, RI 02912
United States

Here is the Coronavirus
related research on SSRN

Paper statistics

Downloads
6
Abstract Views
102
PlumX Metrics