The Sensitivity of Experimental Impact Estimates: Evidence from the National Jtpa Study

50 Pages Posted: 18 Mar 1998 Last revised: 6 Oct 2010

See all articles by James J. Heckman

James J. Heckman

University of Chicago - Department of Economics; National Bureau of Economic Research (NBER); American Bar Foundation; Institute for the Study of Labor (IZA); CESifo (Center for Economic Studies and Ifo Institute)

Jeffrey A. Smith

University of Wisconsin - Madison; National Bureau of Economic Research (NBER); Institute for the Study of Labor (IZA)

Date Written: July 1997

Abstract

The recent experimental evaluation of the U.S. Job Training Partnership Act (JTPA) program found negative effects of training on the earnings of disadvantaged male youth and no effect on the earnings of disadvantaged female youth. These findings provided justification for Congress to cut the budget of JTPA's youth component by over 80 percent. In this paper, we examine the sensitivity of the experimental impact estimates along several dimensions of construction and interpretation. We find that the statistical significance of the male youth estimates is extremely fragile and that the magnitudes of the estimates for both youth groups are sensitive to nearly all the factors we consider. In particular, accounting for experimental control group members who substitute training from other providers leads to a much more positive picture regarding the effectiveness of JTPA classroom training. Our study indicates the value of sensitivity analyses in experimental evaluations and illustrates that experimental impact estimates, like those from nonexperimental analyses, require careful interpretation if they are to provide a reliable guide to policymakers.

Suggested Citation

Heckman, James J. and Smith, Jeffrey Andrew, The Sensitivity of Experimental Impact Estimates: Evidence from the National Jtpa Study (July 1997). NBER Working Paper No. w6105. Available at SSRN: https://ssrn.com/abstract=62768

James J. Heckman (Contact Author)

University of Chicago - Department of Economics ( email )

1126 East 59th Street
Chicago, IL 60637
United States
773-702-0634 (Phone)
773-702-8490 (Fax)

National Bureau of Economic Research (NBER)

1050 Massachusetts Avenue
Cambridge, MA 02138
United States

American Bar Foundation

750 N. Lake Shore Drive
Chicago, IL 60611
United States

Institute for the Study of Labor (IZA)

P.O. Box 7240
Bonn, D-53072
Germany

CESifo (Center for Economic Studies and Ifo Institute)

Poschinger Str. 5
Munich, DE-81679
Germany

Jeffrey Andrew Smith

University of Wisconsin - Madison

716 Langdon Street
Madison, WI 53706-1481
United States

National Bureau of Economic Research (NBER)

1050 Massachusetts Avenue
Cambridge, MA 02138
United States

Institute for the Study of Labor (IZA)

P.O. Box 7240
Bonn, D-53072
Germany

Register to save articles to
your library

Register

Paper statistics

Downloads
42
Abstract Views
1,087
PlumX Metrics