The Mystery of the Vanishing Benefits: Ms. Speedy Analyst's Introduction to Evaluation

Abstract

This entertaining introduction to the concepts and methods of impact evaluation - as seen through the eyes of Ms. Speedy Analyst - assumes readers are familiar with basic statistics up to regression analysis (as covered in an introductory text on econometrics). The setting for this good-natured training guide for impact evaluation is the fictional developing country Labas.

Twelve months ago the government introduced an antipoverty program in Northwest Labas with support from the World Bank. The program aims to provide cash transfers to poor families with school-age children. To be eligible to receive the transfer, households must have observable characteristics that suggest they are poor. To continue receiving the transfer, they must keep their children in school until 18 years of age. The program is called PROSCOL.

The government wants to assess PROSCOL's impact on poverty, to help decide whether the program should be expanded or dropped. The Finance Minister asks the undersecretary, and the under-secretary calls in Ms. Speedy Analyst. Ms. Speedy Analyst's on-the-job training in how to assess the impact of a social program provides the vehicle through which this paper explains: Methods of evaluating a program's impact - randomizing, matching, reflexive comparisons, double difference (or difference in difference) methods, and instrumental variables methods. The types of data used for impact evaluation, typical problems with and uses of data, control variables, instrumental variables, regressions, and so on.

How to form and match comparison groups.

Sources of bias.

The value of baseline surveys.

Measures of poverty (headcount index, poverty gap index, and squared poverty gap).

How to compare poverty with and without the program.

This paper - a product of Poverty and Human Resources, Development Research Group - is part of a larger effort in the group to provide useful training tools for Bank staff.