A Comment on Diagnostic Tools for Counterfactual Inference

Posted: 18 Aug 2009

See all articles by Nicholas Sambanis

Nicholas Sambanis

Department of Political Science, University of Pennsylvania

Alexander Michaelides

Imperial College Business School; Centre for Economic Policy Research (CEPR)

Date Written: Winter 2009

Abstract

We evaluate two diagnostic tools used to determine if counterfactual analysis requires extrapolation. Counterfactuals based on extrapolation are model dependent and might not support empirically valid inferences. The diagnostics help researchers identify those counterfactual “what if” questions that are empirically plausible. We show, through simple Monte Carlo experiments, that these diagnostics will often detect extrapolation, suggesting that there is a risk of biased counterfactual inference when there is no such risk of extrapolation bias in the data. This is because the diagnostics are affected by what we call the n/k problem: as the number of data points relative to the number of explanatory variables decreases, the diagnostics are more likely to detect the risk of extrapolation bias even when such risk does not exist. We conclude that the diagnostics provide too severe a test for many data sets used in political science.

Suggested Citation

Sambanis, Nicholas and Michaelides, Alexander, A Comment on Diagnostic Tools for Counterfactual Inference (Winter 2009). Political Analysis, Vol. 17, Issue 1, pp. 89-106, 2009. Available at SSRN: https://ssrn.com/abstract=1448425 or http://dx.doi.org/10.1093/pan/mpm032

Nicholas Sambanis (Contact Author)

Department of Political Science, University of Pennsylvania ( email )

Philadelphia, PA 19104
United States

Alexander Michaelides

Imperial College Business School ( email )

South Kensington Campus
Exhibition Road
London SW7 2AZ, SW7 2AZ
United Kingdom

Centre for Economic Policy Research (CEPR)

London
United Kingdom

Here is the Coronavirus
related research on SSRN

Paper statistics

Abstract Views
338
PlumX Metrics