Statistical Significance and Policy Significance

Abstract

Research findings sometimes play a part in the policy making process. This can happen through direct use of analysis, or through their impact on public perceptions and preferences as a result of media coverage. There are opportunities in this process for distortions to occur. This paper looks at one aspect of this, namely the potential for statistical significance to be interpreted as policy significance. The two are not the same. They can diverge for several reasons. Some, such as the difference between correlation and causation, are widely recognised in statistics. Others may merit some attention. Two aspects considered here are first, the policy options suggested by statistical significance, and second, some policy analysis criteria that are not covered by statistical significance alone.

On the first aspect, there are two perspectives on an association between two variables from which policy suggestions may be made. One policy approach would be to alter the value of one variable, the policy instrument, to alter the value of another variable, the target variable. This is the usual perspective. Sometimes it is not possible to change policy instrument. The policy focus then turns to means of changing the relationship between the two variables. This second option, altering the relationship, may be generally available when considering relationships between variables. However, if the first option is feasible, then the second is generally overlooked.

The second aspect to be considered in the paper is that there are some well-recognised policy analysis criteria that are not covered by statistical significance alone. Standard economic approaches to policy making require consideration of alternative options, including assessment of their costs and benefits. There could also be thought given to the extent of control available, and risk and uncertainty. Explicit consideration of policy aspects may result in improved use of statistical findings.

Keywords: econometrics, policy, significance, evaluation

JEL Classification: B40, C10, H80