Model Selection Procedures and Their Error-Reduction Targets
16 Pages Posted: 24 Mar 2011 Last revised: 10 Apr 2011
Date Written: April 7, 2011
This brief note compares model selection procedures in regression. On the one hand there is an observed error reduction ratio that we calculate from the data: h = SSE2/SSE1, where SSE1 and SSE2 are the sums of squared errors in Models 1 and 2, respectively. On the other hand there is a target ratio, which we call H, which is set by the model selection procedure via an expression involving the numbers of variables and observations. If h < H the procedure accepts Model 2 as superior to Model 1. The aim of this note is to derive expressions for each model selection procedure that have the form H = 1/(1x), where x varies from procedure to procedure. These common-form expressions for H allow us to compare model selection procedures transparently.
The procedures we examine are: Akaike’s Information Criterion (AIC), Bayesian Information Criterion (BIC), Amemiya’s Prediction Criterion (PC), change in R2 (ΔR2), adjusted R2, and adjusted Rf2 which we have created as a simple generalization of adjusted R2. We show that it is less liberal than adjusted R2 and relates closely to AIC and PC: in so doing it provides fresh insight into their properties.
Keywords: Model Selection, AIC, BIC, Prediction Criterion, Adjusted R2, F-Test, Error Reduction Ratio
JEL Classification: C52, C51, C50, C10, C00
Suggested Citation: Suggested Citation