Quantifying the Value of Models and Data: a Comparison of the Performance of Regression and Neural Nets When Data Quality Varies

Abstract

Under circumstances where data quality may vary, knowledge about the potentialperformance of alternate predictive models can enable a decision maker to design aninformation system whose value is optimized in two ways. The decision maker can selecta model which is least sensitive to predictive degradation in the range of observed dataquality variation. And, once the "right" model has been selected, the decision maker canselect the appropriate level of data quality in view of the costs of acquiring it. This paperexamines a real-world example from the field of finance -- prepayments in mortgage-backedsecurities (MBS) portfolio management -- to illustrate a methodology that enables suchevaluations to be made for two modeling alternative: regression analysis and neural networkanalysis. The methodology indicates that with "perfect data," the neural network approachoutperforms regression in terms of predictive accuracy and utility in a prepayment riskmanagement forecasting system (RMFS). Further, the performance of the neural networkmodel is more robust under conditions of data quality degradation.

Keywords: business value of information technology, data quality, decision support systems, information economics, neural networks, risk management, risk management forecasting systems, systems design