Is it Worth the While? The Relevance of Qualitative Information in Credit Rating

Abstract

This empirical study deals with the question whether soft facts (qualitative information, i.e. subjective judgments of credit analysts) considerably improve the forecast quality of bank internal credit ratings that are solely based on hard facts (financial ratios, checking account data).

An extensive sample (20.000 observations) of German SME credit data has been made available by a commercial bank to compare two models: one including, the other excluding qualitative information. Logistic regression is used to forecast default probabilities. The econometric quality as well as the classification and separation performance of the two models are assessed statistically and graphically, using various measures. It is shown that most of the widely used measures cannot be used for inter-sample comparison and offer therefore little informational content. Further, performance measures that are based on classification tables (i.e. a single cut-off value) should only be used if the (costs) benefits of (mis)classification are known. ROC (Receiver Operating Characteristic curve)-based measures and the inspection of ROC have been found to be the most useful criteria for model comparison. ROC inspection allows to compare models in a more qualitative way, adding information to the common inspection of numerical criteria.

The problem of observing very few defaults is solved by stratifying the estimation sample and using a re-sampling procedure. This results in empirical distributions of the performance measures. While, in similar studies, one (mean) value per measure and model is given, the existence of empirical distributions allows the difference between the two models to be quantified and to decide whether the models differ significantly.

The model including qualitative information significantly dominates the hard facts model in virtually all respects. The inspection of ROC indicates that, contrary to credit analysts' intuition, the hard-facts variable based on financial ratios performs better in the high-risk region than the softfacts variable based on credit analysts' judgments. It is not possible to infer from the results of this study whether the increase in performance of a credit rating model justifies the additional costs of obtaining qualitative information for a particular bank. Yet, this study shows that subjective judgments are indeed capable of yielding valuable information and improve credit rating systems which are based solely on quantitative information by considerable amounts. It uses several methods to quantify this increase in forecast quality and is based, as one of the first studies so far, on an extensive set of "real world" credit data.

JEL Classification: G21, C25, C10