Validation of Credit Rating Systems Using Multi-Rater Information

35 Pages Posted: 20 Dec 2005

See all articles by Kurt Hornik

Kurt Hornik

Vienna University of Economics and Business Administration - Department of Statistics and Mathematics

Rainer Jankowitsch

WU (Vienna University of Economics and Business); Vienna Graduate School of Finance (VGSF)

Manuel Lingo

Oesterreichische Nationalbank (OeNB); Vienna University of Economics and Business Administration

Stefan Pichler

WU - Vienna University of Economics and Business - Department of Finance, Accounting and Statistics; VGSF (Vienna Graduate School of Finance)

Gerhard Winkler

Oesterreichische Nationalbank (OeNB); Vienna University of Economics and Business Administration

Date Written: November 2006

Abstract

We suggest a new framework for the use of multi-rater information in the validation of credit rating systems, applicable in any validation process where rating information from different sources is available. As our validation framework does not rely on historical default information it appears to be particularly useful in situations where such information is inaccessible. We focus on the degree of similarity or - more generally - proximity of rating outcomes stemming from different sources and show that it is important to analyze three major aspects of proximity: agreement, association, and rating bias. We suggest using a weighted version of Cohen's kappa to measure the agreement between two rating systems and we introduce a new measure for rating bias. In contrast to the existing literature we suggest tau-x as a measure of association which is based on the Kemeny-Snell metric and, opposed to other measures, is consistent with a set of basic axioms and should therefore be used in the context of multi-rater information. We provide an illustrative empirical example using rating information stemming from the Austrian Credit Register on partially overlapping sets of customers of 27 banks. Using a multi-dimensional scaling technique in connection with a minimal spanning tree we show that it is possible to consistently detect outliers, i.e., banks with a low degree of similarity to other banks. The results indicate that banks which are less diversified across the size of their loans are more likely to be outliers than others.

Keywords: rating validation, benchmarking, rating agreement, rating association, rating bias, Kemeny-Snell metric, multi-dimensional scaling, minimal spanning tree

JEL Classification: G20, C49

Suggested Citation

Hornik, Kurt and Jankowitsch, Rainer and Lingo, Manuel and Pichler, Stefan and Winkler, Gerhard, Validation of Credit Rating Systems Using Multi-Rater Information (November 2006). Available at SSRN: https://ssrn.com/abstract=871213 or http://dx.doi.org/10.2139/ssrn.871213

Kurt Hornik

Vienna University of Economics and Business Administration - Department of Statistics and Mathematics ( email )

Vienna A-1090, Wien
Austria

Rainer Jankowitsch (Contact Author)

WU (Vienna University of Economics and Business) ( email )

Welthandelsplatz 1
Vienna, Vienna AT1020
Austria
+43 1 31 336 4340 (Phone)
+43 1 310 0580 (Fax)

Vienna Graduate School of Finance (VGSF) ( email )

Welthandelsplatz 1
Vienna, 1020
Austria

Manuel Lingo

Oesterreichische Nationalbank (OeNB) ( email )

Otto-Wagner-Platz 3
1090 Vienna
Austria

Vienna University of Economics and Business Administration

Augasse 2-6
Vienna A-1090
Austria

Stefan Pichler

WU - Vienna University of Economics and Business - Department of Finance, Accounting and Statistics ( email )

Heiligenstaedter Strasse 46-48
Wien 1190
Austria

VGSF (Vienna Graduate School of Finance) ( email )

Heiligenstaedter Strasse 46-48
Vienna, 1190
Austria

Gerhard Winkler

Oesterreichische Nationalbank (OeNB) ( email )

Otto-Wagner-Platz 3
1090 Vienna
Austria

Vienna University of Economics and Business Administration ( email )

Welthandelsplatz 1
Vienna, Wien 1020
Austria

Here is the Coronavirus
related research on SSRN

Paper statistics

Downloads
523
Abstract Views
2,654
rank
59,153
PlumX Metrics