Analyzing Ordinal Data with Metric Models: What Could Possibly Go Wrong?

Abstract

We surveyed all articles in the Journal of Personality and Social Psychology (JPSP), Psychological Science (PS), and the Journal of Experimental Psychology: General (JEP:G) that mentioned the term "Likert," and found that 100% of the articles that analyzed ordinal data did so using a metric model. We present new evidence that analyzing ordinal data as metric is problematic. We demonstrate that treating ordinal data as metric can yield low correct detection rates, distorted effect size estimates, and greatly inflated false alarm rates. Moreover, we demonstrate that the averaging of multiple ordinal items into a Likert scale does not solve these problems. We provide examples of real data in the contexts of the difference of two groups and simple linear regression. To solve these problems we use an ordered probit model with Bayesian estimation of parameters. The ordered probit model shows appropriate correct detection rates and false alarm rates, and produces accurate effect sizes estimates and response probabilities. Bayesian estimation of this ordinal model is straight forward, yields rich and accurate information, and has no need for auxiliary sampling assumptions. We conclude that ordinal data ought to be analyzed with ordinal models, and that Bayesian estimation is an excellent method for accomplishing that goal.

Keywords: Ordinal data, Likert scales, Bayesian data analysis, false alarm rate, effect size