Rating the Rating Scales

Abstract

Rating scales are used quite frequently in research, especially in surveys. Typically, an itemized rating scale asks subjects to choose one response category from several arranged in hierarchical order. Dishonest researchers can, of course, purposefully manipulate the outcome of their research, if they wish, but such biasing may also be totally unintentional. This paper examines issues involved in creating a relatively unbiased rating scale. These include: (1) Connotations of category labels; (2) Response alternative effects; (3) Implicit assumptions of the question; (4) Forced-choice vs. non-forced-choice rating scales; (5) Unbalanced and balanced rating scales; (6) Order effects; (7) Direction of comparison; (8) Optimal number of points; (9) Context effects; (10) Rating approach, e.g., improvement needed, performance, comparison to expectations, comparison to ideal, etc.

Keywords: Rating scales, context effects, measurement bias, unbalanced rating scales, research bias

JEL Classification: C00, C10, C40, I20