Beyond the Bat and the Ball: Overcoming Familiarity Effects in the Cognitive Reflection Test by Rewording Its Questions

Abstract

The cognitive reflection test (CRT) is a task that is commonly used for measuring one’s tendency to form judgments and make decisions by relying on her intuition, as opposed to deliberate thinking. The popularity of the CRT as a research instrument has led to concerns that participant familiarity might inflate scores in the task. Here, we explore the possibility of overcoming this limitation by rewording the questions used in the original 7-item CRT. We generated three new versions of the test and administered them, as well as the original test, to 1006 participants from Amazon Mechanical Turk. Consistent with a familiarity effect, we find that participants who took the original CRT had higher scores (d=.2 ) and took less time (30-55 seconds) to complete the test than participants who took the new versions. The across subject correlations between scores on the new versions and the original version were high (around r=.9), suggesting that scores of the new versions can be used to approximate scores on the original version while overcoming the familiarity bias.