Creating a Bot-tleneck for Malicious AI: Psychological Methods for Bot Detection

Abstract

The standard approach for detecting and preventing bots from doing harm online involves CAPTCHAs. However, recent AI research suggests that bots can complete many common CAPTCHAs with ease. The most effective methodology for identifying potential bots: involves completing image-processing, causal-reasoning based, free-response questions that are hand coded by human analysts. However, this approach is labor intensive, slow, and inefficient. Here, we develop and test various automated, bot-screening questions, grounded in psychological research, to serve as a proactive screen against bots. Utilizing hand coded free-response questions in the naturalistic domain of MTurkers recruited for a Qualtrics survey, we identify 18.9% of our sample to be bots, whereas Google’s reCAPTCHA V3 identified only 1.7% to be bots. We then look at the performance of these identified bots on our novel bot-screeners, each of which has different strengths and weaknesses but all of which outperform CAPTCHAs.

Keywords: Bot-detection, Survey Design, Cybersecurity, Data Validation, Human-computer interaction