How Is Your User Feeling? Inferring Emotion Through Human-Computer Interaction Devices

Abstract

Experiencing negative emotion during system use can adversely influence important user behaviors, including purchasing decisions, technology use, and customer loyalty. The ability to easily assess users' negative emotion during live system use therefore has practical significance for the design and improvement of information systems. We utilize attentional control theory to explain how mouse cursor movements can be a real-time indicator of negative emotion. We report three studies. In Study 1, an experiment with 65 participants from Amazon's Mechanical Turk, we randomly manipulated negative emotion and then monitored participants' mouse cursor movements as they completed a number-ordering task. We found that negative emotion increases the distance and reduces the speed of mouse cursor movements during the task. In Study 2, an experiment with 126 participants from a U.S. university, we randomly manipulated negative emotion and then monitored participants' mouse cursor movements while they interacted with a mock e-commerce site. We found that mouse cursor distance and speed can be used to infer the presence of negative emotion with an overall accuracy rate of 81.7%. In Study 3, an observational study with 80 participants from universities in Germany and Hong Kong, we monitored mouse cursor movements while participants interacted with an online product configurator. Participants reported their level of emotion after each step in the configuration process. We found that mouse cursor distance and speed can be used to infer the level of negative emotion with an out-of-sample R² of 0.17. The results enable researchers to assess negative emotional reactions during live system use, examine emotional reactions with more temporal precision, conduct multi-method emotion research, and create more unobtrusive affective and adaptive systems.

Keywords: negative emotion, attentional control theory (ACT), mouse cursor distance, mouse cursor speed, mouse tracking, human-computer interaction

JEL Classification: L86