Generative AI in Action: Field Experimental Evidence on Worker Performance in E-Commerce Customer Service Operations

Abstract

In collaboration with Alibaba, this study leverages a large-scale field experiment to quantify the impact of a generative AI (gen AI) assistant on worker performance in an e-commerce after-sales service setting, where human agents provide customer support through digital chat. Agents were randomly assigned to either a control or treatment group, with the latter having access to a gen AI assistant that offers two automated functions as text messages: 1) diagnosis of customer order issues in real time and 2) solution suggestions. Agents exhibited varied gen AI usage behavior, choosing to use, modify, or disregard AI suggested messages. We employ two empirical approaches: 1) an intention-to-treat (ITT) analysis to estimate the average treatment effect of gen AI access, and 2) a local average treatment effect (LATE) analysis to estimate the causal impact of gen AI usage. Results show that the gen AI assistant significantly enhanced both service speed and service quality. Interestingly, gen AI automation did not lead agents to reduce effort; rather, it increased their engagement, evidenced by a higher message volume and agent-to-customer message ratio. Analysis by agents' pretreatment performance reveals that low performers experienced greater improvements in speed and quality, narrowing the performance gap, while high performers saw a decline in service quality, likely because gen AI suggestions fell below their expertise. These findings underscore the potential of gen AI to improve operational efficiency and service quality while highlighting the need for tailored deployment strategies to support workers with varying skill levels.

Keywords: generative AI, human-AI interaction, customer service operations, automation, e-commerce