Sound Like Me: Findings from a Randomized Experiment

7 Pages Posted: 7 Dec 2023

Date Written: November 29, 2023

Abstract

A new version of Copilot for Microsoft 365 includes a feature to let Outlook draft messages that “Sound Like Me” (SLM) based on training from messages in a user’s Sent Items folder. We sought to evaluate whether SLM lives up to its name. We find that it does, and more. Users widely and systematically praise SLM-generated messages as being more clear, more concise, and more “couldn’t have said it better myself”. When presented with a human-written message versus a SLM rewrite, users say they’d rather receive the SLM rewrite. All these findings are statistically significant. Furthermore, when presented with human and SLM messages, users struggle to tell the difference, in one specification doing worse than random.

Keywords: Artificial intelligence, large language model, Turing test

JEL Classification: O32, L86, J24, C91

Suggested Citation

Edelman, Benjamin G. and Ngwe, Donald, Sound Like Me: Findings from a Randomized Experiment (November 29, 2023). Available at SSRN: https://ssrn.com/abstract=4648689 or http://dx.doi.org/10.2139/ssrn.4648689

Benjamin G. Edelman (Contact Author)

Microsoft Corporation ( email )

One Microsoft Way
Redmond, WA 98052
United States

HOME PAGE: http://www.benedelman.org/

Donald Ngwe

Microsoft Corporation ( email )

One Microsoft Way
Redmond, WA 98052
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
150
Abstract Views
515
Rank
363,202
PlumX Metrics