Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality

58 Pages Posted: 18 Sep 2023 Last revised: 27 Sep 2023

See all articles by Fabrizio Dell'Acqua

Fabrizio Dell'Acqua

Harvard University - Business School (HBS)

Edward McFowland

Harvard University - Business School (HBS)

Ethan R. Mollick

University of Pennsylvania - Wharton School

Hila Lifshitz-Assaf

Harvard University Lab for Innovation Sciences; Harvard LISH, Lab for Innovation Sciences; University of Warwick, Warwick Business School

Katherine Kellogg

Massachusetts Institute of Technology (MIT) - Sloan School of Management

Saran Rajendran

Boston Consulting Group, Henderson Institute

Lisa Krayer

Boston Consulting Group, Henderson Institute

François Candelon

Boston Consulting Group, Henderson Institute

Karim R. Lakhani

Harvard Business School - Technology and Operations Management Group; Harvard Institute for Quantitative Social Science; Harvard University - Berkman Klein Center for Internet & Society

Date Written: September 15, 2023

Abstract

The public release of Large Language Models (LLMs) has sparked tremendous interest in how humans will use Artificial Intelligence (AI) to accomplish a variety of tasks. In our study conducted with Boston Consulting Group, a global management consulting firm, we examine the performance implications of AI on realistic, complex, and knowledge-intensive tasks. The pre-registered experiment involved 758 consultants comprising about 7% of the individual contributor-level consultants at the company. After establishing a performance baseline on a similar task, subjects were randomly assigned to one of three conditions: no AI access, GPT-4 AI access, or GPT-4 AI access with a prompt engineering overview. We suggest that the capabilities of AI create a “jagged technological frontier” where some tasks are easily done by AI, while others, though seemingly similar in difficulty level, are outside the current capability of AI. For each one of a set of 18 realistic consulting tasks within the frontier of AI capabilities, consultants using AI were significantly more productive (they completed 12.2% more tasks on average, and completed task 25.1% more quickly), and produced significantly higher quality results (more than 40% higher quality compared to a control group). Consultants across the skills distribution benefited significantly from having AI augmentation, with those below the average performance threshold increasing by 43% and those above increasing by 17% compared to their own scores. For a task selected to be outside the frontier, however, consultants using AI were 19 percentage points less likely to produce correct solutions compared to those without AI. Further, our analysis shows the emergence of two distinctive patterns of successful AI use by humans along a spectrum of human-AI integration. One set of consultants acted as “Centaurs,” like the mythical halfhorse/half-human creature, dividing and delegating their solution-creation activities to the AI or to themselves. Another set of consultants acted more like “Cyborgs,” completely integrating their task flow with the AI and continually interacting with the technology.

Suggested Citation

Dell'Acqua, Fabrizio and McFowland, Edward and Mollick, Ethan R. and Lifshitz-Assaf, Hila and Kellogg, Katherine and Rajendran, Saran and Krayer, Lisa and Candelon, François and Lakhani, Karim R., Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality (September 15, 2023). Harvard Business School Technology & Operations Mgt. Unit Working Paper No. 24-013, Available at SSRN: https://ssrn.com/abstract=4573321 or http://dx.doi.org/10.2139/ssrn.4573321

Fabrizio Dell'Acqua (Contact Author)

Harvard University - Business School (HBS) ( email )

Boston, MA 02163
United States

Edward McFowland

Harvard University - Business School (HBS) ( email )

Boston, MA 02163
United States

Ethan R. Mollick

University of Pennsylvania - Wharton School ( email )

The Wharton School
Philadelphia, PA 19104-6370
United States

Hila Lifshitz-Assaf

Harvard University Lab for Innovation Sciences ( email )

Soldiers Field Road
Cotting House 321A
Boston, MA 02163
United States

Harvard LISH, Lab for Innovation Sciences ( email )

William James Hall, Sixth Floor
33 Kirkland Street
Cambridge, MA 02138

University of Warwick, Warwick Business School ( email )

West Midlands, CV4 7AL
United Kingdom

HOME PAGE: http://https://www.hilalifshitz.com/

Katherine Kellogg

Massachusetts Institute of Technology (MIT) - Sloan School of Management ( email )

100 Main Street
E62-416
Cambridge, MA 02142
United States

Saran Rajendran

Boston Consulting Group, Henderson Institute ( email )

Lisa Krayer

Boston Consulting Group, Henderson Institute ( email )

François Candelon

Boston Consulting Group, Henderson Institute ( email )

Karim R. Lakhani

Harvard Business School - Technology and Operations Management Group ( email )

Boston, MA 02163
United States
617-495-6741 (Phone)

Harvard Institute for Quantitative Social Science ( email )

1737 Cambridge St.
Cambridge, MA 02138
United States

Harvard University - Berkman Klein Center for Internet & Society ( email )

Harvard Law School
23 Everett, 2nd Floor
Cambridge, MA 02138
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
59,384
Abstract Views
152,780
Rank
38
PlumX Metrics