Randomized Controlled Trials for Microsoft Copilot for Security

10 Pages Posted: 2 Dec 2023 Last revised: 8 Apr 2024

See all articles by Benjamin G. Edelman

Benjamin G. Edelman

Microsoft Corporation

James Bono

Microsoft Corporation

Sida Peng

Microsoft Corporation

Roberto Rodriguez

Microsoft Corporation

Sandra Ho

Microsoft Corporation

Date Written: March 27, 2024

Abstract

We conducted randomized controlled trials (RCTs) to measure the efficiency gains from using Copilot for Security, including speed and quality improvements. External experimental subjects logged into a M365 Defender instance created for this experiment and performed four tasks: Incident Summarization, Script Analyzer, Incident Report, and Guided Response. We found that Copilot for Security delivered large improvements on both speed and accuracy. Copilot brought improvements for both novices and security professionals.

Keywords: security, AI, RCT

JEL Classification: O32, L86, J24, C93

Suggested Citation

Edelman, Benjamin G. and Bono, James and Peng, Sida and Rodriguez, Roberto and Ho, Sandra, Randomized Controlled Trials for Microsoft Copilot for Security (March 27, 2024). Available at SSRN: https://ssrn.com/abstract=4648700 or http://dx.doi.org/10.2139/ssrn.4648700

Benjamin G. Edelman (Contact Author)

Microsoft Corporation ( email )

One Microsoft Way
Redmond, WA 98052
United States

HOME PAGE: http://www.benedelman.org/

James Bono

Microsoft Corporation ( email )

One Microsoft Way
Redmond, WA 98052
United States

Sida Peng

Microsoft Corporation ( email )

One Microsoft Way
Redmond, WA 98052
United States

Roberto Rodriguez

Microsoft Corporation ( email )

One Microsoft Way
Redmond, WA 98052
United States

Sandra Ho

Microsoft Corporation ( email )

One Microsoft Way
Redmond, WA 98052
United States

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
901
Abstract Views
3,686
Rank
50,720
PlumX Metrics