Measuring Mobile Broadband Performance

20 Pages Posted: 16 Mar 2018 Last revised: 15 Aug 2018

See all articles by Steven Bauer

Steven Bauer

Massachusetts Institute of Technology (MIT) - Laboratory for Computer Science (LCS)

William Lehr

Massachusetts Institute of Technology (MIT) - Computer Science and Artificial Intelligence Laboratory (CSAIL)

Date Written: March 12, 2018

Abstract

The FCC has been running the Measuring Broadband America program since 2011. They have released six well-received yearly reports so far, analyzing the data collected from wired broadband providers. In 2012, the FCC announced an analogous Measuring Mobile Broadband Performance effort and they have been collecting data since 2014. However, the FCC has not released any reports analyzing the mobile data. In this paper, we investigate the technical and policy challenges confronting this and other efforts to collect and report on mobile broadband performance. At the core of the debate over mobile performance measurement are questions such as: what constitutes scientifically valid performance measurement for mobile broadband networks? How does one draw valid inferences about user experience, correctly attributing impairments when they exist to different actors or components in the system? Or, how can mobile performance measurement reporting protect privacy while still remaining informative? Answering these questions is more difficult for mobile than fixed broadband for multiple reasons that we document. We explore the current status of mobile broadband performance measurement, building on earlier work that focused on measuring the performance of fixed broadband services. Traditionally, drive and walk testing of mobile networks (conducted by companies such as RootMeterics and P3 Communications) have competed with crowd-sourced measurements (like Ookla’s Speedtest app and the FCC’s app). For each approach, we examine the measurement methodology, analysis procedures, and data availability. We find that while information describing the general structure of the measurement and reporting methodologies is typically available, important details regarding the measurement methodology and how the raw data is modified to produce the final reports is often missing. We dispute published claims that driving testing is "scientific," whereas other methodologies are suspect. We conclude that there is no single best approach and that the availability of a multiplicity of measurement platforms and approaches offers significant benefits. Although there is no single best mobile measurement approach for all questions, we explain why it is important to carefully match the test methodology and available data to the questions being asked. Finally, our analysis explores the implications for broadband policy of this evolving ecosystem for mobile broadband measurement.

Keywords: broadband, measurement, policy

Suggested Citation

Bauer, Steven and Lehr, William, Measuring Mobile Broadband Performance (March 12, 2018). TPRC 46: The 46th Research Conference on Communication, Information and Internet Policy 2018. Available at SSRN: https://ssrn.com/abstract=3138610

Steven Bauer (Contact Author)

Massachusetts Institute of Technology (MIT) - Laboratory for Computer Science (LCS) ( email )

United States

William Lehr

Massachusetts Institute of Technology (MIT) - Computer Science and Artificial Intelligence Laboratory (CSAIL) ( email )

Stata Center
Cambridge, MA 02142
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
42
Abstract Views
100
PlumX Metrics