Fields of Gold: Generating Relevant and Credible Insights Via Web Scraping and APIs

66 Pages Posted: 7 Apr 2021

See all articles by Johannes Boegershausen

Johannes Boegershausen

Erasmus University Rotterdam (EUR)

Abhishek Borah

INSEAD

Hannes Datta

Tilburg University

Andrew T. Stephen

University of Oxford - Said Business School

Date Written: March 28, 2021

Abstract

Marketing researchers increasingly use web scraping and Application Programming Interfaces (APIs) to collect publicly available data from the internet. While guidance on the technicalities of collecting web data are abundant, much of the design decisions involved in collecting web data have remained largely neglected and undiscussed. A lack of awareness and understanding of these design decisions, both among authors and reviewers, threatens the credibility of research findings based on web data. To address these issues, this article develops a systematic workflow that guides researchers across the different stages of collecting web data. Throughout, the authors discuss how various design decisions affect the relevance and credibility of research findings. The workflow is accompanied by a comprehensive review of the use of web data in marketing research, identifying common themes of how web data has enriched past work. Finally, the authors highlight promising avenues for how future work might leverage web data to address important marketing questions and disseminate research findings.

Keywords: web scraping, application programming interface (APIs), field data, research credibility, reproducibility, replicability, generalizability, ecological value, research methods, workflow, open science, open data

JEL Classification: M30, M31, M10

Suggested Citation

Boegershausen, Johannes and Borah, Abhishek and Datta, Hannes and Stephen, Andrew T., Fields of Gold: Generating Relevant and Credible Insights Via Web Scraping and APIs (March 28, 2021). Available at SSRN: https://ssrn.com/abstract=3820666 or http://dx.doi.org/10.2139/ssrn.3820666

Johannes Boegershausen (Contact Author)

Erasmus University Rotterdam (EUR) ( email )

Burgemeester Oudlaan 50
3000 DR Rotterdam, Zuid-Holland 3062PA
Netherlands

Abhishek Borah

INSEAD ( email )

Boulevard de Constance
77305 Fontainebleau Cedex
France

Hannes Datta

Tilburg University ( email )

Tilburg, 5000 LE
Netherlands

Andrew T. Stephen

University of Oxford - Said Business School ( email )

Park End Street
Oxford, OX1 1HP
Great Britain

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
189
Abstract Views
671
rank
198,596
PlumX Metrics