Econometrics at Scale: Spark Up Big Data in Economics

49 Pages Posted: 18 Aug 2018 Last revised: 13 Feb 2020

See all articles by Benjamin Bluhm

Benjamin Bluhm

affiliation not provided to SSRN

Jannic Cutura

Goethe University Frankfurt, House of Finance (HoF), Graduate School of Economics, Finance and Management (GSEFM), Students

Date Written: February 6, 2020

Abstract

This paper provides an overview of how to use “big data” for economic research. We investigate the performance and ease of use of different Spark applications running on a distributed file system to enable the handling and analysis of data sets which were previously not usable due to their size. More specifically, we explain how to use Spark to (i) explore big data sets which exceed retail grade computers memory size and (ii) run typical econometric tasks including microeconometric, panel data and time series regression models which are prohibitively expensive to evaluate on stand-alone machines. By bridging the gap between the abstract concept of Spark and ready-to-use examples which can easily be altered to suite the researchers need, we provide economists and social scientists more generally with the theory and practice to handle the ever growing datasets available. The ease of reproducing the examples in this paper makes this guide a useful reference for researchers with a limited background in data handling and distributed computing.

Keywords: Time Series Econometrics, Distributed Computing, Apache Spark

JEL Classification: C53, C55

Suggested Citation

Bluhm, Benjamin and Cutura, Jannic, Econometrics at Scale: Spark Up Big Data in Economics (February 6, 2020). SAFE Working Paper No. 266, Available at SSRN: https://ssrn.com/abstract=3226976 or http://dx.doi.org/10.2139/ssrn.3226976

Benjamin Bluhm (Contact Author)

affiliation not provided to SSRN

Jannic Cutura

Goethe University Frankfurt, House of Finance (HoF), Graduate School of Economics, Finance and Management (GSEFM), Students ( email )

Grüneburgplatz 1
Frankfurt
Germany

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
548
Abstract Views
2,435
Rank
100,349
PlumX Metrics