Nailing Prediction: Experimental Evidence on the Value of Tools in Predictive Model Development

Abstract

Predictive model development is understudied despite its importance to modern businesses. Although prior discussions highlight advances in methods (along the dimensions of data, computing power, and algorithms) as the primary driver of model quality, the value of tools that implement those methods has been neglected. In a field experiment leveraging a predictive data science contest, we study the importance of tools by restricting access to software libraries for machine learning models. By only allowing access to these libraries in our control group, we find that teams with unrestricted access perform 30% better in log-loss error — a statistically and economically significant amount, equivalent to a 10-fold increase in the training data set size. We further find that teams with high general data-science skills are less affected by the intervention, while teams with high tool-specific skills significantly benefit from access to modeling libraries. Our findings are consistent with a mechanism we call ‘Tools-as-Skill,’ where tooling automates and abstracts some general data science skills but, in doing so, creates the need for new tool-specific skills.

Keywords: Economics of AI, Predictive Model Development, Skills, Tools