Black Box Analytics and Ethical Decision Making

Abstract

Using an experiment with participants having management experience, we examine sales forecast decisions when using an opaque versus transparent data analytics system. Participants have private information suggesting that the forecast significantly underestimates sales, making the forecast – and their bonus – easily achievable unless adjusted. We explore the extent to which participants act less ethically by not adjusting the sales forecast upwards. We employ a 2 x 2 between-subjects design, manipulating the description of the forecasting system as opaque or transparent, and measuring feelings of responsibility in the presence (absence) of a prompt before making the adjustment decision. We find that participants make less ethical decisions when the system is opaque than when it is transparent, but feelings of responsibility overcome this problem. We also find that the least ethical participants use both more rationalizations and more self-interested reasons than those whose decisions are not as unethical, supporting the use of both economic and psychology theory when studying ethical decision-making. Our results suggest that organizations should attempt to make data analytics systems more transparent to decision-making users. However, when they cannot, they should ensure that decision-makers feel responsible for their decisions; for example, with a prompt or decision aid.

Keywords: ethical decision making, IT, machine learning, moral disengagement theory, accountability

JEL Classification: M41