Institutional Black Boxes Pose an Even Greater Risk than Algorithmic Ones in a Legal Context

Abstract

Black boxes in machine learning (ML) systems can be understood in at least two ways; in relation to (1) an algorithm, i.e., a decision rule, when that rule is impossible for a human to interpret, or (2) the secrecy of that rule (proprietary nature of it), due to business or economic factors. I call the first understanding "algorithmic black boxes" and the second "institutional black boxes". These two understandings are independent of each other, in particular, transparent algorithms can be part of systems that are institutional black boxes. I indicate that when it comes to the application of ML in public institutions applying the law (e.g., courts), institutional black boxes pose a particular threat to the integrity and reliability of ML systems used in such a context. I argue that in the eXplainable Artificial Intelligence trend, more attention should be paid not only to the favourable features of the algorithm (e.g., direct interpretability) but also to the business context in which the ML system is developed. Its secrecy can sabotage the transparency of even the simplest models.

Keywords: AI & Law, black box, transparency, explainability, COMPAS, eXplainable Artificial Intelligence, XAI