Explaining the Algorithm Does Not Explain the Decision: Unpacking Accountabilities in Organisational Decision Making

Abstract

An organisational decision-making process has many component parts (with or without the involvement of a computer-based algorithm). Many technical discussions such as on transparency and explainability of algorithm-supported decisions omit many of them and thus address issues about accountability, liability and explanations for decisions in too narrow a sense. There are many choices made in the construction and operation of an organisational decision-making process, particularly if an algorithm-based model is used as part of it. This may lead to a chain of accountabilities of persons who may be required to explain or justify choices made at any point.

This paper unpacks the general architecture of organisational decision making and examines the location and role of one or more algorithmic components that may feature within it. It will identify the design choices involved in constructing a decision-making process and the corresponding responsibilities and accountabilities. Within those accountabilities, the differences between functional reasons, explanations and justifications will be explored together with the actors who may be responsible for providing them. A case study of a public sector algorithmic decision-making system illustrates how the architecture helps unpack the key issues to interrogate. Crucially, the architecture makes a clear distinction between the generation of a prediction by an algorithmic process and the execution of an organisation’s decision-making policy. In “automated decision-making”, it is the execution of an organisation’s decision-making policy that is automated. A computerised algorithm may or may not provide input to it.

Keywords: decision architecture, organisational decision making, decision policy, algorithmic decision making, artificial intelligence, decision model, explanability, explanation, justification, XAI, automated decision making, accountability.