Strategic Games and Algorithmic Secrecy

Abstract

We challenge a claim commonly made by industry and government representatives and echoed by some academics: that algorithmic decision-making processes are better kept opaque or secret because, otherwise, decision-subjects will “game the system”, leading to inaccurate or unfair results. We show that the range of situations in which people are able to game decision-making algorithms is narrow, even when there is substantial disclosure. We then analyze how to identify when gaming is possible in light of (i) how tightly the decision-making proxies are tied to the factors that would ideally determine the outcome, (ii) how easily those proxies can be altered by decision-subjects, and (iii) whether such strategic alterations ultimately lead to mistaken decisions.

Keywords: algorithmic decision-making, gaming, algorithmic transparency, algorithmic secrecy, algorithmic opacity, algorithmic accountability, automated decision-making, credit scoring, criminal investigation, criminal procedure, employment law game theory & the law