Social Choice Ethics in Artificial Intelligence

AI & Society, vol. 35, no. 1 (March 2020), pages 165-176. DOI 10.1007/s00146-017-0760-1.

17 Pages Posted: 4 Oct 2017 Last revised: 23 Jun 2020

See all articles by Seth Baum

Seth Baum

Global Catastrophic Risk Institute

Date Written: October 2, 2017


A major approach to the ethics of artificial intelligence (AI) is to use social choice, in which the AI is designed to act according to the aggregate views of society. This is found in the AI ethics of “coherent extrapolated volition” and “bottom-up ethics”. This paper shows that the normative basis of AI social choice ethics is weak due to the fact that there is no one single aggregate ethical view of society. Instead, the design of social choice AI faces three sets of decisions: standing, concerning whose ethics views are included; measurement, concerning how their views are identified; and aggregation, concerning how individual views are combined to a single view that will guide AI behavior. These decisions must be made up front in the initial AI design — designers cannot “let the AI figure it out”. Each set of decisions poses difficult ethical dilemmas with major consequences for AI behavior, with some decision options yielding pathological or even catastrophic results. Furthermore, non-social choice ethics face similar issues, such as whether to count future generations or the AI itself. These issues can be more important than the question of whether or not to use social choice ethics. Attention should focus on these issues, not on social choice.

Keywords: artificial intelligence, ethics, social choice, bottom-up ethics, coherent extrapolated volition

JEL Classification: D7

Suggested Citation

Baum, Seth, Social Choice Ethics in Artificial Intelligence (October 2, 2017). AI & Society, vol. 35, no. 1 (March 2020), pages 165-176. DOI 10.1007/s00146-017-0760-1., Available at SSRN:

Seth Baum (Contact Author)

Global Catastrophic Risk Institute ( email )

Do you have negative results from your research you’d like to share?

Paper statistics

Abstract Views
PlumX Metrics