Understanding frontier AI capabilities and risks through semi-structured interviews

Abstract

We describe how semi-structured interviews with frontier AI developers could help governments better understand AI progress and AI-related national security threats. Thus far, governments have largely relied on formal model evaluations to monitor frontier AI capabilities. While such approaches are valuable, they have several limitations that could be complemented with semi-structured, qualitative interviews. Such interviews could provide insights into areas formal evaluations may miss, such as developers' intuitive understanding of AI capabilities, early warning signs of extreme risks, predictions about future AI progress, and information about internal safety practices. We outline an approach in which government officials would have regular (e.g., monthly or quarterly) interviews with employees from the top 5-10 frontier AI developers. We also outline example questions that officials could ask in four categories: (a) AI capabilities and AI progress, (b) National security, (c) Safety culture, and (d) Miscellaneous. Conducting such interviews-either through mandatory reporting requirements or voluntary agreements-would enable governments to better understand and prepare for the national security implications of frontier AI systems.

Keywords: artificial intelligence, national security, AI policy