Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Games

48 Pages Posted: 2 Aug 2021 Last revised: 21 Jul 2022

See all articles by Ben M. Hambly

Ben M. Hambly

University of Oxford - St. Ann's College

Renyuan Xu

University of Southern California - Epstein Department of Industrial & Systems Engineering

Huining Yang

University of Oxford - Mathematical Institute

Date Written: July 27, 2021

Abstract

We consider a general-sum N-player linear-quadratic game with stochastic dynamics over a finite horizon and prove the global convergence of the natural policy gradient method to the Nash equilibrium. In order to prove convergence of the method we require a certain amount of noise in the system. We give a condition, essentially a lower bound on the covariance of the noise in terms of the model parameters, in order to guarantee convergence. We illustrate our results with numerical experiments to show that even in situations where the policy gradient method may not converge in the deterministic setting, the addition of noise leads to convergence.

Keywords: Multi-agent reinforcement learning, linear-quadratic games, policy gradient methods, general-sum games, N-player games

JEL Classification: C70, C72, D80, D82, D83

Suggested Citation

Hambly, Ben M. and Xu, Renyuan and Yang, Huining, Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Games (July 27, 2021). Available at SSRN: https://ssrn.com/abstract=3894471 or http://dx.doi.org/10.2139/ssrn.3894471

Ben M. Hambly

University of Oxford - St. Ann's College ( email )

Woodstock Road
Oxford OX2 6HS
United Kingdom
+44 1865 274800 (Phone)
+44 1865 274899 (Fax)

Renyuan Xu

University of Southern California - Epstein Department of Industrial & Systems Engineering ( email )

United States

HOME PAGE: http://renyuanxu.github.io

Huining Yang (Contact Author)

University of Oxford - Mathematical Institute ( email )

Radcliffe Observatory, Andrew Wiles Building
Woodstock Rd
Oxford, Oxfordshire OX2 6GG
United Kingdom

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
169
Abstract Views
628
rank
247,529
PlumX Metrics