Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon

49 Pages Posted: 9 Jan 2021 Last revised: 2 Aug 2021

See all articles by Ben M. Hambly

Ben M. Hambly

University of Oxford - St. Ann's College

Renyuan Xu

Mathematical Institute, University of Oxford

Huining Yang

University of Oxford - Mathematical Institute

Date Written: November 20, 2020

Abstract

We explore reinforcement learning methods for finding the optimal policy in the linear quadratic regulator (LQR) problem. In particular we consider the convergence of policy gradient methods in the setting of known and unknown parameters. We are able to produce a global linear convergence guarantee for this approach in the setting of finite time horizon and stochastic state dynamics under weak assumptions. The convergence of a projected policy gradient method is also established in order to handle problems with constraints. We illustrate the performance of the algorithm with two examples. The first example is the optimal liquidation of a holding in an asset. We show results for the case where we assume a model for the underlying dynamics and where we apply the method to the data directly. The empirical evidence suggests that the policy gradient method can learn the global optimal solution for a larger class of stochastic systems containing the LQR framework and that it is more robust with respect to model mis-specification when compared to a model-based approach. The second example is an LQR system in a higher dimensional setting with synthetic data.

Keywords: Linear-quadratic regulator, reinforcement learning, policy gradient method, stochastic control, optimal execution, optimal liquidation

JEL Classification: D80, D82, D83, G11

Suggested Citation

Hambly, Ben M. and Xu, Renyuan and Yang, Huining, Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon (November 20, 2020). Available at SSRN: https://ssrn.com/abstract=3734179 or http://dx.doi.org/10.2139/ssrn.3734179

Ben M. Hambly

University of Oxford - St. Ann's College ( email )

Woodstock Road
Oxford OX2 6HS
United Kingdom
+44 1865 274800 (Phone)
+44 1865 274899 (Fax)

Renyuan Xu

Mathematical Institute, University of Oxford ( email )

Andrew Wiles Building
Radcliffe Observatory Quarter (550)
Oxford, OX2 6GG
United Kingdom

Huining Yang (Contact Author)

University of Oxford - Mathematical Institute ( email )

Radcliffe Observatory, Andrew Wiles Building
Woodstock Rd
Oxford, Oxfordshire OX2 6GG
United Kingdom

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
201
Abstract Views
757
rank
188,132
PlumX Metrics