Robust Risk-Aware Reinforcement Learning

13 Pages Posted: 27 Aug 2021 Last revised: 15 Dec 2021

See all articles by Sebastian Jaimungal

Sebastian Jaimungal

University of Toronto - Department of Statistics

Silvana M. Pesenti

University of Toronto

Ye Sheng Wang

University of Toronto

Hariom Tatsat

affiliation not provided to SSRN

Date Written: December 14, 2021

Abstract

We present a reinforcement learning (RL) approach for robust optimisation of risk-aware performance criteria. To allow agents to express a wide variety of risk-reward profiles, we assess the value of a policy using rank dependent expected utility (RDEU). RDEU allows the agent to seek gains, while simultaneously protecting themselves against downside events. To robustify optimal policies against model uncertainty, we assess a policy not by its distribution, but rather, by the worst possible distribution that lies within a Wasserstein ball around it. Thus, our problem formulation may be viewed as an actor choosing a policy (the outer problem), and the adversary then acting to worsen the performance of that strategy (the inner problem). We develop explicit policy gradient formulae for the inner and outer problems, and show its efficacy on three prototypical financial problems: robust portfolio allocation, optimising a benchmark, and statistical arbitrage.

Keywords: Robust Optimisation, Reinforcement Learning, Risk Measures, Wasserstein Distance, Statistical Arbitrage, Portfolio Optimisation

JEL Classification: C61, G11,C63, C15, C44

Suggested Citation

Jaimungal, Sebastian and Pesenti, Silvana M. and Wang, Ye Sheng and Tatsat, Hariom, Robust Risk-Aware Reinforcement Learning (December 14, 2021). Available at SSRN: https://ssrn.com/abstract=3910498 or http://dx.doi.org/10.2139/ssrn.3910498

Sebastian Jaimungal

University of Toronto - Department of Statistics ( email )

100 St. George St.
Toronto, Ontario M5S 3G3
Canada

HOME PAGE: http://http:/sebastian.statistics.utoronto.ca

Silvana M. Pesenti (Contact Author)

University of Toronto ( email )

100 St. George Street
Toronto, Ontario M5S 3G8
Canada

Ye Sheng Wang

University of Toronto ( email )

Toronto, Ontario M5S 3G8
Canada

Hariom Tatsat

affiliation not provided to SSRN

Do you have a job opening that you would like to promote on SSRN?

Paper statistics

Downloads
102
Abstract Views
295
rank
349,895
PlumX Metrics