A Partitioning Algorithm for Markov Decision Processes with Applications to Market Microstructure

Forthcoming in Management Science

44 Pages Posted: 29 Nov 2013 Last revised: 21 Feb 2017

See all articles by Ningyuan Chen

Ningyuan Chen

University of Toronto at Mississauga - Department of Management; University of Toronto - Rotman School of Management

Steven Kou

Boston University

Chun Wang

Columbia University - Department of Industrial Engineering and Operations Research (IEOR)

Date Written: September 20, 2015

Abstract

We propose a partitioning algorithm to solve a class of linear-quadratic Markov decision processes with inequality constraints and non-convex stage-wise cost; within each region of the partitioned state space, the value function and the optimal policy have analytical quadratic and linear forms, respectively. Compared to grid-based numerical schemes, the partitioning algorithm gives the closed-form solution without discretization error, and in many cases does not suffer from the curse of dimensionality. The algorithm is applied to two applications. In the main application, we present a model for limit order books with stochastic market depth to study the optimal order execution problem; stochastic market depth is consistent with empirical studies and necessary to accommodate various order activities. The optimal execution policy obtained by the algorithm significantly outperforms that of a deterministic market depth model in numerical examples. In the second application, we use the algorithm to compute the exact optimal solution to the renewable electricity management problem, for which previously only an approximate solution is known. As a comparison, we show that the approximate solution can be quite inaccurate for some initial states and thus demonstrate an advantage of the exact solution.

Keywords: Markov chains, Large order execution, Electricity trading/production, Partitioning, Quadratic stochastic programming

JEL Classification: C61, D49, G10, G20

Suggested Citation

Chen, Ningyuan and Kou, Steven and Wang, Chun, A Partitioning Algorithm for Markov Decision Processes with Applications to Market Microstructure (September 20, 2015). Forthcoming in Management Science. Available at SSRN: https://ssrn.com/abstract=2360552 or http://dx.doi.org/10.2139/ssrn.2360552

Ningyuan Chen

University of Toronto at Mississauga - Department of Management ( email )


Canada

University of Toronto - Rotman School of Management ( email )

105 St. George st
Toronto, ON M5S 3E6
Canada

Steven Kou (Contact Author)

Boston University ( email )

595 Commonwealth Avenue
Boston, MA 02215
United States
6173583318 (Phone)

Chun Wang

Columbia University - Department of Industrial Engineering and Operations Research (IEOR) ( email )

331 S.W. Mudd Building
500 West 120th Street
New York, NY 10027
United States

Register to save articles to
your library

Register

Paper statistics

Downloads
482
Abstract Views
2,448
rank
58,099
PlumX Metrics