Monotonicity Properties for Two-Action Partially Observable Markov Decision Processes on Partially Ordered Spaces

18 Pages Posted: 11 Dec 2019

See all articles by Erik Miehling

Erik Miehling

University of Illinois at Urbana-Champaign - Coordinated Science Laboratory

Demosthenis Teneketzis

University of Michigan at Ann Arbor - Department of Electrical Engineering and Computer Science

Date Written: October 30, 2019

Abstract

This paper investigates monotonicity properties of optimal policies for two-action partially observable Markov decision processes when the underlying (core) state and observation spaces are partially ordered. Motivated by the desirable properties of the monotone likelihood ratio order in imperfect information settings, namely the preservation of belief ordering under conditioning on any new information, we propose a new stochastic order (a generalization of the monotone likelihood ratio order) that is appropriate for when the underlying space is partially ordered. The generalization is non-trivial, requiring one to impose additional conditions on the elements of the beliefs corresponding to incomparable pairs of states. The stricter conditions in the proposed stochastic order reflect a conservation of structure in the problem – the loss of structure from relaxing the total ordering of the state space to a partial order requires stronger conditions with respect to the ordering of beliefs. In addition to the proposed stochastic order, we introduce a class of matrices, termed generalized totally positive of order 2, that are sufficient for preserving the order. Our main result is a set of sufficient conditions that ensures existence of an optimal policy that is monotone on the belief space with respect to the proposed stochastic order.

Keywords: Dynamic programming, decision analysis, partially observable Markov decision processes, partially ordered sets

Suggested Citation

Miehling, Erik and Teneketzis, Demosthenis, Monotonicity Properties for Two-Action Partially Observable Markov Decision Processes on Partially Ordered Spaces (October 30, 2019). Available at SSRN: https://ssrn.com/abstract=3492364 or http://dx.doi.org/10.2139/ssrn.3492364

Erik Miehling (Contact Author)

University of Illinois at Urbana-Champaign - Coordinated Science Laboratory ( email )

1406 West Green Street
Urbana, IL 61801
United States

Demosthenis Teneketzis

University of Michigan at Ann Arbor - Department of Electrical Engineering and Computer Science ( email )

1101 Beal Avenue
Ann Arbor, MI 48109
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
17
Abstract Views
263
PlumX Metrics