Ambiguous Partially Observable Markov Decision Processes: Structural Results and Applications

35 Pages Posted: 12 Oct 2014 Last revised: 18 Aug 2018

See all articles by Soroush Saghafian

Soroush Saghafian

Harvard University - Harvard Kennedy School (HKS)

Date Written: August 16, 2018

Abstract

Markov Decision Processes (MDPs) have been widely used as invaluable tools in dynamic decision-making, which is a central concern for economic agents operating at both the micro and macro levels. Often the decision maker's information about the state is incomplete; hence, the generalization to Partially Observable MDPs (POMDPs). Unfortunately, POMDPs may require a large state and/or action space, creating the well-known "curse of dimensionality." However, recent computational contributions and blindingly fast computers have helped to dispel this curse. This paper introduces and addresses a second curse termed "curse of ambiguity," which refers to the fact that the exact transition probabilities are often hard to quantify, and are rather ambiguous. For instance, for a monetary authority concerned with dynamically setting the inflation rate so as to control the unemployment, the dynamics of unemployment rate under any given inflation rate is often ambiguous. Similarly, in worker-job matching, the dynamics of worker-job match/proficiency level is typically ambiguous. This paper addresses the "curse of ambiguity" by developing a generalization of POMDPs termed Ambiguous POMDPs (APOMDPs), which not only allows the decision maker to take into account imperfect state information, but also tackles the inevitable ambiguity with respect to the correct probabilistic model of transitions. Importantly, this paper extends various structural results from POMDPs to APOMDPs. These results enable the decision maker to make robust decisions. Robustness is achieved by using a-maximin expected utility (a-MEU), which (a) differentiates between ambiguity and ambiguity attitude, (b) avoids the over conservativeness of traditional maximin approaches, and (c) is found to be suitable in laboratory experiments in various choice behaviors including those in portfolio selection. The structural results provided also help to handle the "curse of dimensionality," since they significantly simplify the search for an optimal policy. The analysis identifies a performance guarantee for the proposed approach by developing a bound for its maximum reward loss due to model ambiguity.

Suggested Citation

Saghafian, Soroush, Ambiguous Partially Observable Markov Decision Processes: Structural Results and Applications (August 16, 2018). Journal of Economic Theory, Forthcoming, Available at SSRN: https://ssrn.com/abstract=2508776 or http://dx.doi.org/10.2139/ssrn.2508776

Soroush Saghafian (Contact Author)

Harvard University - Harvard Kennedy School (HKS) ( email )

79 John F. Kennedy Street
Cambridge, MA 02138
United States

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
179
Abstract Views
1,021
Rank
305,000
PlumX Metrics