Contextual Learning with Online Convex Optimization: Theory and Application to Medical Decision-Making

Abstract

Optimizing the treatment regimen is a fundamental medical decision-making problem. This can be thought of as a two-dimensional decision-making problem with a nested structure, because it involves determining both the optimal medication and its optimal dose. Identifying the most effective medication for an individual often poses considerable difficulty, and even when a suitable medication is ascertained, dosing it optimally remains a significant challenge. Making these two nested decisions necessitates the adaptive learning of a personalized disease progression control model. To address this problem, we propose a novel contextual multi-armed bandit model under a two-dimensional control with a nested structure. For this model, we develop a new joint contextual learning and optimization algorithm, termed the stochastic sub-gradient descent atop contextual bandit algorithm (SIENNA). It sequentially selects for a patient: (i) the best medication based on their contextual information, and (ii) the corresponding dose optimized over the prior history of those patients who received the same medication. We prove that it admits a sub-linear regret, which is tight up to a logarithmic factor. Our regret analysis leverages the strengths of both contextual bandit approaches and online convex optimization techniques in a seamless fashion. We substantiate the practicality of SIENNA using clinical data on patients with hypertension and heightened cardiovascular risks. Our analysis indicates that SIENNA has the potential to surpass current practices. We benchmark several policies to show the advantages of our approach and offer critical insights. Our framework holds promise for various applications beyond healthcare that require nested decision-making.

Keywords: online learning algorithms, regret analysis, contextual multi-armed bandit, stochastic sub-gradient descent, online convex optimization, personalized medicine, medical decision-making