Offline Statistical Reinforcement Learning with Applications in Precision Health

Precision medicine seeks to tailor medical treatment to the individual characteristics of each patient to achieve the goal of better patient outcomes. As a broader conceptualization that includes precision medicine, precision health involves approaches that everyone can do on their own to protect their health as well as steps that public health can take.

Reinforcement Learning (RL) is a powerful technique that allows an agent to learn and take actions in a given environment in order to maximize the cumulative reward that the agent receives. The interest in developing new statistical RL methods for precision health is emerging. The potential impacts of this work can be summarized in the following four goals.

First, the project contributes to both the fields of semiparametric inference and RL. The theoretical results include non-asymptotic distribution, risk bounds with novel empirical process technical tools. These results will be fundamentally important and generally applicable for studying semiparametric inference empowered by RL.

Second, the clinical findings based on analyzing the electronic medical record (EMR) data will lead to major progress in addressing important clinical questions on the treatment recommendations for patients. Third, although EMR and mobile health (mHealth) data are the main applications of this project, the developed methods are general enough to apply to a variety of data sources including clinical data and economic data.

The developed methods are expected to greatly enhance the acquisition and analysis of large-scale data with population heterogeneity for medical, scientific and engineering communities. Fourth, the integration of research and education is a key aspect of this project. The PI will develop new courses and improve existing courses on RL and semiparametric inference, will train graduate students, and will reach out to the K-12 education levels by training high school teachers and students.

Despite the tremendous impacts that RL has achieved in areas such as games and robots, a direct deployment of RL algorithms in precision health can be costly, risky or even infeasible, due to significant real-world challenges. The main objective of this proposal is to develop new statistical offline RL methods to handle real-world challenges by developing flexible and efficient off-policy learning and robust and efficient off-policy evaluation methods.

The off-policy learning in RL refers to the problem of finding the best target policy that maximizes the value, given samples collected from a possibly different policy. In Aim 1, we will develop an efficient advantage learning framework in order to efficiently use pre-collected data for policy optimization. In Aim 2, we consider the problem of off-policy evaluation where the objective is to learn the value under a target policy with data collected under a possibly different policy.

There is a growing literature on estimating the value under a given policy in off-policy settings. However, very limited work have been considered regarding statistical inference such as hypothesis testing and confidence intervals (CIs) of the value, which is the focus of this aim. In Aim 3, we will discuss the plan of analyzing EMR and mHealth data with policy learning and policy evaluation based on the proposed methods in Aims 1 and 2.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.