Transferable Offline Reinforcement Learning: Tackling Unmeasured Confounding and Domain Shift

Offline Reinforcement Learning (Offline RL) aims to learn decision-making policies from historical data. However, in real-world deployment, the environment often differs from the domain where data was collected, and recorded variables may omit confounders that simultaneously affect actions and outcomes. This paper focuses on the joint challenge of domain shift and unmeasured confounding, proposing a causally adjusted transferability framework that leverages complementary information from multiple heterogeneous source datasets to identify causal quantities in the target domain.

Problem Background: When Offline Data Is Imperfect

Traditional offline RL assumes that data comes from the target domain and that all confounders are observable, but these two assumptions are often violated in real-world scenarios. For example, in medical decision-making, data distributions differ across hospitals, and confounding factors such as patients' socioeconomic status are not recorded. The paper addresses both problems simultaneously:

• Domain shift: Source data and target deployment environment have different distributions
• Unmeasured confounding: Historical actions and outcomes are influenced by hidden variables

Multiple heterogeneous source datasets may provide complementary information, but how to integrate them to identify policies for the target domain remains an open problem.

Confounders are third variables that simultaneously affect the treatment variable (here, actions) and the outcome variable. In causal inference, if confounders are not observed and adjusted for, estimates of action effects from observational data will be biased. For instance, in medical decision-making, doctors prescribe more aggressive treatments to severely ill patients, causing the observed association between treatment and outcome to deviate from the true causal effect. In offline reinforcement learning, unmeasured confounding makes behavior policies correlated with outcomes, and directly using recorded data for policy evaluation or optimization introduces bias, necessitating auxiliary structures such as instrumental variables, negative controls, or proxy variables for causal identification.

Core Framework: Causally Adjusted Transferability

The authors combine transferability theory with instrumental variables, negative controls, and proxy variables to build a unified framework for identifying causal quantities in the target domain. For confounded contextual bandits, the identification problem is transformed into a system of integral equations involving bridge functions, enabling minimax estimation under finite samples. Based on confidence sets of the identified quantities, the authors propose a pessimistic policy optimization algorithm, CAT-CCB.

The key insight is that even if a single source dataset cannot identify causal effects, the complementary structure of multiple source datasets may still satisfy identification conditions.

Instrumental variables, negative controls, and proxy variables are three commonly used causal identification tools. An instrumental variable isolates confounding through a variable that affects the action but does not directly affect the outcome (except through the action); a negative control is a variable known to have no causal effect on the outcome, used to detect and correct residual confounding; proxy variables indirectly control for unmeasured confounders through two observed variables respectively related to the action or outcome. Combining these tools with transferability theory allows establishing identification equations for causal quantities in the target domain when source and target distributions differ and hidden confounding exists.

Bridge functions are key functions in causal inference that connect observed distributions with latent causal quantities. Under unmeasured confounding, bridge functions satisfy integral equations that link observable conditional distributions with unobservable confounding distributions. By solving the system of integral equations for bridge functions, the identification problem can be transformed into a finite-dimensional parameter estimation problem, enabling the construction of minimax estimators. In confounded contextual bandits, the bridge function approach allows identifying conditional expected rewards of actions from observational data alone, providing a statistical foundation for policy optimization in the target domain.

Extension to Sequential Decision-Making: CAT-CMDP

The framework is further extended to episodic confounded Markov decision processes (MDPs). Under structural and completeness conditions, complementary source data can identify the reward and transition mechanisms of the target domain. Under Gaussian dynamics and linear function approximation, the algorithm CAT-CMDP is obtained.

On the theoretical side, the authors establish high-probability finite-sample suboptimality guarantees, quantifying the impact of approximation error, estimation error, source-to-target shift, offline coverage, and sample size. For bounded finite-dimensional linear classes, the guarantee for contextual bandits recovers the standard $\widetilde{\mathcal O}(N^{-1/2})$ rate.

Episodic confounded Markov decision processes are extensions of standard MDPs where state transitions and rewards depend not only on the current state and action but are also affected by unobserved confounding variables, with each episode being independent. In this setting, both transition kernels and reward functions involve hidden confounding, and direct estimation from offline data introduces bias. Identifying the reward and transition mechanisms of the target domain requires additional structural and completeness conditions; the former constrains how confounding operates, while the latter ensures that source data can cover the relevant distributions of the target domain. Under Gaussian dynamics and linear function approximation assumptions, these conditions can be converted into computable moment equations, thereby supporting the estimation and optimization of CAT-CMDP.

Experiments and Implications

Numerical experiments validate the effectiveness of the framework in learning target-domain policies from heterogeneous confounded data. This work provides a principled approach for scenarios where no single source dataset is sufficient for causal identification and policy optimization.

Implications for practitioners: When faced with multi-source offline data, one should not simply merge or ignore confounding, but rather use the causally adjusted transferability framework to explicitly identify the required conditions and leverage pessimistic optimization to ensure policy robustness.

Note: This article is based on the paper abstract and does not cover implementation details or experimental numbers.

One-line Summary

Proposes a causally adjusted transferability framework that leverages multiple heterogeneous data sources to identify target-domain policies under domain shift and unmeasured confounding.

Paragraph Summary

The paper studies the challenge of coexisting domain shift and unmeasured confounding in offline reinforcement learning, proposing a causally adjusted transferability framework. The framework combines instrumental variables, negative controls, and proxy variables to leverage complementary information from multiple heterogeneous source datasets to identify causal quantities in the target domain. For confounded contextual bandits, it transforms the identification problem into a system of integral equations involving bridge functions and proposes a pessimistic policy optimization algorithm, CAT-CCB; it further extends to episodic confounded MDPs with CAT-CMDP and establishes finite-sample suboptimality guarantees. Experiments validate the effectiveness of learning target-domain policies from heterogeneous confounded data.