DFQI: Direct Knowledge Transfer Boosts Offline Q-Learning Efficiency

Offline reinforcement learning (Offline RL) holds great potential in data-driven decision-making scenarios such as marketing and healthcare. However, when target-environment data is scarce, how to leverage data from related environments becomes a key challenge. A recent paper proposes Direct Transfer Fitted Q-Iteration (DFQI), which directly incorporates source-environment information into target value function learning through a joint estimation framework, offering a new approach to cross-environment knowledge transfer.

The Knowledge Transfer Challenge in Offline RL

Offline RL aims to learn policies using only fixed datasets, avoiding the risks of online exploration. In practice, data from the environment of interest (e.g., a new market) is often limited, while related environments (e.g., mature markets) may have accumulated large amounts of data. Directly mixing data introduces bias because reward functions and transition dynamics differ across environments. Therefore, knowledge transfer must explicitly model and correct for these differences.

The paper points out that existing transfer methods either rely on two-stage estimation or fail to jointly account for differences in rewards and dynamics, limiting transfer effectiveness. The core innovation of DFQI is that it performs target Q-function updates and source information borrowing simultaneously within the Bellman update, achieving tighter coupling.

Fitted Q-Iteration (FQI) is a classic algorithmic framework in offline reinforcement learning, belonging to the family of batch RL methods. It does not rely on online interaction; instead, it repeatedly performs Bellman backups using a fixed dataset: in each iteration, it computes target values for state-action pairs in the dataset using the current Q-function, then updates the Q-function via regression (i.e., "fitting") until convergence. FQI's advantages are simple implementation and high sample efficiency, but its performance depends heavily on data coverage and the quality of the function approximator. In cross-environment transfer scenarios, FQI's iterative structure provides a natural entry point for "injecting source-environment information at each Bellman update," and DFQI is built by modifying this framework.

The DFQI Method: L1 Penalty Linking Source and Target

DFQI adopts the Fitted Q-Iteration framework. In each Bellman update, the target Q-function is updated by minimizing the Bellman error, while an L1 penalty term is introduced to link the target coefficients to a weighted aggregation of source coefficients. Specifically, the target coefficients are pulled toward a weighted combination of source coefficients, with weights determined adaptively from data. This design allows more source information to be borrowed when target data is scarce, and reduces its influence when target data is abundant.

The L1 penalty also induces sparsity, helping identify which source information is most valuable for the target. Joint estimation ensures that source and target coefficients are solved within the same optimization problem, avoiding error accumulation. The paper uses sieve approximation to handle high-dimensional state spaces, laying the foundation for statistical analysis.

Sieve approximation is a technique in statistical learning theory for handling high-dimensional or nonparametric estimation. Its core idea is to approximate an infinite-dimensional target function space using a sequence of progressively growing finite-dimensional function spaces (called "sieves"). As sample size increases, sieve complexity grows accordingly, balancing bias and variance. In offline RL, state spaces may be continuous and high-dimensional, and directly using nonparametric methods leads to extremely slow convergence rates; sieve approximation controls the complexity of the function class (e.g., using polynomial bases, wavelet bases, or neural networks), endowing statistical estimators with analyzable convergence properties. DFQI uses sieve approximation to model the Q-function, laying the theoretical foundation for establishing statistical guarantees for target estimation and policy learning.

Statistical Guarantees and Theoretical Results

The authors establish statistical guarantees for target estimation and policy learning in stationary discounted environments, covering both homogeneous and heterogeneous transition kernels. Theoretical analysis shows that under certain regularity conditions, the convergence rates of DFQI's target Q-function estimation and policy value are comparable to estimators using only target data, while effectively leveraging source data to reduce variance. For heterogeneous transition kernels, DFQI automatically adjusts the contribution of source information through weighted aggregation, avoiding negative transfer.

These theoretical results provide rigorous support for the feasibility of transfer learning in offline RL and explain the role of the L1 penalty in the bias-variance tradeoff.

Negative transfer refers to the phenomenon where knowledge from the source environment actually harms learning performance in the target environment. In transfer learning, when reward functions or transition dynamics differ substantially between source and target environments, forcibly borrowing source data introduces bias, leading to suboptimal or even divergent target policies. DFQI uses an L1 penalty term to perform weighted aggregation of source coefficients, with weights determined adaptively from data: when target data is abundant or source information is irrelevant, the penalty term shrinks the contribution of source coefficients, automatically suppressing negative transfer. In the theoretical analysis, the convergence guarantee under heterogeneous transition kernels precisely characterizes the effectiveness of this adaptive mechanism, showing that DFQI can leverage source data to reduce variance while controlling bias caused by environment differences.

Empirical Results: Improvements with Limited Target Data

Empirical studies further validate DFQI's effectiveness. In multiple simulated environments, when target data is limited, DFQI significantly improves policy learning performance by transferring source information. Compared with baselines (e.g., using only target data, simple data mixing), DFQI demonstrates advantages in both policy return and stability. The paper also shows how the L1 penalty adaptively selects relevant source information, reducing interference from irrelevant sources.

One detail worth noting: the DFQI framework is general and can be extended to other offline RL algorithms based on value iteration. Future directions include handling non-stationary environments and more complex function approximation.

Conclusion

DFQI provides a direct, joint solution for knowledge transfer in offline RL. By linking source and target coefficients through an L1 penalty and estimating them simultaneously within Bellman updates, it comes with statistical guarantees. For data-scarce decision-making scenarios, this transfer approach is expected to improve learning efficiency and reduce the need for large amounts of target-environment data.

Paper link: https://arxiv.org/abs/2510.00104

One-line Summary

DFQI jointly estimates source and target Q-functions within Bellman updates via an L1 penalty, enabling cross-environment knowledge transfer in offline RL.

Paragraph Summary

This article introduces DFQI, a knowledge transfer method for offline reinforcement learning. Targeting scenarios where target-environment data is scarce while related-environment data is abundant, DFQI operates within the fitted Q-iteration framework, simultaneously updating the target Q-function at each Bellman update and introducing an L1 penalty term that pulls target coefficients toward a data-adaptive weighted aggregation of source coefficients. This design allows more source information to be borrowed when target data is insufficient and automatically reduces its influence when data is sufficient, while also inducing sparsity to identify valuable source information. The paper uses sieve approximation to handle high-dimensional state spaces and establishes statistical guarantees under stationary discounted environments for both homogeneous and heterogeneous transition kernels, showing that convergence rates are comparable to estimators using only target data and that negative transfer is avoided. Empirical results show that DFQI significantly improves policy return and stability when target data is limited, and the framework can be extended to other value-iteration-based offline RL algorithms.