DFQI

The paper proposes Direct Transfer Fitted Q-Iteration (DFQI), which directly incorporates source-environment information into target value-function learning through a joint estimation framework.

In each Bellman update, DFQI simultaneously updates the target Q-function and borrows source information, achieving tighter coupling.

DFQI introduces an L1 penalty term in the fitted Q-iteration framework, pulling the target coefficients toward a weighted combination of the source coefficients, with weights determined adaptively from the data.

DFQI's L1 penalty induces sparsity, helping identify which source information is most valuable for the target.

DFQI uses sieve approximation to handle high-dimensional state spaces, laying the foundation for statistical property analysis.

The authors establish statistical guarantees for DFQI's target estimation and policy learning in a stationary discounted setting, covering both homogeneous and heterogeneous transition kernels.

Theoretical analysis shows that, under certain regularity conditions, the target Q-function estimation and policy value convergence rates of DFQI are comparable to estimators that use only target data, while it can effectively exploit source data to reduce variance.

For heterogeneous transition kernels, DFQI automatically adjusts the contribution of source information through weighted aggregation, avoiding negative transfer.

In multiple simulated environments, when target data are limited, DFQI significantly improves policy learning performance by transferring source information.

Compared with baselines (such as using only target data, simple data mixing, etc.), DFQI shows advantages in both policy return and stability.

The DFQI framework is general and can be extended to other offline RL algorithms based on value iteration.