Reinforcement Learning for Deep-Space Communications: A Goal-Oriented Relay Scheduling and Flow Control Framework

Deep-space communications have long suffered from three persistent challenges: intermittent connectivity, extremely long signal propagation delays, and limited storage capacity at relay nodes. In such a harsh network environment, squeezing maximum value out of scarce transmission resources can make or break a mission. A recent study proposes using reinforcement learning to reimagine the packet management and flow control logic at relay nodes — enabling every deep-space communication link to be more "mission-aware."

The Core Challenges of Deep-Space Communications

Communication between Earth and the Moon or Mars cannot maintain stable links the way terrestrial networks do. Round-trip signal delays can span minutes to tens of minutes, and contact windows are scarce and brief. To cope with this intermittent nature, deep-space networks widely adopt the Bundle Protocol (BP), which relays data hop-by-hop between nodes via a store-carry-forward mechanism.

The problem is that relay nodes have extremely limited storage space and transmission opportunities. When multiple data sources — such as a lunar base station and a Mars rover — share the same relay buffer to send data back to Earth, the relay must decide: which packets get forwarded first, and which should be dropped. Traditional scheduling policies like FIFO (First In, First Out) process packets purely in arrival order, ignoring the actual mission value of the data and often creating service inequity across different sources.

The Bundle Protocol (BP) is the core protocol of Delay-Tolerant Networks (DTN), standardized as RFC 9171. Unlike TCP/IP, which assumes end-to-end connectivity, BP introduces a "bundle layer" below the application layer, encapsulating data into "bundles" that are relayed hop-by-hop. Each node fully stores a received bundle before forwarding it once the next link becomes available — the store-carry-forward paradigm. This design enables reliable transmission even during prolonged link outages, at the cost of end-to-end delays that can stretch to hours or even days. BP also includes built-in priority fields and a lifetime mechanism, but these simple static configurations struggle to adapt to complex multi-source, multi-objective deep-space mission scenarios — precisely the gap this study aims to address by replacing or augmenting them with reinforcement learning.

Measuring Information Freshness with "Excess Age"

One of the study's core innovations is the introduction of a new metric to quantify information freshness: excess age, defined as Δᵢᵉ(t) = [Δᵢ(t) − dᵢ(t)]⁺. Here, Δᵢ(t) denotes the Age of Information (AoI) for data source i — the elapsed time since the data was generated — while dᵢ(t) is a source-specific delay baseline.

The elegance of this design lies in the fact that it does not penalize all delays indiscriminately. Instead, it only imposes a cost on the portion of delay that exceeds an acceptable baseline. Since different missions have vastly different tolerances for data timeliness — Mars scientific data may be far more tolerant of staleness than real-time lunar monitoring — this differentiated baseline more closely reflects real mission requirements.

Building on this metric, the researchers further define a goal-oriented loss function that supports nonlinear penalties tailored to individual data sources. In other words, the system can flexibly adjust priorities based on mission relevance, ensuring that truly critical data receives higher transmission priority.

Age of Information (AoI) is a network performance metric introduced around 2012 by Kaul et al. to measure how "stale" the information held by a receiver is relative to the current state of the source. Unlike traditional latency metrics, AoI not only grows during the time a packet spends traversing the network, but also accumulates continuously between updates — even when the network is idle. Formally, if the most recently successfully received update was generated at time uᵢ, then the AoI at time t is Δᵢ(t) = t − uᵢ. This property makes AoI particularly well-suited for evaluating applications sensitive to information freshness, such as status monitoring and telemetry. This study builds on AoI by introducing a delay baseline dᵢ(t) to construct excess age — effectively defining a customized "minimum acceptable freshness threshold" for each data source, penalizing only staleness that genuinely exceeds the tolerable range. This avoids over-optimizing for sources that inherently permit low-frequency updates.

Modeling Relay Control as a Constrained Semi-Markov Decision Process

On the algorithmic side, the research team formalizes the relay control problem as a Constrained Semi-Markov Decision Process (CSMDP) and solves it using masked Lagrangian Proximal Policy Optimization (masked Lagrangian PPO).

Two technical aspects are particularly noteworthy:

First, feasibility-aware action masking. This restricts the system to selecting only packets that can be fully transmitted within the remaining contact window, preventing precious window time from being wasted on packets that cannot possibly be delivered in time. This is especially critical in deep-space scenarios where contact windows are fleeting.

Second, Lagrangian methods for handling constraints. Deep-space relays must optimize mission utility while satisfying hard constraints such as fairness. Lagrangian relaxation allows these constraints to be naturally incorporated into the reinforcement learning objective, rather than being treated as hard limits that fragment the optimization process.

The overall framework achieves joint optimization of packet scheduling, drop decisions, and flow control at the relay, while explicitly accounting for fairness across heterogeneous data sources.

Semi-Markov Decision Processes (SMDP) generalize standard MDPs by allowing the time intervals between consecutive decisions to be random variables rather than fixed unit steps. In deep-space relay scenarios, events such as contact window openings and closings and packet arrivals occur asynchronously, making the intervals between decision points naturally irregular. SMDPs can model these continuous-time dynamics more faithfully. Proximal Policy Optimization (PPO) is one of the most widely used deep reinforcement learning algorithms today, ensuring training stability by constraining the magnitude of each policy update (the "proximal" constraint). This study augments PPO with a Lagrangian multiplier framework, converting fairness constraints and others into penalized terms with adaptive coefficients added to the objective function. The multipliers self-adjust during training to automatically balance constraint satisfaction against mission utility, eliminating the need for repeated manual tuning.

Simulation Results and Significance

The research team validated the framework on DTNSim (a Delay-Tolerant Network simulator). Experiments simulated lunar and Mars data updates being forwarded to Earth through a shared-buffer relay. Results showed that, compared to baseline scheduling policies such as FIFO, the proposed method achieved notable improvements in both age-based utility and inter-flow fairness.

This means the system not only delivers critical mission data to Earth more "freshly," but also prevents any single data source from monopolizing relay resources and starving other sources.

From a broader perspective, this research represents a step in the evolution of deep-space communications from "best-effort" toward "goal-driven" operation. As lunar and Mars missions grow increasingly dense, intelligent scheduling of relay resources will become a critical capability in deep-space network infrastructure. Combining reinforcement learning with the Bundle Protocol and Age of Information theory offers a practically viable technical pathway for resource management in future interplanetary communications.

It is worth noting that current results remain at the simulation stage, with significant engineering and robustness validation work still ahead before deployment in real deep-space missions. Nevertheless, the "goal-oriented loss + feasibility masking + constrained optimization" combination proposed here holds broad reference value for any resource-constrained, opportunity-scarce network environment.