Reinforcement Learning Lands Falcon 9 Precisely on Droneship: 67 Million Simulated Training Runs

When SpaceX's Falcon 9 rocket lands vertically on a droneship, the sight always takes your breath away. Now, a developer has replicated this feat using reinforcement learning (RL) in a purely simulated environment, with incredible detail: rocket physics, ocean waves, wind, fuel slosh, and sensor noise are all simulated—without using any real flight data.

According to a Reddit user's share, this RL agent was trained over 59 million simulated flights and eventually learned to perform a "Hoverslam" landing on a tossing droneship—meaning the rocket cannot hover and must bring its velocity to zero at the exact moment of touchdown.

Training Configuration: PPO and 67 Randomized Parameters

The agent uses the PPO algorithm, and its policy network has only about 80,000 parameters, controlling 10 actuators: throttle, gimbal, grid fins, thrusters, landing legs, and ignition. In each attempt, 67 world parameters are randomized, including wind speed, wave height, and fuel slosh amplitude. This domain randomization is key to enabling the agent to adapt to real-world uncertainty.

The developer reports that training involved a total of 59 million flights: 34.7 million splashes, 10.8 million explosions, and 13.2 million successful landings. The success rate is about 22.4%—a remarkable figure given the difficulty of the task.

PPO (Proximal Policy Optimization) is a policy-gradient-based reinforcement learning algorithm proposed by OpenAI in 2017. Its core idea is to limit the magnitude of policy changes at each update, preventing training collapse due to overly large single updates, thereby balancing sample efficiency and training stability. PPO enforces this constraint by clipping the importance sampling ratio, without requiring complex second-order optimization; it is simple to implement and stable in performance, making it one of the most commonly used baseline algorithms for continuous control tasks (such as robotic manipulation and game AI). In this project, PPO's stability is crucial for coordinating 10 continuous actuators and learning a landing policy in a randomized environment.

Domain randomization is a technique that enables agents to adapt to real-world uncertainty during simulation. Its core idea is to randomly vary physical parameters in the simulation environment during training (such as mass, friction, wind force, and sensor noise), so that the agent encounters a wide variety of "worlds" and thus learns a policy robust to environmental changes. This way, when the policy is deployed in the real world, the agent can cope even if real parameters deviate from the simulation. The cost of domain randomization is increased training difficulty, because the policy needs to cover a broader state distribution, but it can significantly narrow the "sim-to-real" gap. Randomizing 67 parameters in this project is a typical application of this idea.

Landing Success Rate and Failure Modes

Although the overall success rate is not high, the agent performs stably under moderate conditions. The developer admits that it is still weak in strong wind and high wave environments and is working to improve it. Among failure modes, explosions account for 18.3% and splashes 58.8%, indicating that the agent still needs more precise control in the final phase.

Interestingly, all training is based on trial and error, without using any real flight data. This means the agent explores entirely from scratch, guided by a reward function to develop a landing policy.

Technical Details and Challenges

Hoverslam landing is extremely demanding: the rocket engine's thrust exceeds gravity, so it cannot hover like a helicopter; it must reduce velocity to zero at the instant of touchdown. This places stringent requirements on the control policy. The agent needs to coordinate throttle, grid fins, and thrusters to adjust attitude in real time in a dynamic environment.

Fuel slosh and sensor noise further increase the difficulty. In the simulation, fuel slosh affects the center of gravity, while sensor noise makes observations imperfect, forcing the policy to be robust.

Hoverslam (also known as "suicide burn" or "hover slam") is a landing method used in vertical rocket recovery. Because the rocket engine's thrust typically exceeds the rocket's own weight (thrust-to-weight ratio greater than 1), the rocket cannot hover or descend slowly in the air like a helicopter; once the engine ignites, the rocket must continuously decelerate and bring its velocity precisely to zero at the instant of touchdown. This requires extremely precise landing ignition timing, throttle depth, and attitude control—any deviation can lead to a hard landing or explosion. SpaceX's Falcon 9 landings use exactly this mode, which imposes extremely high demands on the robustness and precision of control algorithms and is a major challenge in reinforcement learning tasks.

Potential of Reinforcement Learning in Aerospace

This experiment demonstrates the potential of reinforcement learning in complex control tasks. Traditional rocket landing relies on precise physical models and manual tuning, whereas RL can autonomously discover policies through extensive simulation. As computing resources grow, it may in the future assist in the design of real rocket landing systems.

The developer says the next step will focus on improving the success rate under strong wind and high wave conditions. Community feedback has been positive, with many asking for technical details and looking forward to open-source code.

(The End)

One-line Summary

A developer trained an RL agent entirely in simulation to achieve Hoverslam landing of a Falcon 9 rocket on a droneship.

Paragraph Summary

A developer used reinforcement learning in a fully simulated environment to replicate SpaceX Falcon 9 vertical landing on a droneship. The agent uses the PPO algorithm with a policy network of only about 80,000 parameters, controlling 10 actuators including throttle, gimbal, and grid fins. During training, 67 world parameters were randomized (wind speed, wave height, fuel slosh, sensor noise, etc.), and after 59 million simulated flights, 13.2 million successful landings were achieved—a success rate of about 22.4%. The agent performs stably under moderate conditions but remains weak in strong wind and high waves. The experiment is entirely based on trial and error without real flight data, demonstrating RL's potential in complex aerospace control tasks.