Training a Humanoid Sword-Fighting Agent in Unity with SAC Reinforcement Learning

A developer shared their reinforcement learning project on Reddit: using the Soft Actor-Critic (SAC) algorithm to train a humanoid agent from scratch in Unity's physics engine, teaching it to sword fight against various opponents. The project not only demonstrates the potential of RL in physics simulation environments, but also sheds light on the practical challenges of reward shaping — one of the core difficulties in reinforcement learning.

Project Overview: A Physics-Based Humanoid Trained from Scratch

At the heart of the project is a physics-based humanoid agent built in Unity, trained through SAC to autonomously learn sword-fighting movements. The developer used a standard MLP (multilayer perceptron) architecture, forgoing complex network designs in favor of focusing on the training pipeline and reward design.

Training a physics-based humanoid from scratch is inherently challenging. Unlike characters driven by pre-set animations or motion capture data, this agent must evolve meaningful attack and defense behaviors through trial and error — starting from completely uncoordinated limb spasms. The developer openly admitted that watching the agent train was "really fun" — a classic sight in the early stages of reinforcement learning, where agents explore the action space in all sorts of unexpected, even comical ways.

Why Soft Actor-Critic?

SAC is a reinforcement learning algorithm that has shown strong performance on continuous control tasks in recent years. It's an off-policy actor-critic method whose core advantage lies in its maximum entropy framework — it encourages the policy to maintain sufficient randomness and exploratory behavior while still maximizing rewards.

This property is especially critical for physics-based humanoid control. The humanoid's action space is high-dimensional and continuous (involving torque or angle control across multiple joints), and training can easily get stuck in local optima or converge prematurely to rigid, repetitive movement patterns. SAC's entropy regularization helps the agent maintain exploration throughout training, making it easier to discover flexible and varied sword-fighting strategies. SAC also offers relatively high sample efficiency — a practical advantage for physics simulation tasks that require massive amounts of environment interaction before convergence.

SAC was proposed by Haarnoja et al. in 2018. Its maximum entropy objective can be described as: maximize cumulative reward while simultaneously maximizing the entropy of the policy (i.e., the uncertainty of the action distribution). Intuitively, the entropy bonus penalizes overly "confident" policies — if the agent always selects the same action in a given state, it gets penalized. This forces the agent to maintain a degree of exploration rather than locking in on a locally optimal behavior pattern too early.

Another engineering advantage of SAC is its off-policy nature: the agent can repeatedly sample from a historical experience replay buffer rather than discarding old data after each update, as on-policy methods like PPO must do. This gives SAC a significant edge in sample utilization, particularly for physics simulation where each simulation step is computationally costly. By contrast, while PPO is more stable and easier to tune, it typically requires more environment interaction steps to converge on high-dimensional continuous control tasks.

Reward Shaping: The Hardest Part of RL Engineering

The developer placed special emphasis on the lessons learned from reward shaping, hoping to help others working on similar projects. Reward shaping is widely regarded as both the most artful and the most error-prone aspect of reinforcement learning engineering.

In adversarial tasks like sword fighting, the reward function must balance multiple objectives: rewarding hits on opponents, penalizing getting hit, incentivizing stable upright posture — all while preventing the agent from finding "reward loopholes" (reward hacking), where it discovers unintended high-scoring behaviors that technically satisfy the reward function but violate the designer's intent. Even a slight miscalibration in reward design can lead to bizarre behaviors: the agent jittering in place, passively avoiding conflict, or gaming the score in ways that completely undermine the task goals. The developer compiled these hard-won lessons into a video precisely because they're the things textbooks rarely cover in detail, yet repeatedly trip up practitioners.

Reward hacking is a well-documented phenomenon in reinforcement learning, where an agent discovers high-scoring strategies the designer never anticipated — satisfying the literal requirements of the reward function while entirely defeating the task's intent. Classic examples include spinning in place to rack up lap counts in racing games, pushing objects into sensor blind spots to "eliminate" grasping failure penalties, and lying down to avoid getting hit in combat tasks.

Common mitigation techniques include: potential-based reward shaping, which adds intermediate rewards via differences in a state potential function without theoretically altering the optimal policy; curriculum learning, which starts from simpler subtasks and gradually increases difficulty to guide the agent toward correct foundational behaviors; and adversarial testing — actively constructing scenarios designed to trigger reward exploits to stress-test the robustness of the reward function. In adversarial tasks like sword fighting, self-play is also frequently introduced, having the agent compete against previous versions of itself to continuously generate challenging training signals.

Making Reinforcement Learning More Accessible

Beyond the technical implementation, this project has a communicative purpose. The developer produced a video breaking down the training process and final results, with the goal of making reinforcement learning "more approachable and entertaining."

This approach deserves recognition. Reinforcement learning has long carried a reputation for high barriers to entry and black-box opacity. By visualizing the humanoid agent's sword-fighting behavior, viewers can directly observe the agent's evolution from clumsy to capable — far more convincing than loss curves or paper equations alone. Unity, as a game engine, is naturally well-suited for this kind of visual and interactive RL experimentation. Combined with its ML-Agents toolkit, it's become a go-to platform for many independent researchers and enthusiasts looking to get started with physics simulation-based reinforcement learning.

Unity ML-Agents is an open-source RL toolkit from Unity Technologies that provides a communication bridge between a Python training backend and the Unity game engine. It supports mainstream algorithms including PPO and SAC, with built-in modules for curriculum learning and self-play. Its core value lies in integrating physics simulation, rendering, and the RL training loop into a single unified workflow, so developers don't need to build environment interfaces from scratch.

For humanoid control tasks specifically, ML-Agents provides a joint control interface backed by the PhysX engine, allowing torque or target-angle commands to be applied to each joint. The observation space typically includes proprioceptive information such as joint angles, angular velocities, center-of-mass position, and velocity. This design philosophy aligns with academic benchmark environments like DeepMind Control Suite and MuJoCo, meaning that tuning experience gained in Unity carries meaningful cross-platform transfer value.

Takeaways for RL Practitioners

This project is modest in scope, yet it distills several key principles of physics simulation reinforcement learning: algorithm selection should match task characteristics (SAC for continuous control), reward design is the deciding factor, and meaningful results are achievable even with standard architectures. For developers looking to get into RL, it's far better to first master the fundamentals — environment setup, reward shaping, and training debugging — than to chase complex network architectures from the start.

The developer actively invited community feedback on the agent's behavior in their post and offered to answer questions. That spirit of open sharing is itself one of the most valuable aspects of open-source, community-driven AI practice.