decisionrl: An Open-Source Reinforcement Learning Library Built for Operational Decision-Making

The Blind Spot in Reinforcement Learning Applications

When most people think of reinforcement learning (RL), images of AlphaGo playing chess, robotic arms grasping objects, or agents achieving high scores in Atari games come to mind. This is no coincidence — mainstream RL libraries and benchmarks are almost entirely built around games and robotics, with classic environments like Atari and MuJoCo long established as the field's default "standard kit."

Yet there exists another class of enormously valuable, chronically overlooked RL applications: operational decision-making. How much inventory should be restocked? How should dynamic pricing be set? When is the most cost-effective time to charge a battery? Which incoming requests should a system accept? These questions share a common trait — even small optimizations in each decision can translate directly into significant economic gains.

It's worth noting that reinforcement learning and operational decision-making share a deep historical connection. RL grew out of behavioral psychology's trial-and-error learning theory, with its mathematical foundation rooted in Markov Decision Processes (MDPs). MDPs themselves are a core tool of Operations Research (OR), introduced by Richard Bellman in the 1950s — hence the famous "Bellman equation." The two fields share nearly identical mathematical foundations, yet diverged over the decades into different engineering practices: OR emphasizes closed-form solutions and convex optimization, while RL focuses on approximating solutions from data. In a sense, decisionrl's emergence is an effort to bridge two fields that should have remained close but gradually drifted apart.

Developer Denis Drobyshev shared his solution on Reddit: decisionrl, an open-source RL library specifically designed for operational decision problems. He candidly admitted that every time he wanted to apply RL to these real-world problems, he had to build environments, baselines, and evaluation pipelines from scratch — and it was this recurring pain point that gave birth to the project.

What decisionrl Solves

Making Operational Problems First-Class Citizens

The core design philosophy of decisionrl is to provide operational optimization problems as built-in, first-class environments. In software framework design, "first-class citizen" means a type of object enjoys equal design status with the framework's core concepts — with full type support, standard interfaces, documentation, and test coverage, rather than being bolted on as an afterthought. By contrast, while OpenAI Gym provides a standardized environment interface, its official built-in environments are almost exclusively Atari games and MuJoCo physics simulations. Operational problems require users to implement everything from scratch, setting the bar extremely high. decisionrl elevates the following six operational environments to framework-core status, meaning their simulation logic, reward design, and baseline comparisons all undergo unified quality assurance:

• Inventory management
• Dynamic pricing
• Queue admission control
• Thermostat / HVAC
• Energy microgrid
• 2-echelon supply chain

These are precisely the classic problems that Operations Research has studied for decades. decisionrl packages them into ready-to-use RL environments, eliminating the tedious work of building simulations from the ground up.

Built-In OR Baselines to Prevent Self-Deception

Another commendable design choice is that each environment comes paired with a corresponding classical OR baseline. This may seem simple, but it carries significant engineering value.

There's a common trap in real projects: you train an RL policy that appears to converge nicely, then assume it outperforms traditional heuristics. Without a reliable baseline for comparison, you simply cannot tell whether the learned policy is genuinely effective or just got lucky. By providing built-in baselines, decisionrl forces you to verify — does the RL policy actually beat the heuristic — rather than blindly assuming it does.

Benchmark Results: Five Out of Six Tasks Won

The author reproduced a set of comparison experiments on CPU, with quite compelling results:

Task	RL Learned Policy	Baseline Method
Inventory management	194.5	196.7 (base-stock)
Dynamic pricing	24.6	11.5 (random)
Queue admission control	25.6	−16.2 (admit all)
Thermostat / HVAC	−35.8	−304.0 (bang-bang control)
Energy microgrid	21.3	13.1 (no battery)
2-echelon supply chain	−31.3	−175.5 (no ordering)

The RL policy clearly outperforms the baseline on five of the six tasks. A few highlights worth noting:

• Queue admission control: The RL policy leaps from −16.2 to 25.6, a qualitative shift from loss to profit;
• HVAC: Cost improves dramatically from −304.0 to −35.8, nearly a nine-fold improvement. The baseline here is the classic bang-bang control (also known as on-off or relay control) — a naive control strategy with only two extreme states: fully on or fully off, as seen in typical home thermostats. Bang-bang control is cheap to implement but comes at the cost of frequent system oscillation and poor energy efficiency. The RL policy learns a smoother control curve in the continuous state space, which is the source of its advantage;
• Inventory management: RL (194.5) falls slightly short of the baseline (196.7), but the author points out that RL independently rediscovered the analytically optimal base-stock policy from scratch. The base-stock policy is one of the most classic optimal solutions in inventory management: set a target inventory level S and replenish up to that level each time. Under the assumptions of i.i.d. demand and fixed lead times, this strategy can be rigorously proven to be the analytically optimal solution for minimizing total cost. The fact that RL autonomously approximated this structure through pure trial-and-error interaction, with no prior knowledge, is itself a compelling validation of algorithmic capability.

Honestly presenting results that fall short of a clean sweep actually enhances the project's credibility.

Underlying Architecture: More Than a Toy

Many domain-specific libraries fall into the trap of "built-in problems work fine, but extensions are a nightmare." decisionrl attempts to avoid this — at its core, it is a complete, type-annotated, test-covered general-purpose RL framework.

The supported algorithms cover a broad spectrum:

• Mainstream online algorithms: DQN, PPO, SAC, TRPO
• Offline Reinforcement Learning (Offline RL)
• Model-based methods (Model-based RL)
• Multi-agent RL
• Meta-RL

Support for Offline RL is particularly noteworthy. In operational scenarios like inventory management and pricing, allowing an RL agent to explore in a real production system incurs actual losses (e.g., misjudged pricing leading to significant financial damage). Offline RL enables policy learning from historically accumulated operational data without real-time interaction with the live environment, which is critical for reducing exploration risk. Model-based RL learns an environment dynamics model first, then plans using it, typically offering superior sample efficiency over model-free methods — equally valuable for operational scenarios where simulation costs are high. Native support for both of these methods reflects decisionrl's engineering consideration for real-world deployment, not merely academic demonstration.

This means that when business problems extend beyond the built-in environments, there's no need to switch toolchains — you can continue working within the same framework. The tech stack is built on NumPy + PyTorch, released under the MIT license, and is friendly for commercial use.

Value Proposition and Practical Limitations

decisionrl fills a genuine gap in the RL ecosystem. The academic community and open-source world have invested heavily in games and robotics for years, yet mature, ready-to-use tools for operational decision problems like pricing, inventory, energy scheduling, and queue control remain scarce. For engineers and researchers looking to deploy RL in enterprise operational settings, decisionrl can significantly lower the barrier to entry.

That said, a clear-eyed assessment is warranted: the author explicitly states this is a solo project, and long-term maintenance, community support, and production-grade stability still require time and contributions from a broader community. While the current benchmark results are impressive, they all come from relatively simplified simulation environments, and a meaningful gap remains between these and the complex constraints, noise, and non-stationarity of real industrial scenarios.

The author also sincerely invites community feedback, particularly around which new application environment scenarios people would like to see added next. This kind of open-minded approach is often the key factor determining whether an early-stage open-source project can go the distance.

Summary

decisionrl's value lies not in algorithmic innovation, but in its problem framing and engineering approach — it organically integrates classic OR problems with modern RL algorithms, and uses built-in baselines to instill a habit of rigorous evaluation. For practitioners looking to explore the practical value of reinforcement learning in operational decision-making, it is a starting point well worth serious consideration.

Project repository: github.com/DenisDrobyshev/decisionrl