DecisionRL: An Open-Source Reinforcement Learning Library Built for Operational Decision-Making

Reinforcement Learning Goes Beyond Games and Robotics

When most developers think of reinforcement learning (RL), their minds jump to Atari games or MuJoCo robot simulations. This association has deep historical roots: RL's theoretical foundations trace back to the Bellman Equation and dynamic programming from the 1950s, with the Markov Decision Process (MDP) — defined by state space, action space, transition functions, and reward functions — as its core abstraction. In 2013, DeepMind combined deep neural networks with Q-learning, and DQN's superhuman performance on Atari games kicked off a series of milestones — AlphaGo, OpenAI Five — that thrust RL into the public eye while cementing its "game benchmark" stereotype. Mainstream RL libraries, whether in the OpenAI Gym ecosystem or Stable-Baselines3, are almost entirely organized around games and control tasks.

Yet a huge number of high-value real-world decision problems live at the operational level: How much inventory should a company restock? How should dynamic pricing be set? When should a battery storage system charge? Which jobs should a queuing system accept?

These are fundamentally sequential decision-making problems — perfectly suited for RL modeling. Inventory levels, market demand, and similar states evolve with each decision, while profit and cost form clear reward signals that satisfy the core MDP assumptions. Traditional operations research (OR) methods like linear programming and stochastic programming struggle with the "curse of dimensionality" when state spaces grow large or demand distributions are unknown or non-stationary. RL sidesteps this by using neural network function approximation and online sampling. Yet practical tooling for this domain has remained scarce. Developer Denis Drobyshev published his open-source project DecisionRL on Reddit to fill exactly this gap. He candidly noted that every time he wanted to apply RL to an operational problem, he had to build environments from scratch, construct baselines, and write evaluation pipelines — exhausting repeated work.

Making Operational Decision Environments First-Class Citizens

Six Built-In Operational Environments

DecisionRL's core design philosophy is to treat real operational decision problems as first-class environments built directly into the library. It currently ships with six ready-to-use application scenarios:

• Inventory Management: deciding reorder quantities
• Dynamic Pricing: adjusting pricing strategies in real time
• Queue Admission Control: deciding whether to accept or reject incoming jobs
• Thermostat / HVAC: intelligently regulating heating and cooling
• Energy Microgrid: optimizing battery storage and dispatch
• Supply Chain (two-echelon): coordinating multi-level inventory optimization

Each environment comes bundled with a corresponding classical Operations Research (OR) baseline policy — and this is what sets DecisionRL apart from generic RL libraries.

Why Built-In Baselines Matter

In operational settings, practitioners typically already have well-validated heuristics. Take inventory management: the Base-stock policy is the analytically optimal solution. Under the assumption that demand follows a known distribution and lead times are fixed, there exists an optimal target inventory level S* such that replenishing to that level each period minimizes expected cost — a result rigorously proven by Arrow et al. in 1951. Similarly, bang-bang (on/off) control in thermostat management comes from optimal control theory, where a system switches between two extreme states (fully on / fully off) — often time-optimal in linear systems.

These baselines represent "theoretically optimal under full information" — a demanding benchmark against which to evaluate RL policies. If a learned policy can't beat these simple methods, there's no point in deploying RL. DecisionRL pairs baselines directly with environments, letting developers directly verify whether a learned policy actually outperforms the heuristic — rather than blindly assuming RL is better. This pragmatic evaluation mindset is precisely what many academic RL projects lack.

Reproducible Benchmark Results

The author provides benchmark data reproducible on CPU, measured by cumulative return per task (higher is better):

Task	Learned Policy (RL)	Baseline Policy
Inventory Management	194.5	196.7 (base-stock)
Dynamic Pricing	24.6	11.5 (random)
Queue Admission Control	25.6	−16.2 (accept all)
Thermostat / HVAC	−35.8	−304.0 (bang-bang)
Energy Microgrid	21.3	13.1 (no battery)
Supply Chain (two-echelon)	−31.3	−175.5 (no restocking)

A few noteworthy observations emerge from the data:

RL beats the naive baseline on five of the six tasks. The improvements in thermostat control and supply chain are particularly dramatic (thermostat jumps from −304 to −35.8), demonstrating that RL has real value in high-dimensional, nonlinear operational optimization.

Inventory management is an interesting exception. RL scores 194.5, slightly below the base-stock baseline of 196.7 — but the author notes that RL essentially learned the analytically optimal base-stock policy from scratch. This is actually a positive signal: it demonstrates the algorithm's convergence capability, approaching the theoretical upper bound on a problem with a known optimal solution, while also validating the rigor of base-stock as a baseline.

The dynamic pricing baseline is only a random policy. The bar for beating a random baseline is relatively low, so while the 24.6 vs. 11.5 advantage is clear, stronger industry-standard baselines would be needed before production deployment.

A Complete Reinforcement Learning Algorithm Toolkit

DecisionRL is more than a collection of scenario environments — at its core is a complete, typed, and tested RL framework supporting a broad family of algorithms:

• Online algorithms: DQN, PPO, SAC, TRPO
• Offline Reinforcement Learning (Offline RL)
• Model-based methods (Model-based RL)
• Multi-agent RL
• Meta-RL

Each algorithm family has its strengths: DQN suits discrete action spaces, stabilized via experience replay and target networks; PPO uses a clip mechanism to constrain policy update magnitude, performing robustly on continuous control tasks; SAC introduces entropy regularization for a better exploration-exploitation balance and is the go-to choice for continuous action spaces; TRPO is PPO's predecessor, using KL divergence constraints to guarantee monotonic improvement.

Offline RL is particularly relevant for operational settings — historical order logs, pricing records, and similar data naturally form offline datasets. Policies can be pre-trained without interrupting live operations, then gradually deployed via A/B testing, significantly reducing deployment risk. The core challenge in offline RL is distribution shift; representative algorithms like CQL and IQL address this through conservative constraints.

The practical benefit of this design: when built-in environments can't meet your needs or the problem scales up, developers don't need to switch tools — everything can grow within the same framework. The tech stack is built on NumPy + PyTorch under an MIT license, making it friendly for both commercial and research use.

Energy Microgrid: A Scenario Worth Closer Attention

Among the six scenarios, energy microgrid scheduling deserves special mention. A microgrid is a small-scale power system containing distributed generation (solar PV, wind), energy storage (batteries), and local loads — capable of operating independently or grid-connected. Its dispatch optimization is a classic stochastic sequential decision problem: solar output and load demand are stochastic, electricity prices fluctuate throughout the day, and battery charge/discharge must account for cycle life degradation.

Traditional Model Predictive Control (MPC) depends on accurate forecasting models, whereas RL can learn policies directly from historical data and is more robust to prediction errors. DecisionRL demonstrates real value here with a score of 21.3 versus the "no battery" baseline of 13.1, echoing the growing academic and industrial interest in applying RL to virtual power plants and demand response programs.

An Honest Assessment: Opportunities and Limitations

The author is refreshingly candid that this is a solo project and actively solicits community feedback — especially on which new application scenarios are most needed. That openness is commendable, but prospective users should approach the project with clear-eyed expectations.

A single-maintainer project will typically lag behind established libraries like Stable-Baselines3 or CleanRL in terms of algorithmic rigor, long-term maintenance, and community support. Additionally, some baseline choices (a random policy for dynamic pricing, "no restocking" for supply chain) set a relatively low bar — more representative industry benchmarks would be needed before production deployment.

That said, DecisionRL addresses a genuine and long-neglected pain point: the engineering path from RL to operational decision-making. Its approach of packaging "environment + baseline + reproducible evaluation" into a standardized workflow makes it a valuable starting point for teams and researchers looking to explore reinforcement learning in inventory optimization, dynamic pricing, energy dispatch, and similar domains.

The project repository and documentation are fully open-source. Developers interested in these use cases are encouraged to test it against their own business scenarios and share real-world feedback with the author.

Key Takeaways