Greenhouse Controller Showdown: Why Reinforcement Learning Lost to a Rule-Based Baseline

The Overlooked Control Challenge: Greenhouse Economic Optimization

In fields like autonomous driving and robotics, deep reinforcement learning (RL) has long been a superstar. Yet in some seemingly mundane industrial control settings, RL often delivers surprising — and disappointing — results. Greenhouse climate control is a textbook example.

Recently, a researcher launched an open, reproducible benchmark project on Reddit, systematically comparing four types of controllers on the open-source greenhouse simulation environment GreenLight-Gym2 (released under the AGPL license). GreenLight-Gym2 is built on the GreenLight physical model developed by Wageningen University in the Netherlands, and is one of the most realistic open-source greenhouse dynamics simulators available. Importantly, greenhouse control is fundamentally a high-dimensional, multi-objective problem: it requires simultaneously regulating temperature, CO₂ concentration, relative humidity, light intensity, and other tightly coupled state variables, while balancing economic factors like energy costs, crop growth rates, and market prices. The challenges include a continuous high-dimensional state space, action effects that manifest with delays of hours or even days, and external weather disturbances that are difficult to predict precisely — all of which put both traditional control methods and modern machine learning approaches to the test.

This project goes beyond raw control performance, focusing on three core dimensions: data efficiency, generalization to unseen weather years, and constraint safety. The greatest value of this work lies in bringing together methods that the academic community has been evaluating in isolation and putting them through a fair, unified comparison.

Four Controllers Enter the Ring

The benchmark divides competitors into four tracks (arms):

1. Rule-Based Controller

The "reigning champion" of the greenhouse industry — a carefully hand-tuned rule-based control strategy that encodes decades of domain expert experience and represents the mainstream approach in current production environments.

2. Model-Free Reinforcement Learning

This track uses mainstream algorithms — PPO (Proximal Policy Optimization) and SAC (Soft Actor-Critic) — allowing agents to learn optimal policies through environment interaction without explicit modeling of system dynamics.

These two algorithms differ fundamentally in design philosophy. PPO, introduced by OpenAI in 2017, is an on-policy algorithm that uses only data collected under the current policy for each update; it stabilizes training by clipping the policy update objective, yielding good stability but lower sample efficiency. SAC, introduced by UC Berkeley in 2018, is an off-policy algorithm that reuses historical data via an experience replay buffer and incorporates a maximum entropy objective — maximizing cumulative reward while simultaneously maximizing policy entropy to encourage broader exploration. This property generally gives SAC higher sample efficiency and greater robustness on continuous-action control tasks, making it theoretically better suited to greenhouse-style scenarios with sparse rewards and long time horizons.

3. Pure Physics-Based Economic MPC (Model Predictive Control)

Model Predictive Control originated in the petrochemical industry in the 1970s. Its core idea is "rolling optimization with feedback correction": at each control step, use a system dynamics model to predict the next N steps of system state, solve a finite-horizon optimal control problem, execute the first step of the optimal solution, then repeat the process at the next time step. Economic MPC (eMPC) embeds economic objectives (e.g., profit maximization) directly into the optimization cost function rather than minimizing traditional state-tracking error. MPC's strengths include explicit constraint handling (e.g., temperature bounds, actuator saturation) and the ability to incorporate lookahead information like weather forecasts; its weaknesses are high computational cost and heavy dependence on model accuracy.

4. RL-in-MPC (Reinforcement Learning Embedded in MPC)

A hybrid architecture combining reinforcement learning and model predictive control, aiming to capture the best of both worlds. The project references van Laatum et al. (arXiv:2607.07365), adopting a trajectory-selection RL-MPC approach.

The first two tracks (rule-based baseline and model-free RL) have been completed first, yielding an initial finding that deserves careful reflection.

Key Finding: PPO Lost to Hand-Crafted Rules

The project author reports a key conclusion submitted to the community for verification:

PPO's held-out economic return plateaus somewhere between roughly 50,000 and 300,000 steps, and falls clearly short of a well-tuned rule-based baseline.

In other words, model-free reinforcement learning, even after extensive training, still underperforms manual rule strategies on economic metrics — and simply adding more training steps cannot close the gap.

Three deeper reasons explain this phenomenon:

Sparse and delayed reward signals: Economic returns in greenhouse settings typically materialize only after a long growth cycle, making credit assignment extremely difficult. Credit assignment is one of the core challenges in reinforcement learning, first articulated by Minsky in 1961. The fundamental question is: when a sequence of actions eventually produces a reward, how do you determine which historical actions contributed most to that outcome? In the greenhouse context, a tomato plant may take 90–120 days from transplanting to harvest, meaning a temperature control decision on any given day might only manifest in crop yield or quality weeks later. Traditional temporal-difference (TD) methods propagate value estimates incrementally via bootstrapping, but over such long time horizons the value signal degrades severely — which is one of the root reasons PPO struggles to break through even after 300,000 steps.
A strong baseline is hard to beat: The hand-tuned strategy already encodes substantial domain knowledge and is effectively near a local optimum; RL starting from scratch has enormous ground to make up.
Questionable generalization: Faced with weather years outside the training set, RL policies tend to overfit to specific climate patterns.

This also confirms a growing consensus in academia: in industrial settings where reliable physical models and mature expert knowledge exist, end-to-end model-free RL is not necessarily the optimal solution.

Four Paths to Improvement

To address PPO's underperformance, the author solicited improvement ideas from the community. The main directions are:

Switch to SAC

SAC's maximum entropy framework and off-policy nature typically yield better sample efficiency than PPO on continuous control tasks, along with stronger exploration. Its maximum entropy objective not only prevents premature convergence to local optima, but also makes the agent more robust to unseen weather patterns — particularly valuable for cross-year generalization in greenhouse settings.

Reward Shaping

Design denser, more informative intermediate reward signals to ease the learning difficulties caused by sparse rewards. This is a double-edged sword, however: a classic 1999 result by Ng et al. proved that only potential-based reward shaping is guaranteed to preserve the optimal policy. Arbitrarily added reward signals can lead to reward hacking — the agent exploits loopholes in the reward design, scoring high on intermediate rewards while violating the true objective. In the greenhouse context, a poorly designed intermediate reward that overemphasizes short-term energy savings at the expense of long-term crop quality could lead the agent to learn a damaging strategy of cutting energy during critical growth stages, ultimately causing real economic losses. This is precisely why reward engineering requires deep involvement from domain experts.

Extend the Training Budget

Although current curves show a plateau after 300,000 steps, longer training combined with sound exploration mechanisms could theoretically yield further gains — though the marginal returns from simply adding more steps are limited.

Shift to Model-Based RL

This is perhaps the most promising direction. Greenhouses have relatively reliable physical models, and model-based reinforcement learning that fully leverages this model information — or hybrid architectures like RL-in-MPC — may be what's needed to truly break through the performance ceiling. Compared to model-free RL, model-based methods can exploit physical priors to perform "imagination rollouts," dramatically reducing the number of real environment interactions required, while model-based constraints help avoid unsafe states — a natural fit for greenhouse control's high safety requirements.

RL-in-MPC: The Reproducibility Problem in Reproduction Work

The most technically challenging of the four tracks is reproducing the trajectory-selection RL-MPC method proposed by van Laatum et al. The core idea is to use reinforcement learning to assist MPC in trajectory selection, injecting RL's adaptive advantages while preserving MPC's constraint safety guarantees.

However, the author ran into a familiar pain point in research: no publicly available implementation code. This detail reveals a deeper systemic problem — the reproducibility crisis in AI and control research. This has become a widely recognized issue: the 2019 NeurIPS Reproducibility Challenge and a related 2020 Science survey both noted that a large proportion of top conference papers are extremely difficult to reproduce independently due to lack of open-sourced code, incomplete hyperparameter reporting, or reliance on proprietary datasets. In the RL domain specifically, Henderson et al. (2018) showed that even the same algorithm can exhibit performance differences of several-fold across different implementations, owing to high sensitivity to random seeds, hyperparameters, and environment versions. The situation in industrial control is even more severe: commercial incentives push companies to protect core implementations, while academic researchers lack motivation to maintain production-quality codebases.

This is precisely what makes the author's open benchmark project so valuable: they have committed to releasing the entire benchmark harness back to the community upon project completion.

Closing Thoughts: A Sober View of RL and the Value of Hybrid Methods

This ongoing open benchmark has already delivered several important signals:

First, we need to temper the "RL can do everything" narrative. In settings with strong physical models and mature expert baselines, hybrid approaches combining MPC and RL are often more practical and effective than pure model-free RL.

Second, fair and reproducible cross-method benchmarks are scarce — and enormously valuable. Only under a unified environment and evaluation standard can the true strengths and weaknesses of different methods be objectively measured.

Third, a culture of open, shared research is the key driver of healthy progress in industrial AI and agricultural control. We look forward to this benchmark framework becoming a reliable, authoritative reference point for the field once complete.