From PID to Reinforcement Learning: A Survey of HVAC Control Architectures

Control strategies for building heating, ventilation, and air conditioning (HVAC) systems are caught between three competing objectives: energy savings, comfort, and equipment operability. A recent survey paper systematically maps the spectrum of control architectures from classical PID regulation to data-driven reinforcement learning (RL), and puts forward a key argument: these methods are not competing philosophies, but nested intervention points on the same set of HVAC equipment.

The Hierarchical Control Stack: PID Is Not the Whole Story

The paper places building HVAC control within a hierarchical stack: at the bottom is the thermostat-PID regulation layer, covering everything from two-position control and scheduled setpoint control to proportional, PI, and PID regulation and their tuning; above it are rule-based and model-based supervisory control, as well as data-driven reinforcement learning. This perspective breaks down the "PID vs. RL" binary—the real distinction lies in where the intervention point sits within the stack.

The thermostat-PID layer is the control layer that most buildings actually run on. It is low-cost and its coding is mature, but it is limited by nonlinearity, time variation, discrete mode switching, loop coupling, and slow plant dynamics. These limitations are precisely the gaps that upper-layer supervisory strategies attempt to fill.

PID control (proportional-integral-derivative) is the most classic feedback regulation algorithm in industrial control: the P term responds to the current error, the I term accumulates past error to eliminate steady-state offset, and the D term predicts the trend of error change to suppress overshoot. Its output is (u(t)=K_p e(t)+K_i\int e,dt+K_d \frac{de}{dt}), and the three gain parameters determine the balance among response speed, steady-state accuracy, and stability. Tuning is the process of determining this set of parameters, commonly using Ziegler-Nichols, internal model methods, or optimization-based approaches. PID does not rely on an explicit model of the plant and requires very little computation, which is why it has become the default choice for terminal loops such as thermostats, dampers, and water valves. But its limitations stem from the same source: when faced with strong nonlinearity, parameter drift, mode switching, and inter-loop coupling, fixed gains struggle to remain optimal across all operating conditions—which is precisely the motivation for introducing upper-layer supervisory or learning-based strategies.

Four Ways RL and PID Meet

The paper's core contribution is a taxonomy of the intersection points between RL and PID, comprising four architecture types:

RL as a gain tuner: RL does not directly control actuators but instead adjusts the gain parameters of existing PID loops. This approach preserves the integrity of the physical loop and has relatively low requirements for state information and modeling, but RL's scope for action is limited by the granularity of gain adjustment.

RL as a direct replacement for the regulating controller: The RL policy directly outputs control signals, replacing PID's regulation function. This requires higher actuation authority and state information, and must also handle constraint satisfaction, making it the highest barrier to deployment.

Hybrid of RL and model predictive control (MPC): RL is combined with MPC, leveraging MPC's constraint-handling capability and RL's adaptability. This hybrid architecture seeks a balance between constraint handling and performance, but increases the complexity of modeling and debugging.

RL as a supervisory layer generating setpoints and schedules: RL generates setpoints or operating schedules at the upper layer, while the lower layer still relies on PID for regulation. This architecture has the lowest actuation authority requirements, but RL's optimization space is limited to the setpoint space.

Model predictive control (MPC) solves a constrained optimization problem over a future horizon based on a system model at each sampling instant, executes only the first control action, and then recomputes in a rolling manner. Its core advantage is that it can explicitly handle hard constraints such as actuator limits and temperature bounds, and coordinate multiple loops in multivariable coupled scenarios. In building HVAC, MPC is often used for upper-layer supervision—for example, optimizing chiller supply water temperature, supply air setpoints, or thermal storage scheduling with energy consumption and comfort as objectives. MPC's weakness lies in its reliance on an available prediction model and online solving compute; modeling error and computational burden limit its adoption on small-scale equipment. The hybrid of RL and MPC attempts to use learning to compensate for model bias or reduce online solving costs while retaining MPC's constraint-handling framework.

One Unchanging Fact: The Physical Loop Stays the Same

A key observation running through all four architectures is that the physical loop itself remains unchanged—only the intervention point changes. This insight has important engineering significance: it means that deploying RL does not require overthrowing existing PID infrastructure, but can instead be layered on at different levels. The paper emphasizes that RL's "acceptable position" must be argued rather than assumed, and that different architectures place different demands on actuation authority, state information, modeling, debugging, and constraint handling.

Actuation authority refers to the range of physical quantities that a given control layer can directly manipulate. Directly outputting valve opening or compressor frequency constitutes high actuation authority, whereas being able to modify only setpoints or operating schedules constitutes low actuation authority. The higher the authority, the more directly the policy affects equipment behavior and the greater the potential energy and comfort benefits—but the stricter the requirements for safety, constraint satisfaction, and fault tolerance. Even if a low-authority supervisory layer makes a policy error, the underlying PID can still maintain basic stability, so deployment risk is lower. When evaluating an RL architecture, clearly defining the boundaries of its actuation authority is a prerequisite for judging feasibility and the consequences of failure, and is also the basis for deciding whether additional safety shields or fallback mechanisms are needed.

Deployment Trade-offs: No Free Lunch

The paper points out that deployment trade-offs must be evaluated according to specific control interfaces and scaffolding requirements. PID's advantage lies in its low cost and mature coding, but its limitations are especially pronounced in nonlinear, time-varying, and loop-coupled scenarios. RL offers adaptability and optimization potential, but introduces new requirements for data, modeling, and debugging. Hybrid implementations are possible and are often the realistic path.

The practical value of this survey is that it provides engineers with a decision framework: rather than asking "should we use PID or RL," ask "under the current control interface and scaffolding conditions, at which layer should RL intervene?"

One-line Summary

This survey proposes a hierarchical HVAC control stack and categorizes the intersection of RL and PID into four nested architectures.

Paragraph Summary

The paper systematically maps the architecture spectrum of building HVAC control from classical PID to data-driven reinforcement learning, and offers a hierarchical control stack perspective: the bottom layer is thermostat-PID regulation, and the upper layers are rule-based and model-based supervisory control and RL. The paper's core contribution is a taxonomy of four architectures at the intersection of RL and PID—RL as a gain tuner, as a direct replacement for the regulating controller, as a hybrid with MPC, and as a supervisory layer generating setpoints and schedules. A key observation running through all four is that the physical loop remains unchanged; only RL's intervention point changes. Different architectures have different requirements for actuation authority, state information, modeling, debugging, and constraint handling, and deployment trade-offs must be evaluated in light of specific control interfaces and scaffolding conditions, with hybrid implementations often being the realistic path.