NANO: Non-Interfering Asymmetric Updates under AdamW for Breaking Through Hybrid Model Equifinality

In hybrid modeling, neural networks provide unresolved parameters to process-based simulators, but observational data often only constrain aggregate quantities and cannot constrain internal partitioning, leading to the equifinality problem—multiple internal configurations are indistinguishable in observed aggregate quantities yet give radically different interpretations of mechanisms. A recent study proposes Non-Interfering Asymmetric Network Optimization (NANO), which achieves true non-interference between auxiliary gradients and the main objective gradient under the AdamW optimizer, thereby actively exploring the equifinality space while maintaining fitting accuracy.

The Equifinality Dilemma: Internal Invisibility under Observed Aggregates

The study uses evapotranspiration as an example: total flux is observable, but its decomposition into transpiration and evaporation components cannot be directly observed. This causes model ensembles built solely from random seeds to cover only a narrow segment of the partitioning space allowed by the data. In other words, the models appear diverse but in fact severely underexplore internal mechanisms.

Equifinality is not unique to evapotranspiration; any hybrid model where observations constrain aggregate fluxes or storages rather than internal partitioning faces the same problem. In such cases, even when models perform consistently on observations, their answers to "what mechanism corresponds to the same data" may differ dramatically.

Equifinality was first systematized in the fields of systems science and hydrological modeling, referring to the fact that different internal structures or parameter combinations can produce the same observed output. In hybrid models, when observational data only constrain aggregate fluxes (such as total evapotranspiration), the internal partitioning (such as the ratio of transpiration to evaporation) has a large number of equivalent solutions. These solutions are indistinguishable at the observational level yet yield different or even contradictory conclusions for mechanistic interpretation. Equifinality differs from parameter unidentifiability: the former emphasizes multi-solution at the model structure level, while the latter focuses on variance issues in parameter estimation. In scenarios coupling process-based simulators with neural networks, the unresolved parameters provided by neural networks often lack physical constraints, exacerbating the dimensional expansion of the equifinality space. Therefore, when evaluating model uncertainty, relying solely on observational goodness-of-fit is insufficient; the diversity of internal configurations must also be actively explored.

Actively Creating Divergence: Auxiliary Training with Repulsion

To deliberately expand the distribution in the equifinality space, the researchers apply an auxiliary repulsion to the transpiration component during training while constraining observational fitting not to degrade. The core idea is: not settling for the natural differences brought by random seeds, but directionally pushing the model to explore internal configurations that are allowed by the data but not yet covered.

The auxiliary repulsion is equivalent to adding a term to the loss function that encourages different ensemble members to move away from each other on the transpiration component. But it must be ensured that the main loss (fitting to observations) is not harmed; otherwise the model would lose its explanatory power for the data.

The Non-Interference Challenge under AdamW: Gradient Orthogonality Is Not Enough

The usual approach to achieving non-interference is to make the auxiliary gradient orthogonal to the main gradient. However, the study points out that under the AdamW optimizer, the actual applied update step is a preconditioned momentum direction rather than the raw gradient. Therefore, orthogonality between raw gradients no longer implies non-interference between update steps.

This is a key insight: many orthogonalization techniques effective in the SGD era fail under adaptive optimizers such as AdamW. AdamW's preconditioning matrix distorts gradient directions, causing originally orthogonal gradients to produce cross-effects after updating.

The AdamW optimizer is an improved version of Adam, whose core change is decoupling weight decay from the gradient update into an independent parameter shrinkage term. AdamW maintains the first moment (momentum) and second moment (exponential moving average of squared gradients) for each parameter, and uses the square root of the second moment to precondition the momentum coordinate-wise, forming adaptive step sizes. This preconditioning makes the update direction no longer equivalent to the raw gradient direction: gradients on different coordinates are scaled by different magnitudes, so two directions orthogonal in the raw gradient space may no longer be orthogonal after the preconditioning matrix acts on them. Therefore, techniques that achieve multi-task non-interference through gradient orthogonalization under SGD will fail when directly transferred to AdamW. To ensure that the auxiliary objective does not interfere with the main objective, projection or constraint mechanisms must be redesigned for the preconditioned momentum direction—this is precisely the theoretical starting point of NANO.

NANO: Asymmetric Updates with Exact Non-Interference

The researchers propose NANO (Non-interfering Asymmetric Network Optimization), whose updates have three properties:

• The first-order effect on the main loss is exactly the same as when using only the main objective;
• The auxiliary step size is bounded, preventing excessive repulsion from destroying the fit;
• The repulsion is concentrated on coordinate directions where the main gradient is "idle," i.e., dimensions that the main gradient itself does not attend to.

This means NANO mathematically guarantees that the auxiliary objective will not interfere with the optimization trajectory of the main objective, while effectively using the degrees of freedom left by the main gradient to explore equifinality. The so-called "asymmetric" aspect is reflected in the different treatment of the main update and the auxiliary update: the main update remains as is, while the auxiliary update is projected onto directions that do not affect the first-order change of the main loss.

Invariance of the first-order effect is the mathematical core of NANO's non-interference property: the influence of the auxiliary update on the main loss is zero at infinitesimal step sizes, i.e., the directional derivative of the main loss along the auxiliary update direction is zero. This is achieved by projecting the auxiliary update onto a subspace orthogonal to the main loss gradient, but the key is that the projection target is the AdamW-preconditioned momentum direction rather than the raw gradient. Main-gradient idle coordinates refer to dimensions where the main loss gradient is close to zero; the main objective has almost no constraint in these dimensions, so applying auxiliary repulsion here will not significantly change the main loss. Bounded auxiliary step size is achieved by constraining the norm of the repulsion term or adopting adaptive scaling, preventing excessive repulsion in early training from causing the model to deviate from the data-fitting region. These three properties together ensure that NANO is mathematically strictly non-interfering while effectively using the degrees of freedom left by the main objective.

Implications: Uncertainty in Hybrid Models Requires Active Design

This work reminds us that equifinality in hybrid models cannot be adequately exposed by random seeds alone. When observational constraints are incomplete, the diversity of model ensembles needs to be actively designed. NANO provides a method for achieving non-interfering auxiliary training within the AdamW framework, enabling controlled exploration of the internal mechanism space while maintaining data fitting.

For fields that rely on hybrid combinations of process-based simulators and neural networks, such as Earth system modeling and hydrological forecasting, this approach helps to more honestly express the uncertainty of internal partitioning, avoiding underestimation of mechanism-level predictive divergence due to insufficient ensemble diversity.

In the future, how to generalize NANO to more complex constraint scenarios and how to evaluate the actual coverage of the explored equifinality space remain directions worth attention.

One-line Summary

NANO achieves non-interfering auxiliary training under AdamW, actively exploring the equifinality space of hybrid models.

Paragraph Summary

In hybrid models, observational data only constrain aggregate fluxes, leading to equifinality in internal partitioning—multiple configurations are indistinguishable in observations yet give different interpretations of mechanisms. Random-seed ensembles can only cover a narrow internal space. The study proposes the NANO method, which achieves true non-interference between auxiliary gradients and the main objective gradient under the AdamW optimizer. The key insight is: AdamW's preconditioned momentum direction means that orthogonality of raw gradients no longer guarantees non-interference of update steps. By projecting auxiliary updates onto main-gradient idle coordinates, keeping the first-order effect on the main loss unchanged, and bounding the auxiliary step size, NANO actively expands the exploration range of the equifinality space while maintaining fitting accuracy. Using evapotranspiration as an example, NANO effectively expands the exploration of the transpiration/evaporation partitioning.