Does Agentic Model Orchestration Really Work? FairAMO Reveals the Boundaries of Ensemble Gains

Time series foundation models (TSFMs) perform unevenly across different forecasting tasks, which has given rise to Agentic Model Orchestration (AMO)—using an agent mechanism to dynamically select or combine multiple TSFMs in hopes of surpassing any single model. However, a new study raises a fundamental question about where such gains actually come from: do the improvements stem from smarter orchestration, or simply because the candidate pool happens to contain a stronger model?

The Popularity of AMO and an Overlooked Confounding Variable

The typical AMO approach is to build a candidate pool containing multiple TSFMs, then have a scheduler decide which model to use or how to weight their combination based on task characteristics. The paper points out that many prior works have reported AMO outperforming individual forecasters, but these comparisons often lack controls: the candidate pool may already contain models that excel on specific tasks, making it difficult to separate the contribution of "orchestration" from that of "candidate quality."

In other words, system-level improvements in forecast accuracy cannot necessarily be attributed to the orchestration mechanism itself. This is precisely the motivation behind the study's proposal of FairAMO.

Time series foundation models (TSFMs) refer to models pre-trained on large-scale, multi-domain time series data that can be transferred to downstream forecasting tasks via zero-shot or lightweight fine-tuning. Representative works include Chronos, TimesFM, and Moirai. Unlike traditional forecasting models customized for a single dataset, TSFMs attempt to cover heterogeneous tasks in retail, energy, transportation, and meteorology with a unified architecture. However, due to differences in pre-training data distribution, context length, and frequency adaptation, different TSFMs perform very differently on the same task—their rankings can even flip depending on task type. This "each has its weaknesses" characteristic is the practical foundation for model orchestration: if there were no cross-task capability differentiation, combining multiple models would yield no benefit.

FairAMO: Freeze the Candidates, Test Only the Orchestration

FairAMO is a controlled evaluation protocol whose core design is "frozen candidate forecasts"—the outputs of all candidate TSFMs are fixed before evaluation, and the orchestrator can only select or combine existing forecast results. This way, any change in accuracy can only come from the orchestration strategy, and cannot be "gamed" by introducing new models or retraining.

The study conducts experiments on the GIFT-Eval benchmark. This benchmark covers multiple types of time series forecasting tasks and is well-suited for testing the robustness of orchestration strategies across different task distributions.

MASE (Mean Absolute Scaled Error) is a commonly used accuracy metric in time series forecasting. It divides the model's mean absolute error by the mean absolute error of a naive baseline (typically seasonal naive forecasting, i.e., using the value from the same position in the previous cycle as the forecast). Therefore, a MASE below 1 indicates better performance than the naive baseline, and lower is better. Because it is scale-normalized, it can be compared and averaged across series of different magnitudes and frequencies. This makes it suitable as a cross-task aggregate metric, and it is also why this paper uses "less than 6% MASE improvement" to measure orchestration gains—the number reflects the proportional reduction in relative error, not the magnitude of absolute error.

The Ceiling of Gains: Less Than 6% MASE Improvement

The experiments yield a rather restrained conclusion: AMO does improve accuracy, but the improvement is firmly bounded by the candidate TSFMs themselves. Relative to the best single TSFM across all tasks, the best orchestrator reduces MASE (Mean Absolute Scaled Error) by less than 6%.

This number deserves closer scrutiny. It means the marginal value of orchestration is real but limited—if the candidate pool itself is not strong enough, no amount of sophisticated scheduling can break through the ceiling. For practitioners hoping to achieve order-of-magnitude improvements with AMO, this is a necessary expectation calibration.

Gains Come from Complementarity, Not Candidate Count

The paper further breaks down the sources of gains: the benefit depends on "complementarity" among candidates—whether different candidates excel on different tasks—rather than the size of the candidate pool.

This finding directly challenges the intuition that "more is better." Experiments show that adding members yields diminishing returns; more seriously, a redundant member can directly erase existing gains—redundant models not only fail to contribute diversity but may also interfere with orchestration decisions. The engineering implication is clear: when building a candidate pool, pursue differentiation in task coverage rather than piling up model count.

Candidate complementarity is closely related to but not equivalent to model diversity: diversity describes the degree of difference in predictive behavior among models, while complementarity emphasizes whether that difference happens to fall in each other's error regions—i.e., whether one model's weakness can be covered by another model's strength. Two models that are both strong but whose errors are highly correlated have low diversity and low complementarity; a model that is strong on short series paired with one that is strong on long series has structural complementarity. This is exactly the mechanism by which redundant members are harmful: they do not change the coverage structure of candidates along the task dimension, but they dilute the discriminative signal available to the scheduler, making selection based on historical performance more susceptible to noise.

Historical Evidence Saturates Quickly

Another noteworthy result concerns the time window. AMO typically relies on historical evidence to determine which model to use for the current task, but experiments find that the value of such evidence saturates very quickly: just a few recent windows are enough to capture all the gains, and longer history brings no additional benefit.

This has direct implications for system design—the orchestrator does not need to maintain lengthy historical records; recent performance signals already contain most of the information needed for decisions, reducing dependence on storage and compute resources.

Reunderstanding System-Level Gains

The core contribution of this work is separating "system-level forecasting gains" from "the contribution of orchestration itself." Under the controlled condition of frozen candidates, the independent value of orchestration is quantified: limited but measurable, and highly dependent on the complementary structure of candidates.

For the research community, FairAMO provides a more honest evaluation paradigm—before claiming that an orchestration method is superior, first rule out the confounding factor of candidate quality. For engineering teams, the conclusion is equally pragmatic: investing in the diversity and quality of candidate models yields higher returns than investing in more complex scheduling logic; and when the candidate pool is already homogeneous, adding an orchestration layer is most likely just creating complexity.

One-line Summary

Controlled evaluation with frozen candidates shows that the independent gain of model orchestration is less than 6%, and depends on complementarity rather than quantity.

Paragraph Summary

The article introduces a controlled study on Agentic Model Orchestration (AMO). AMO has often been reported to outperform single time series foundation models (TSFMs), but the candidate pool itself may already contain strong models, conflating the contribution of orchestration with candidate quality. To address this, the study proposes the FairAMO protocol, which freezes the forecast outputs of all candidate TSFMs so that the orchestrator can only select or combine them, thereby isolating the independent contribution of orchestration. On the GIFT-Eval benchmark, the best orchestrator achieves less than 6% MASE improvement over the strongest single TSFM, with gains constrained by the ceiling of the candidate pool. Further analysis shows that gains stem from task complementarity among candidates rather than scale, and redundant members may even offset existing gains; historical evidence also saturates quickly, with only a few recent windows sufficient to capture all gains.