[KongchangAI]
· 2 min read· 1,017 words

How to Call Multiple Foundation Models on a Limited Budget? New Framework Uses Q-Learning for Dynamic Selection

How to Call Multiple Foundation Models on a Limited Budget? New Framework Uses Q-Learning for Dynamic Selection

Under an average query budget constraint, fixed querying or Q-learning-based dynamic combination of multiple foundation models can improve prediction accuracy.

This article introduces a study on combining multiple foundation models under budget constraints. Foundation models vary in accuracy and query costs, and deployment often faces an average per-input cost cap. The researchers propose two methods: querying a fixed set of models for all inputs and fusing outputs—theoretically proving that multiple less accurate models jointly can surpass the best single model and achieve the minimax rate—and using Q-learning to dynamically decide which models to query per input and when to stop, with finite-sample risk and cost upper bounds. Simulations and a multimodal laptop price prediction application show both methods outperform existing approaches. The article notes that treating model calling as an optimizable decision problem helps achieve better cost-effectiveness within a budget.

Why We Should Care About the Query Costs of Foundation Models

Thanks to large-scale pre-training, foundation models can provide out-of-the-box predictive capabilities for downstream tasks. However, different models vary significantly in accuracy on specific tasks, and the cost of each query is not the same—some models are lightweight and cheap, while others require calling massive parameters or complex reasoning at a high cost.

In real-world deployment scenarios, we often face a budget constraint: there is an upper limit on the average query cost allowed per input. How to combine the outputs of multiple foundation models within a budget—improving prediction accuracy without overspending—has become a key problem. A recent study proposes a systematic framework to resolve this tension. (Source: rss, Screenshot: screenshots/source/54398_0.png)

Prediction with Foundation Models under Deployment Budgets source screenshot

Two Strategies: Fixed Querying and Adaptive Q-Learning

The researchers designed two methods that use training data to learn "which models to query" and "how to combine outputs" with features that can be obtained without querying.

Method 1: Query the Same Models for Every Input

The first method queries a fixed set of models for all inputs and then fuses their outputs. It seems simple, but the paper proves a counterintuitive conclusion: multiple individually less accurate outputs can jointly surpass the best single model among them. The underlying logic is that different models may have complementary error patterns, and ensembling can offset some of the bias.

At the theoretical level, when the regression function depends only on model outputs within the budget, this method achieves the minimax rate. Conversely, if the regression function depends on models outside the budget, any budget-constrained method is forced into a slower rate, and this method also achieves that slower rate. This means Method 1 is theoretically near-optimal.

Method 2: Dynamically Select Models per Input

The second method allows different sets of models to be queried for each input and uses Q-learning to decide "which model to query next" and "when to stop querying." This is similar to a reinforcement learning agent that, at each step, decides based on currently available information whether it is worth continuing to query.

The paper clarifies the conditions under which varying the queried models per input can strictly outperform querying the same fixed set of models, and provides upper bounds on prediction risk and average cost under finite samples. This provides theoretical guarantees for the effectiveness of the dynamic strategy.

Minimax rate is a core concept in statistical learning theory for measuring the optimality of estimation methods. It characterizes the fastest rate at which the expected error in the worst case converges as the sample size increases, over all possible true data distributions. If a method achieves the minimax rate, it means no other method can significantly outperform it in the worst case, so the method is rate-optimal in a theoretical sense. Method 1 in the paper achieves this rate when the regression function depends only on model outputs within the budget, indicating that its error convergence rate cannot be generally improved; when the regression function depends on models outside the budget, any budget-constrained method can only achieve a slower rate, and Method 1 also matches this lower bound, further confirming its theoretical tightness.

Q-learning is a model-free reinforcement learning algorithm that learns an optimal policy by iteratively updating the state-action value function Q(s, a). In this framework, the "state" encodes the models already queried and their outputs, the "action" corresponds to which model to query next or whether to stop querying, and the "reward" combines the improvement in prediction accuracy with the query cost. Q-learning enables the agent, in an unknown environment dynamic, to balance exploration (trying new queries) and exploitation (making decisions based on existing knowledge) through trial and error, thereby learning an input-adaptive query strategy. Compared with fixed querying, Q-learning does not require a pre-specified query order and can flexibly handle the heterogeneity of different inputs, which explains why it can be strictly superior to static methods in theory.

Experimental Validation: Wins in Both Simulations and Real Applications

The researchers compared the two methods with existing methods through simulation experiments and a multimodal laptop price prediction application. The results show that both methods outperform existing approaches. Laptop price prediction involves multimodal features such as images and text, and model query costs vary, which precisely reflects the practical value of model selection under budget constraints.

Schematic of the prediction framework under budget constraints

Implications for AI Application Development

For AI application developers, this research conveys a pragmatic signal: there is no need to fetishize a single strongest model, nor to call the most expensive model for every input. By learning a query strategy from training data, multiple foundation models can be combined within a budget to achieve better cost-effectiveness. In particular, dynamic querying driven by Q-learning offers a practical approach for allocating computational resources on demand.

However, the paper's theoretical analysis relies on certain assumptions, and practical deployment still requires validation for specific tasks. Nevertheless, the framework itself points in a clear direction: treat model calling as an optimizable decision problem rather than a fixed pipeline.

One-line Summary

Under an average query budget constraint, fixed querying or Q-learning-based dynamic combination of multiple foundation models can improve prediction accuracy.

Paragraph Summary

This article introduces a study on combining multiple foundation models under budget constraints. Foundation models vary in accuracy on specific tasks and have different query costs, and real-world deployment often faces an upper limit on average cost per input. The researchers propose two methods: first, querying a fixed set of models for all inputs and fusing their outputs, with theoretical proof that multiple less accurate models jointly can surpass the best single model and achieve the minimax rate; second, using Q-learning to dynamically decide which models to query per input and when to stop, with finite-sample risk and cost upper bounds. Simulation experiments and a multimodal laptop price prediction application show that both methods outperform existing approaches. The article points out that treating model calling as an optimizable decision problem helps achieve better cost-effectiveness within a budget.

#foundation models#budget constraints#Q-learning#minimax rate#model combination
Share:

Related articles