LlMBO: Fine-Tuning LLMs with Bayesian Optimization, Leaving Gradient Descent Behind

Gradient descent is almost the default answer for large model training, but a new paper proposes a counterintuitive idea: when the number of fine-tuning parameters is as few as a few dozen, perhaps gradients aren't needed at all.

From TinyLoRA to Bayesian Optimization

TinyLoRA, previously proposed by Morris et al., has already demonstrated that the number of parameters required for LLM fine-tuning can be surprisingly small—fewer than 10 parameters can yield accuracy improvements, whereas traditional LoRA methods typically require over ten thousand parameters.

But TinyLoRA still relies on the local search approach of gradient descent. This is precisely where the new work comes in: in an extremely low-dimensional parameter space, global optimization methods become feasible. The authors thus propose LlMBO (Large language Model Bayesian Optimization), introducing Bayesian optimization into LLM fine-tuning.

Bayesian Optimization (BO) is a class of global optimization methods for black-box objective functions, particularly suited to scenarios where evaluation is costly, gradients are unavailable, or noise is significant. Its core idea is to model the objective function with a surrogate model (typically a Gaussian process) and combine it with an acquisition function (such as EI, UCB) to balance "exploring unknown regions" and "exploiting near the current optimum", thereby approaching the global optimum with as few function evaluations as possible.

The computational complexity of Bayesian optimization typically grows exponentially with parameter dimensionality, so it has traditionally only been applicable to low-dimensional problems (generally within a few dozen dimensions). This is precisely the premise on which LlMBO rests: only when the tunable parameters are compressed to extremely low dimensionality does BO become computationally feasible. In contrast, gradient descent is a local method designed for high-dimensional spaces; it uses first-order derivative information to descend efficiently in a massive parameter space, but cannot guarantee global optimality and requires storing intermediate activations and optimizer states. The two have a complementary relationship in terms of the dimensional ranges where they are applicable.

Core Advantages of LlMBO

Several characteristics of LlMBO are worth noting:

• Sample efficient: Good parameter configurations can be found without massive training data
• Gradient-free: Completely bypasses backpropagation, eliminating gradient computation and storage overhead
• Memory-light: No need to store optimizer states and activation values
• Naturally parallel: No synchronization needed when parallelizing across devices, avoiding the communication bottleneck of distributed training
• Directly optimizes the target metric: Unlike policy gradient methods, LlMBO can directly optimize greedy decoding accuracy

Together, these characteristics mean that in low-dimensional fine-tuning scenarios, LlMBO can operate at a fraction of the computational and memory cost of gradient methods.

GRPO (Group Relative Policy Optimization) is a policy gradient method used for LLM reinforcement learning, often employed to enhance a model's reasoning capabilities (such as math and code). Its approach is to sample a group of responses to the same question and use the relative magnitudes of rewards within the group to estimate advantage values, thereby eliminating the value network (critic) required in traditional PPO.

Although GRPO is more resource-efficient than PPO, it still belongs to the policy gradient family: it requires forward + backward propagation, stores activations and optimizer states, and the reward signal in policy gradients typically only provides gradients in an expectation sense, making it difficult to directly optimize discrete, non-differentiable objectives like "greedy decoding accuracy". Moreover, when parallelizing across multiple devices, policy gradient methods often require synchronizing gradients or parameters, incurring communication overhead. LlMBO contrasts with GRPO precisely along these dimensions.

Experimental Comparison: LlMBO vs GRPO

On a suite of math benchmarks, the authors conducted fine-tuning experiments on multiple models, comparing LlMBO with GRPO (Group Relative Policy Optimization).

The results show that in this low-dimensional fine-tuning scenario, LlMBO achieves fine-tuning gains on par with or even better than GRPO, while using only a fraction of the latter's computation and memory.

The significance of this conclusion lies in the fact that when the parameter space is small enough, we need not be fixated on gradient methods—global search strategies can actually utilize resources more efficiently.

Understanding the Significance of This Shift

The value of this work lies not only in proposing a new method, but also in pointing out the boundaries of gradient methods' applicability.

The default assumption in large model training is that the parameter space is enormous and gradient information is indispensable. But work like TinyLoRA shows that effective fine-tuning may require only an extremely small number of parameters. Once in this low-dimensional regime, the computational complexity of global methods like Bayesian optimization becomes acceptable—even advantageous over local search.

This is a positive signal for researchers and developers with limited compute: fine-tuning LLMs doesn't necessarily require expensive GPU clusters; on specific tasks, clever optimization strategies can substantially lower the barrier. Of course, LlMBO is currently still limited to low-dimensional fine-tuning scenarios—for full-parameter training or large-scale adaptation, gradient methods remain irreplaceable.

Parameter-Efficient Fine-Tuning (PEFT) is a class of techniques that adapts to downstream tasks by training only a very small number of newly added or selected parameters, with representative methods including LoRA, prefix tuning, adapters, etc. LoRA reduces trainable parameters by introducing low-rank decomposition alongside weight matrices, but it is essentially still a gradient method, with the number of trainable parameters typically in the thousands to tens of thousands.

TinyLoRA further compresses trainable parameters to single digits or a few dozen, thereby reducing the optimization problem from high-dimensional to extremely low-dimensional. This dimensional leap brings about a methodological shift—in low-dimensional spaces, the feasibility of global search is greatly enhanced, and gradients are no longer the only option. It is precisely under this paradigm that LlMBO replaces gradient descent with Bayesian optimization, opening up a non-gradient route for PEFT.

One-line Summary

A new paper proposes LlMBO, using Bayesian optimization to replace gradient descent in low-dimensional fine-tuning

Paragraph Summary

This article introduces a new work, LlMBO (Large language Model Bayesian Optimization), which brings Bayesian optimization into LLM fine-tuning. Previously, TinyLoRA had already demonstrated that LLM fine-tuning can require fewer than 10 parameters, but it still relied on gradient descent. LlMBO seizes on the key insight that "global optimization is feasible in extremely low-dimensional parameter spaces", replacing gradient methods with Bayesian optimization. It features sample efficiency, gradient-freedom, memory lightness, synchronization-free cross-device parallelism, and the ability to directly optimize greedy decoding accuracy. Compared with GRPO on math benchmarks, LlMBO achieves on-par or even better fine-tuning gains while consuming only a fraction of the latter's computation and memory. The article notes that this work reveals the boundaries of gradient methods' applicability and is a positive signal for developers with limited compute, though it remains limited to low-dimensional fine-tuning scenarios.