LARA

LARA (Lightweight Additive Residual Adaptation) is an open-source project shared by researchers on Reddit, with the goal of making post-training modular by stacking pluggable behavioral capabilities onto frozen language models without modifying the base model's weights.

LARA trains a low-rank residual adapter on selected layers rather than directly modifying the model's original weights.

Behaviors learned by LARA can be loaded, removed, blended, or routed during inference while the model itself remains frozen.

LARA's Mixture of Behaviors (MoBs) demo allows multiple independently trained behaviors to share the same frozen base model and selects or combines these behaviors one by one at the token level through a soft router.

In an example given by the LARA authors, a single model can simultaneously possess four independent behaviors—coding, mathematics, medical, and summarization—without needing to maintain four separately fine-tuned independent models.

LARA's soft router performs selection and weighted fusion only at the lightweight adapter level; the experts themselves are low-rank matrices rather than complete networks, and adding or removing a behavior requires no retraining of the router or other existing behaviors.

The LARA project repository includes a direct comparison with LoRA as an evaluation reference baseline.

LARA's second demo trained behavioral styles imitating the writing of Hemingway, Fitzgerald, and Gertrude Stein.

LARA is still an ongoing research project, but the accompanying PyTorch library is already available, and the repository provides complete training code, examples, and instructions for reproducing paper results.

For scenarios requiring a single base model to serve multiple tasks, LARA's architecture of one model plus multiple lightweight behaviors can effectively reduce operational complexity.