LoRA
Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning technique for large models. Its core principle is to introduce low-rank decomposition matrices alongside the weight matrices of a pre-trained model, training only these additional parameters while keeping the original model weights frozen. This achieves task-specific adaptation at reduced computational and storage costs. LoRA is widely applied in the customized fine-tuning of large language models and image generation models.
Core Facts
LoRA reduces fine-tuning costs by training only a small number of adapter layers (low-rank matrix bypass training).
LoRA achieves fine-tuning by inserting low-rank decomposition matrices alongside the model's weight matrices, typically requiring training of only 0.1%–1% of the original parameter count.
LoRA (Low-Rank Adaptation) was proposed by Microsoft in 2021, with the core idea of freezing the weights of the pretrained model and only training injected low-rank decomposition matrices.
Timeline (last 90 days)
For each token, the log-likelihood ratio under the two LoRA adapters is computed as local evidence of whether that position was generated or modified by AI.
The framework uses a frozen code language model equipped with two low-rank adapters (LoRA), trained separately on human-written files and machine-written files.
Low-rank experts are lightweight adapter modules constructed via low-rank decomposition (such as LoRA); each expert contains only a small number of trainable parameters and is inserted into specific layers of the frozen backbone to learn domain-specific residual adjustments.
Speaker-independent experiments on ADReSS and ADReSSo show that dual-task vector merging outperforms frozen probing, LoRA, and full fine-tuning.
On the PROCESS dataset, when paired with LoRA, the advantage of this method transfers to the three-class task of HC/MCI/dementia.
Dual-task vector merging works best when combined with LoRA, indicating that the two parameter-efficient methods are complementary.
The system executes all three multiplication operations involving frozen weights on the NPU, while keeping the training graph, adapters, and optimizer on the CPU.
During LoRA backward propagation there is no need to compute the gradient of W, but the gradient of the loss with respect to x (the input gradient) is still needed to continue propagating back to shallower layers, and this step must use W itself.
LoRA greatly reduces VRAM requirements and training costs, making it possible to fine-tune large models on a single consumer-grade GPU.
WeAct trains multiple capability-specific LoRA adapters on a frozen backbone model, with each adapter corresponding to one capability; adapter selection is modeled as an internal action of the agent, on par with calling external tools.
40 more timeline events
All Facts (20)
LoRA reduces fine-tuning costs by training only a small number of adapter layers (low-rank matrix bypass training).
90%VerifiedLoRA achieves fine-tuning by inserting low-rank decomposition matrices alongside the model's weight matrices, typically requiring training of only 0.1%–1% of the original parameter count.
90%VerifiedLoRA (Low-Rank Adaptation) was proposed by Microsoft in 2021, with the core idea of freezing the weights of the pretrained model and only training injected low-rank decomposition matrices.
90%VerifiedThe VRAM usage and inference latency of MoE architectures are calculated based on the number of activated parameters, not the total parameter count.
90%VerifiedUnsloth is an open-source tooling team focused on large model quantization and fine-tuning acceleration, providing GGUF quantized versions for mainstream open-source models.
85%VerifiedQLoRA was proposed by a research team at the University of Washington in 2023, quantizing base model weights to 4-bit (NF4 format)
85%VerifiedLoRA decomposes the weight update matrix ΔW into the product of two small matrices A and B whose ranks are much lower than the original matrix, reducing parameter count to as little as one ten-thousandth of the original or even less.
80%VerifiedLoRA is a lightweight model fine-tuning technology that can train a dedicated character model with only 20–50 reference images, requiring far less compute than full fine-tuning.
80%VerifiedLoRA and QLoRA are the currently mainstream parameter-efficient fine-tuning methods, allowing fine-tuning to be completed on consumer-grade GPUs.
80%VerifiedLoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method proposed by Microsoft Research in 2022.
80%VerifiedLoRA (Low-Rank Adaptation) was originally proposed by Microsoft Research for efficient adaptation of large language models.
80%VerifiedLoRA (Low-Rank Adaptation) files are typically only tens of MB in size, far smaller than the several GB size of complete models.
80%VerifiedNeural Tangent Kernel (NTK) theory states that when the width of a neural network approaches infinity, the network's behavior during gradient descent training is equivalent to kernel regression with the NTK as the kernel, and the network does not perform feature learning.
75%VerifiedThe principle of LoRA is to decompose the weight change ΔW during fine-tuning into the product of two low-rank matrices (ΔW = A×B).
75%VerifiedDuring LoRA training, the pre-trained model weights W₀ are frozen and only the two matrices B and A are trained, with the forward propagation formula H = W₀X + (α/r)BAX.
75%VerifiedThe three main strategies for mitigating catastrophic forgetting are regularization-based methods (such as EWC), replay-based methods (such as Experience Replay), and architecture-based methods (such as Progressive Neural Networks).
75%VerifiedThe core idea of PEFT is to not update all parameters, but instead introduce a small number of trainable adapter parameters (such as LoRA's low-rank matrix decomposition), reducing VRAM and computation requirements while retaining performance close to full fine-tuning.
75%VerifiedLoRA reduces the number of trainable parameters by injecting low-rank decomposition matrices into the model's weight matrices, making it possible to fine-tune models with billions of parameters on consumer-grade GPUs.
75%VerifiedQLoRA quantizes the base model to 4-bit precision for storage, while applying LoRA adapters on top of the quantized model for training.
75%VerifiedQLoRA quantizes pre-trained model weights to 4-bit, making it possible to fine-tune a 65-billion-parameter large language model on a single consumer-grade GPU with 24GB of VRAM
75%