LoRA
Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning technique for large models. Its core principle is to introduce low-rank decomposition matrices alongside the weight matrices of a pre-trained model, training only these additional parameters while keeping the original model weights frozen. This achieves task-specific adaptation at reduced computational and storage costs. LoRA is widely applied in the customized fine-tuning of large language models and image generation models.
Core Facts
LoRA reduces fine-tuning costs by training only a small number of adapter layers (low-rank matrix bypass training).
LoRA achieves fine-tuning by inserting low-rank decomposition matrices alongside the model's weight matrices, typically requiring training of only 0.1%–1% of the original parameter count.
LoRA (Low-Rank Adaptation) was proposed by Microsoft in 2021, with the core idea of freezing the weights of the pretrained model and only training injected low-rank decomposition matrices.
Timeline (last 90 days)
A developer who does LoRA fine-tuning for enterprises trained a customer service support model for a client, and when answering simple questions it output step-by-step reasoning and Python code blocks, causing the demo to be postponed.
The loss during training is calculated per token, and long samples contribute far more gradient than short samples, so a 10% share of rows can absolutely correspond to more than 50% of the token volume and gradient signal.
Applying LoRA to all linear layers means adapters are injected into the Q/K/V/O projections of the attention layers and the up and down projections of the feedforward layers, giving broad coverage and strong expressiveness, but also making it easier to learn the stylistic features in the training data as a whole.
LoRA adapters can be saved independently and hot-swapped, and multiple business adapters can be attached to the same base model, making them popular in multi-client delivery scenarios.
There is tension between supporting the model's dialogue generalization ability and its domain fit: previously mixing in general data was to remedy the side effects of narrow-domain training, and tightening replay will bring part of that problem back.
VKORE enabled LoRA Transformer training to be successfully completed on an NPU, which was previously infeasible on an NPU.
Executing a certain LoRA training task on a server CPU requires about 320 hours, while on an NVIDIA B200 GPU it takes only about 1 hour.
Solutions to the character consistency problem include: using character reference images as conditional input, training lightweight LoRA or DreamBooth models, and post-processing alignment via facial embedding vectors.
LoRA is a lightweight model fine-tuning technique, originally used for large language models, later introduced into the diffusion model ecosystem, which learns new styles or concepts at low cost by injecting low-rank matrices as a side path.
LoRA freezes the base model weights and learns low-rank updates, but does not guarantee that all prior behaviors are preserved, requiring regression testing beyond the target task.
40 more timeline events
All Facts (20)
LoRA reduces fine-tuning costs by training only a small number of adapter layers (low-rank matrix bypass training).
90%VerifiedLoRA achieves fine-tuning by inserting low-rank decomposition matrices alongside the model's weight matrices, typically requiring training of only 0.1%–1% of the original parameter count.
90%VerifiedLoRA (Low-Rank Adaptation) was proposed by Microsoft in 2021, with the core idea of freezing the weights of the pretrained model and only training injected low-rank decomposition matrices.
90%VerifiedThe VRAM usage and inference latency of MoE architectures are calculated based on the number of activated parameters, not the total parameter count.
90%VerifiedUnsloth is an open-source tooling team focused on large model quantization and fine-tuning acceleration, providing GGUF quantized versions for mainstream open-source models.
85%VerifiedQLoRA was proposed by a research team at the University of Washington in 2023, quantizing base model weights to 4-bit (NF4 format)
85%VerifiedLoRA decomposes the weight update matrix ΔW into the product of two small matrices A and B whose ranks are much lower than the original matrix, reducing parameter count to as little as one ten-thousandth of the original or even less.
80%VerifiedLoRA and QLoRA are the currently mainstream parameter-efficient fine-tuning methods, allowing fine-tuning to be completed on consumer-grade GPUs.
80%VerifiedLoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method proposed by Microsoft Research in 2022.
80%VerifiedLoRA (Low-Rank Adaptation) was originally proposed by Microsoft Research for efficient adaptation of large language models.
80%VerifiedLoRA (Low-Rank Adaptation) files are typically only tens of MB in size, far smaller than the several GB size of complete models.
80%VerifiedThe principle of LoRA is to decompose the weight change ΔW during fine-tuning into the product of two low-rank matrices (ΔW = A×B).
75%VerifiedThe three main strategies for mitigating catastrophic forgetting are regularization-based methods (such as EWC), replay-based methods (such as Experience Replay), and architecture-based methods (such as Progressive Neural Networks).
75%VerifiedThe core idea of PEFT is to not update all parameters, but instead introduce a small number of trainable adapter parameters (such as LoRA's low-rank matrix decomposition), reducing VRAM and computation requirements while retaining performance close to full fine-tuning.
75%VerifiedLoRA reduces the number of trainable parameters by injecting low-rank decomposition matrices into the model's weight matrices, making it possible to fine-tune models with billions of parameters on consumer-grade GPUs.
75%VerifiedQLoRA quantizes the base model to 4-bit precision for storage, while applying LoRA adapters on top of the quantized model for training.
75%VerifiedStable Diffusion Checkpoint models are typically 2–7GB, while LoRA models are typically only tens to hundreds of MB.
75%VerifiedLoRA (Low-Rank Adaptation) can reduce the number of trainable parameters by more than 99%.
75%VerifiedLoRA attaches two low-rank matrices A and B in parallel alongside each target weight matrix, where A is randomly initialized and B is initialized to zero; only these two small matrices are trained, and during inference the product A×B is added back to the original weights.
70%VerifiedFor a 7-billion-parameter model, LoRA typically requires training less than 1% of the total parameters, and GPU memory requirements can be reduced by more than 60%.
70%