Diffu-LoRA

Diffu-LoRA is a parameter-efficient diffusion model personalization method whose core idea is to learn how adaptation capacity is distributed across different network layers through gated low-rank adaptation.

Diffu-LoRA inserts trainable low-rank components into the linear layers of Transformer blocks and assigns each component a learnable gate to measure its importance.

Diffu-LoRA adopts bilevel optimization, where the adaptation weights and gate parameters are updated separately on different data splits.

Diffu-LoRA employs progressive pruning, gradually removing the components with the lowest gate values during training until a preset rank budget is satisfied.

During Diffu-LoRA training, the pretrained backbone network remains frozen throughout.

The research team validated Diffu-LoRA on Stable Diffusion using the DreamBooth dataset and additionally collected datasets.

Compared to the evaluated fine-tuning baseline methods, Diffu-LoRA achieved improvements on both overall subject fidelity and prompt alignment.

The paper examines the contributions of bilevel optimization, progressive pruning, and adapter placement positions through ablation studies.

Diffu-LoRA proposes that rank allocation itself should be a learnable objective rather than a fixed hyperparameter.

As a newly released preprint, Diffu-LoRA still needs to be validated on larger-scale diffusion models and more diverse subjects, and the additional training overhead introduced by bilevel optimization is a factor to weigh in practical applications.