Directly Optimizing Face Similarity: A Practical Guide to Accelerating Character LoRA Training

Introduction: When LoRA Training Meets Differentiable Face Similarity

In the field of AI image generation, training character-specific LoRAs (Low-Rank Adapters) is a core technique for keeping a particular person's face consistent within a generative model. LoRA was introduced by Microsoft Research in 2021, and its core idea is to decompose the weight-update matrix into the product of two low-rank matrices (W = BA, where rank r is far smaller than the original dimension), thereby compressing the number of trainable parameters from billions down to the millions. Within the Stable Diffusion ecosystem, LoRA is widely used to inject specific characters, art styles, or concepts. Its training cost is extremely low—typically completing in anywhere from a few minutes to a few hours on consumer-grade GPUs—while leaving the original model weights intact. The underlying mathematical principle ensures that the gradients of the low-rank matrices can be computed and backpropagated efficiently, which is precisely the key property exploited in this article: the entire optimization pipeline remains fully differentiable.

The traditional approach uses standard supervised fine-tuning (SFT), having the model learn to predict noise or the velocity field. However, while exploring reinforcement-learning-related techniques, GitHub developer KONAKONA666 discovered a key fact that had long been overlooked: the entire face-similarity computation pipeline is itself differentiable.

This insight led to a rather ingenious approach—rather than making the LoRA take a detour to learn noise prediction, why not optimize directly for "face similarity" as the objective? The training objective shifts from an indirect reconstruction loss to directly aligning the distance between face feature embeddings. The author combined this idea with the paper DRaFT (arxiv.org/abs/2309.17400) to implement a fast and stable character LoRA training pipeline.

Core Idea: From Predicting Noise to Directly Aligning Faces

Limitations of Traditional SFT

The training logic of standard SFT has the model predict noise or velocity during the diffusion process. While this approach is general-purpose, it is clearly indirect for highly specific objectives like "character consistency"—the model must indirectly "sculpt" facial features into its weights through many steps, which is inefficient and often requires more steps to converge.

Differentiable Face-Similarity Loss

The author's key change is this: designing a loss function that directly computes the distance between the face embedding of the generated image and the target face embedding. Achieving "end-to-end differentiable" face-similarity computation relies on a processing pipeline composed entirely of differentiable operations: (1) Face detection: using a deep-learning-based detector (such as RetinaFace or MTCNN) that outputs face bounding boxes and keypoints, fully differentiable throughout; (2) Face alignment: cropping via an affine transformation based on keypoints, for which the Spatial Transformer Network provides a differentiable implementation; (3) Feature extraction: using a face-recognition network such as ArcFace, FaceNet, or AdaFace to encode the aligned face image into a 512-dimensional or 128-dimensional embedding vector; (4) Similarity computation: typically using cosine similarity, which is inherently differentiable. The entire pipeline involves no non-differentiable sampling or argmax operations, so the automatic differentiation engines of PyTorch/JAX can directly compute the full gradient from the similarity loss to the LoRA parameters—a fundamental difference from traditional image-processing pipelines that rely on non-differentiable OpenCV operations.

Because the entire pipeline—from face detection and feature extraction to similarity computation—is differentiable, gradients can be backpropagated all the way to the LoRA parameters. This means the LoRA is trained to "precisely" maximize face similarity, rather than to reconstruct images in a general sense.

This idea captures the very spirit of DRaFT (Direct Reward Fine-Tuning). Proposed by Kevin Black et al. in 2023, DRaFT is a representative work that directly brings the "reward maximization" idea from reinforcement learning into diffusion model fine-tuning. Traditional RLHF methods usually require training a separate reward model and going through a sample-evaluate-update loop, which is extremely computationally expensive. DRaFT's innovation lies in exploiting the differentiability of the diffusion model's denoising process, backpropagating the gradient of the reward function directly to the model parameters, without requiring reinforcement learning's policy gradient estimators (such as REINFORCE)—thereby avoiding the high-variance problem. The paper also proposes a truncated backpropagation strategy: backpropagating gradients only through the last K denoising steps rather than unrolling the entire process, dramatically reducing memory usage. The "500-step DRAFT + 60-step fine-tuning" described in this article is precisely the engineering embodiment of this truncation strategy.

Performance Comparison: Fewer Steps, Better Results

To fairly compare compute budgets, the author designed a controlled experiment:

DRAFT approach: 500 steps + 60 steps of fine-tuning
Pure SFT approach: 1000 steps

Approach	Time per step	Notes
SFT	0.5 sec/step (2 steps/sec)	Noise prediction only
DRAFT	4.11 sec/step	Includes image generation + VAE decoding + face detection + loss and backpropagation

DRAFT's per-step time is roughly 8x that of SFT, because each step requires actually generating an image, decoding, detecting the face, and computing the similarity loss. But because the total number of steps is greatly reduced, the entire training completes in only about 10-12 minutes on an RTX 4090. The author notes that DRAFT demonstrates "unexpected speed and stability" on almost any dataset, with results far superior to traditional SFT.

Test conditions: 512×512 resolution, batch size of 1, 12 sampling steps during training, GPU is an RTX 4090.

Engineering Details: Mixed Precision and Anti-Overfitting Techniques

INT8 + bf16 Mixed Precision Strategy

Mixed precision training is a key technique in modern deep learning engineering for controlling memory usage and computational efficiency. Standard AMP approaches use fp16 or bf16 for forward/backward computation while retaining fp32 "master weights" for parameter updates to prevent gradient underflow. This article's INT8+bf16 strategy goes a step further: the original model weights use INT8 quantization (each parameter takes only 1 byte, saving 50% memory compared to fp16), implemented via dynamic quantization-dequantization from the bitsandbytes library (the LLM.int8() method); while the LoRA portion uses bf16 (paired with fp32 master weights) to guarantee the numerical precision of the adapter layers. The rationale for this tiered strategy is that the base model is frozen during training, so precision loss in its weights has limited impact on the final output; meanwhile, the LoRA parameters are small (typically less than 1% of the total), so the cost of maintaining them at high precision is entirely acceptable. For the RTX 4090 (24GB of memory), this strategy allows SDXL-level model training to run smoothly on a single card, making it a practical solution for efficient training on consumer-grade GPUs.

Anti-Overfitting Design

A potential risk of directly optimizing face similarity is overfitting—the generated results may "look alike" but sacrifice diversity or introduce artifacts. Through techniques such as the staged 500+60 step training design, the author strikes a balance between similarity and generalization ability.

Inference-Side Deployment

For inference in ComfyUI, the author uses an INT8-quantized turbo model paired with the character LoRA. The complete code has been open-sourced to the GitHub repository KONAKONA666/krea-2 and can be reproduced directly.

In-Depth Commentary: A Landing Example of Differentiable Reward Fine-Tuning

The value of this practice lies not only in "faster training," but more importantly in demonstrating how to apply the idea of differentiable reward fine-tuning to a concrete, verifiable task. Face similarity is naturally a clear, quantifiable, and differentiable reward signal, making DRaFT-style methods almost "tailor-made" for this scenario.

Traditional SFT compresses all objectives into a general reconstruction loss, which is a waste of information for independently measurable attributes like "face consistency." When the objective function can be explicitly written out, direct optimization is often more efficient than indirect learning—this is precisely why differentiable rewards are gaining increasing attention in generative model fine-tuning. Compared to RLHF approaches that require human-annotated feedback, direct optimization schemes based on differentiable reward functions demonstrate stronger engineering feasibility and reproducibility on specific tasks (such as face consistency, aesthetic scoring, and text-image alignment).

Of course, this method has its boundary conditions: it relies on a high-quality, differentiable face-similarity model as the reward source; the per-step computation cost is relatively high; and it may not be applicable to attributes with unclear reward signals, such as art style and composition. But as a specialized solution for character LoRA training, it offers a highly inspiring direction.

Conclusion

From "predicting noise" to "directly aligning faces," this shift reflects the evolution of thinking in generative model fine-tuning: when an objective can be clearly measured, direct optimization is often far more effective. This developer's experiment, born from exploring RL, unexpectedly provided an efficient, stable, and reproducible open-source solution for character LoRA training. For creators and developers who need to frequently train character models, it is well worth a try.