LoRA Training Showdown: Which Base Model Is Best for Character Fine-Tuning? A 6-Model Real-World Test

A Four-Day LoRA Training Marathon

As new text-to-image models continue to emerge, developers face a practical question: which base model is best suited for training character LoRAs? One developer running an AI portrait generation site — with two 4070Ti GPUs in his basement plus a rented 5090 and H100 on RunPod — ran a continuous four-day training experiment to find out.

First, a quick primer on the technology itself. LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique proposed by Microsoft Research in 2021. The core idea: instead of modifying the original model weights, small low-rank matrices are inserted alongside attention layers to adapt the model. Compared to full fine-tuning, LoRA trains only a tiny fraction of parameters (typically less than 1% of the original model), dramatically reducing VRAM requirements and training time. In the text-to-image space, LoRA is widely used to train characters, styles, and custom concepts — users need only a dozen to a few dozen reference images to make a model "remember" a specific person or style. The rank value is LoRA's core hyperparameter, determining the dimensionality of the low-rank matrices: higher values mean greater expressive capacity, but also a higher risk of overfitting and increased VRAM usage.

This developer's site has long run on Flux 1 Dev, primarily serving AI portrait and avatar use cases. He noticed a pattern: after new models launched, the number of character LoRAs shared on CivitAI was far less active than during the Flux 1 Dev era. Rather than passively waiting for community feedback, he decided to run the tests himself using idle GPU compute.

Worth noting: the entire training pipeline was handed almost entirely to Claude — from hyperparameter tuning (rank, learning rate, text encoder learning rate, etc.) to debugging. What traditionally required engineers to manually adjust over a dozen hyperparameters and painstakingly interpret training logs was handled by an AI agent. The agent reads error logs, retrieves relevant documentation, proposes configuration changes, and applies them automatically — a closed-loop process that essentially converts "experiential knowledge" into an automated reasoning chain. This reflects a broader trend: AI-assisted engineering is dramatically lowering the technical barrier to model training, enabling individual developers to run large-scale comparative experiments at relatively low cost.

Test Subjects and Methodology

The test covered six mainstream models: Krea, Ideogram, Flux 1 Dev, Flux 2 Dev, Flux 2 Klein, and Z Image. Except for Flux 2 Dev — which required too much VRAM due to its Mistral encoder — all other models fit comfortably within 16GB of VRAM. Flux 2 Dev was moved to the 5090 for two attempts, but results were poor, and it was ultimately abandoned mid-test due to RunPod compute costs.

The reason Flux 2 Dev requires so much VRAM comes down to its text encoder architecture. The Flux series originally used CLIP and T5 as dual text encoders, but Flux 2 Dev reportedly introduces a large language model based on the Mistral architecture as its text encoder. Mistral is an autoregressive language model with roughly 7 billion parameters — its embedding dimensions and sequence processing capacity far exceed T5-XXL, enabling much richer understanding of complex prompts. However, this comes with a significant VRAM cost: during training, the text encoder's activations must reside in VRAM at all times, making 16GB GPUs insufficient. This design tradeoff reflects a common tension in modern multimodal models: stronger language understanding often comes at the cost of higher hardware requirements, creating a real barrier for consumer GPU users.

The Logic Behind Subject Selection

The developer's choice of training subjects was particularly insightful. The first subject was an ordinary white woman — most models handle such faces well and are unlikely to produce bad results.

The second subject was deliberately more challenging. Based on his experience, South Asian and Black faces tend to reproduce poorly across most models, with models prone to overfitting and even degrading into distorted, stereotyped renderings during later training stages. There's a deep technical reason for this: the training datasets behind mainstream text-to-image models (such as LAION-5B) are severely imbalanced across ethnicities — white faces dominate, leaving the model with an underdeveloped feature space for other groups. This bias gets amplified during LoRA training: models have rich "prior knowledge" to fine-tune for white faces, while their internal representations of South Asian or Black faces are sparse to begin with, making them more prone to collapsing into stereotyped distortions. By selecting this subject, the developer was effectively testing each model's fairness and generalization limits — a truly capable base model shouldn't show significant disparities when handling different skin tones and facial features.

Iterating from v1 to v3

The evaluation methodology itself went through several rounds of refinement. The first two versions (v1, v2) used prompts that were too simple — if a model overfit, it would just mechanically return something close to the input images, revealing nothing about actual learning quality. Overfitting is the most common failure mode in LoRA training: when a model performs too "precisely" on the training set, it's essentially memorizing pixel-level features of the training images rather than learning the subject's fundamental characteristics (skeletal structure, skin texture patterns, facial proportions). In novel scenes, the character breaks down. Version v3 switched to more complex prompts to test whether the model could reconstruct the subject's features in entirely new scenarios, not just recall a handful of training images. This improvement was critical for evaluation rigor — the ability to generalize to novel scenarios is the core standard for measuring character LoRA quality.

Final Rankings and Key Conclusions

After four days of continuous training, the developer arrived at a clear ranking:

Ideogram > Flux 1 Dev > Z Image > Flux 2 Klein > Krea > Flux 2 Dev

The most surprising finding: Ideogram won — and by a clear margin. The developer described Ideogram as a significant improvement over Flux 1 Dev, and said he plans to migrate his production pipeline from Flux 1 Dev to Ideogram.

The only concern is inference speed — his current production Flux 1 Dev LoRA runs on an H100, and Ideogram's inference latency on H100 remains unknown. This is a reminder that model selection isn't just about generation quality; deployment cost and inference efficiency must also factor into the decision.

The Disappointing Flux 2 Dev

As the flagship of the Flux series, Flux 2 Dev landing at the bottom of the rankings was genuinely surprising. The developer candidly said the training "felt cursed from the start," and publicly asked the community to share better-optimized Flux 2 Dev training configurations.

This result deserves some nuance. Flux 2 Dev's high VRAM demands from the Mistral encoder already create a significant barrier for training environment setup. The developer only ran two training attempts on the 5090 before abandoning it due to cost — a very limited sample size. Furthermore, LoRA training is extremely sensitive to learning rate and training steps — too high a learning rate or too many steps accelerates memorization, and optimal hyperparameters for new architectures often require extensive experimentation to determine. Flux 2 Dev placing last more likely reflects under-optimized training configurations rather than any fundamental capability limitation — an important caveat to keep in mind when interpreting these results.

Practical Takeaways for AI Creators

Though conducted by one person with a limited sample, this experiment offers genuinely useful reference points for developers working in AI portraits and character generation.

First, newer models don't necessarily mean better LoRA training performance. Flux 2 Dev, the new flagship, came in last, while the more established Ideogram and Flux 1 Dev performed reliably. This suggests that a base model's "trainability" is a dimension independent of its raw generation quality — one worth evaluating separately.

Second, evaluation methodology determines the credibility of conclusions. The iteration from simple prompts to complex scenario testing directly affected the ability to detect overfitting. Anyone running their own character LoRA evaluations should design the testing framework first, ensuring prompts cover out-of-distribution scenarios that genuinely test generalization rather than memorization.

Third, AI agents are reshaping training workflows. Handing the entire pipeline to Claude for automated hyperparameter tuning and debugging allowed a single developer to complete a large-scale comparative experiment at low cost — a concrete example of AI engineering being put to practical use. Worth noting, however, is a potential risk: as hyperparameter tuning itself becomes automated, practitioners' intuitive understanding of these parameters may gradually atrophy, creating a new form of capability dependency.

Of course, these rankings are not the final word. Only two training subjects, some models abandoned mid-test, configurations likely not fully optimized — these are real limitations to keep in mind. But as firsthand practical experience, it at least provides a pragmatic starting point: if you're choosing a base model for character LoRA training, Ideogram is worth considering first.

Key Takeaways