Training a Text-to-Image Model on 4x 3090s: MiniDog's Full Pipeline in Just 70 Minutes

How much compute does it take to train a text-to-image diffusion model? Most people's intuition says: at least a GPU cluster. The MiniDog project breaks that barrier—on four RTX 3090s, 55 minutes of pretraining plus another 15 minutes of fine-tuning is enough to produce a diffusion Transformer that can generate images of specific dog breeds.

Why text-to-image training seems so expensive

When people hear "training a text-to-image diffusion model," they think of Stable Diffusion or FLUX-level engineering: thousands of GPU hours, massive image-text pairs, complex distributed training pipelines. That impression isn't wrong, but it only describes the training cost of frontier models.

The problem is that this perception has a side effect—beginners and small teams are locked out, convinced they'll never be able to train a diffusion model from scratch. The authors of MiniDog clearly reject that premise. Their thinking: if you keep model scale, data volume, and engineering optimizations under control, consumer-grade hardware is entirely capable of running the full training pipeline.

This judgment matters for AI education and small-scale research alike: not everyone has an A100 cluster, but plenty of people have three or four 3090s.

Data strategy: pretraining and fine-tuning each play their part

MiniDog's data approach has two stages, and the design is thoughtful.

The pretraining stage uses 26,000 dog images from ImageNet with generated text descriptions. ImageNet's class labels naturally carry breed information, which makes image-text alignment relatively easy to establish.

The fine-tuning stage takes a different route—using 2,000 images generated by FLUX.1-schnell for adaptation. The goal here isn't to teach the model what a dog is, but to align the representations learned during pretraining with the target generation style. Using generated images for fine-tuning rather than real photos is a pragmatic choice: zero data acquisition cost, controllable quality, and no copyright or cleaning issues from real datasets.

To improve training efficiency, MiniDog precomputes and caches image and text features. This means the training loop doesn't need to repeatedly pass through encoders—it just reads cached features directly. This is an extremely practical optimization for small-scale training.

FLUX.1-schnell is the fast distilled version of the FLUX series released by Black Forest Labs, capable of generating high-quality images in very few steps (typically 1–4), under the Apache 2.0 license that permits commercial use and redistribution. MiniDog uses it to batch-generate 2,000 dog breed images as fine-tuning data—so-called synthetic data fine-tuning. The key consideration is that collecting, cleaning, and copyright-reviewing real image-text pairs is costly, whereas synthetic data can be generated on demand, has consistent style, and carries no copyright disputes. The risk is that the model may inherit FLUX's own biases and style, causing the fine-tuned output to carry an obvious "FLUX flavor"—which is exactly why MiniDog's output style is influenced by the fine-tuning data.

Architecture choices: every piece in service of efficiency

MiniDog's architecture design revolves around one core principle—minimize parameter count and compute while maintaining generation quality.

On the VAE side, it uses E2E-INVAE, chosen for its semantic latent space. Compared to conventional VAEs, a semantic latent space means the latent variables themselves carry more interpretable structural information, which helps downstream generation tasks.

The transformer backbone has only 22 million parameters. This scale is extremely small for the diffusion model world—for reference, mainstream text-to-image models typically have parameter counts in the billions. MiniDog clearly gives up generalization ability in exchange for trainability on consumer-grade hardware.

Text encoding uses a small Qwen model, combined with in-context conditioning for condition injection. Using a small model for text encoding is a reasonable trade-off: with only 26,000 training images, the text distribution is relatively concentrated, and a large encoder would bring limited benefit.

On the training acceleration side, it introduces the iREPA and EUPE perceptual encoders. iREPA is used for representation alignment, while EUPE provides perceptual-level supervision signals. Working together, they let the model converge in fewer steps.

The evaluation pipeline covers FID, HPSv2, PickScore, and various VAE reconstruction metrics. The evaluation dimensions are quite complete, showing that the authors weren't just aiming for "it runs"—they also care about quantifiable comparison of generation quality.

Diffusion Transformer (DiT) is the basic architectural paradigm adopted by MiniDog's backbone. Traditional diffusion models mostly use U-Net as the denoising network, while DiT replaces U-Net with a pure Transformer, slicing the latent image into patch sequences and performing self-attention as if processing tokens. DiT's advantage lies in its clear scaling laws—the relationship between parameter count, compute, and generation quality is predictable, which is why it has become the foundation of new-generation models like Stable Diffusion 3 and FLUX. MiniDog chooses DiT over U-Net partly to reuse mature Transformer training techniques, and partly because at such an extremely small scale of 22 million parameters, the Transformer's structure is simpler and cheaper to implement and debug.

In-context conditioning means the text condition isn't injected separately through cross-attention, but instead concatenated with image tokens into the same sequence, letting the model naturally fuse image and text information through self-attention. This approach eliminates the extra cross-attention module, making the structure lighter—suitable for small models. The cost is longer sequences and increased attention computation, but that's entirely acceptable at MiniDog's scale.

iREPA (representation alignment) is a class of representation alignment techniques. The core idea is to align the features of intermediate layers in the diffusion model with the features of a pretrained visual encoder (such as DINOv2), essentially giving the denoising network a "semantic teacher." With this alignment supervision, the model doesn't have to learn visual semantics from scratch, significantly accelerating convergence—especially effective under few-step training. EUPE, meanwhile, provides supervision at the perceptual level, focusing on the plausibility of generated images in human visual perception, complementing iREPA's semantic alignment. Together, they enable MiniDog to reach usable generation quality within 55 minutes of pretraining—something pure end-to-end training struggles to achieve.

What can be trained in 70 minutes

Back to the most practical question: how well does the model trained with this approach actually perform?

Honestly, a 22-million-parameter model trained for 55 minutes on 26,000 images cannot generate photorealistic arbitrary images. Its capability boundary should be centered on dog-breed-related image generation, with style influenced by the FLUX.1-schnell fine-tuning data.

But that's precisely where MiniDog's value lies: what it validates isn't "can a small model replace a large model," but "can the complete text-to-image training pipeline run on consumer-grade hardware." For anyone wanting to learn the full diffusion model training process, this is far more useful than reading paper equations—you can run it yourself, from data preparation to evaluation, and touch every step.

The authors designed the repository to support a complete set of evaluation metrics, which also shows this is a framework oriented toward teaching and research rather than a product-grade solution.

The evaluation metrics mentioned each have their own focus: FID (Fréchet Inception Distance) measures the statistical distance between generated and real distributions—lower is better—but it's heavily influenced by the feature extractor and sample size, with noticeable variance on small datasets; HPSv2 and PickScore are both scoring models trained on human preferences, used to assess image aesthetics and text alignment, closer to subjective perception; VAE reconstruction metrics (such as PSNR and LPIPS) separately assess the fidelity of the VAE after encoding and decoding an image, because diffusion models denoise in the latent space, and the VAE's reconstruction quality directly determines the generation ceiling. A complete evaluation covering all three dimensions—distribution distance, human preference, and reconstruction fidelity—is needed to more comprehensively judge whether a small model can "really generate," rather than just looking at a single sample.

Takeaways for practitioners

The most noteworthy aspect of MiniDog is that it demonstrates a "good enough" engineering philosophy. The current AI community's obsession with scale easily obscures a fact: many learning, validation, and prototyping scenarios simply don't need frontier-level models.

Four 3090s, 70 minutes, one complete training pipeline—the significance of this combination isn't in the performance numbers, but in turning "training a text-to-image model" from something that requires applying for a compute budget into something you can do by hand in an afternoon. For education and entry-level research, this improvement in accessibility may matter more than the model's own metrics.

The project code and evaluation tools are open-sourced. If you're interested, you can reproduce it yourself and feel the actual rhythm of the entire pipeline.

One-line Summary

MiniDog runs the complete text-to-image diffusion model training pipeline in 70 minutes on four RTX 3090s

Paragraph Summary

The MiniDog project proves that training a text-to-image diffusion model doesn't require a large-scale GPU cluster—on four RTX 3090s, 55 minutes of pretraining plus 15 minutes of fine-tuning is enough to complete a diffusion Transformer that generates images of specific dog breeds. Its data strategy has two stages: pretraining on 26,000 ImageNet dog images with text descriptions, fine-tuning on 2,000 synthetic images generated by FLUX.1-schnell for style alignment. Architecturally, it uses a 22-million-parameter small transformer, E2E-INVAE semantic latent space, and a small Qwen text encoder, with feature caching, iREPA, and EUPE accelerating convergence. Evaluation covers FID, HPSv2, PickScore, and VAE reconstruction metrics. The project's real value lies in validating that the complete training pipeline can run on consumer-grade hardware, providing a reproducible framework for teaching and entry-level research.