AI Image Generation in Practice: From Diffusion Models to Agents

AI image generation is moving from the lab into production. Diffusion models laid the foundation for generation quality, and agents are now reshaping the entire workflow. This article, based on a practical guide on Hacker News, maps out the key path from diffusion principles to agent orchestration, helping you understand how this tech stack is actually deployed.

Diffusion Models: The Engine Behind Image Generation

The core idea of diffusion models is not complicated: gradually add noise to an image until it becomes pure noise, then train a network to learn the reverse denoising process. At inference time, you start from random noise and iterate over multiple steps to reconstruct an image. This "denoising" process gives the model powerful detail-generation capabilities, and it is the common foundation behind products like Stable Diffusion and Midjourney.

Understanding diffusion models means grasping a few engineering variables: sampling steps determine the trade-off between quality and speed—more steps yield richer detail but increase time linearly; the guidance scale (CFG Scale) controls how closely the output adheres to the prompt, and too high a value causes oversaturated colors and rigid compositions; the seed determines reproducibility, which is especially critical when you need to batch-produce assets in a consistent style. In practice, treating these parameters as adjustable knobs rather than a fixed recipe is more valuable than memorizing "optimal parameters."

Guidance Scale (CFG Scale) stands for Classifier-Free Guidance Scale, a key parameter during diffusion model inference. It works by computing both a "conditional prediction" (based on the prompt) and an "unconditional prediction" (empty prompt) during sampling, then extrapolating via the formula final prediction = unconditional prediction + CFG × (conditional prediction - unconditional prediction), thereby amplifying the prompt's influence on the output. The higher the CFG value, the closer the image adheres to the prompt, but too high a value leads to oversaturated colors, distorted contrast, and rigid compositions; typical values range from 7 to 12. Understanding this mechanism helps you make an informed trade-off between "creative freedom" and "instruction adherence."

From Text-to-Image to Controllable Generation

The biggest pain point of pure text-to-image is the "gacha" feeling—no matter how well you write the prompt, the result is still random. Production environments therefore rely more on controllable generation techniques. ControlNet restores compositional control to the user by introducing condition maps such as edges, depth, and pose; image-to-image and inpainting allow targeted edits on existing images, avoiding full regeneration.

How these techniques are combined determines workflow efficiency. A typical production pipeline is: first quickly explore composition at low resolution, then lock in the direction and use ControlNet to constrain pose and structure, and finally fix error-prone areas like hands and text through inpainting. Splitting generation into three stages—"explore, constrain, refine"—is more practical than chasing a perfect output in one shot.

ControlNet is a network architecture that adds spatial conditioning control to a pretrained diffusion model. By copying the original model's encoder weights and introducing zero-convolution layers, it injects condition signals such as edge maps (Canny), depth maps (Depth), and human pose (OpenPose) into the generation process while keeping the original model's parameters frozen, achieving precise compositional control without compromising generation quality. ControlNet supports stacking multiple conditions, but each additional condition increases computational cost; in practice, enabling 1–2 condition maps as needed is recommended.

Inpainting is the technique of regenerating content within a given mask region. Unlike image-to-image, inpainting only modifies pixels inside the mask, leaving the area outside the mask unchanged. In production, it is commonly used to fix error-prone local areas such as hands, text, and edge artifacts; combined with higher denoising strength and targeted prompts, it can significantly reduce the number of full-image regenerations.

How Agents Change the Generation Workflow

A single model call is only the starting point. The value of an agent lies in connecting prompt optimization, parameter selection, multi-round generation, quality filtering, and post-processing into an automated closed loop. An image generation agent typically includes several roles: a planner that breaks vague requirements into concrete prompts and condition maps; an executor that calls the image model and manages batch generation; and a critic that scores results against preset criteria (composition, clarity, style consistency) and decides whether a retry is needed.

The direct benefit of this orchestration is stability. With manual operation, a single failure often means readjusting all parameters; an agent can log the inputs and outputs of each attempt and automatically narrow the parameter search space after a failure. The cost is increased system complexity—you need to define clear quantitative standards for the critic, otherwise automation will only accelerate the production of low-quality results.

The critic in an image generation agent typically relies on multimodal large models or dedicated scoring models to automatically evaluate generated results. Common quantitative dimensions include: compositional soundness (subject placement, whitespace ratio), clarity (edge sharpness, noise level), style consistency (similarity to a reference image's color distribution and brushwork), and prompt adherence (CLIP score or VLM rating). The critic's rules need to align with business goals—for example, e-commerce scenarios prioritize subject completeness and clean backgrounds, while illustration scenarios may care more about stylistic consistency. Without clearly defined evaluation criteria, automation will only accelerate the production of low-quality results, so the critic's design often affects final output quality more than the choice of generation model.

Practical Issues You Can't Avoid in Deployment

When integrating image generation into real business operations, several issues deserve priority over model selection. On copyright and compliance, the source of training data and commercial licensing of generated content must be verified case by case, and policies vary widely across models and platforms. On cost control, GPU inference costs accumulate quickly during batch generation; caching results for frequently used prompts and downsampling low-priority tasks are common practices.

Quality evaluation is also easily underestimated. Automatic metrics (such as CLIP scores) diverge from human aesthetics, and relying entirely on automated scoring may select images that are "technically correct but not attractive." A workable compromise is: use automatic metrics for initial screening, human spot checks for calibration, and feed human feedback back into the critic's rules.

CLIP score is a commonly used automatic metric for evaluating image-text matching, measuring how well generated results adhere to prompts by computing the cosine similarity between image embeddings and text embeddings. However, CLIP scores have systematic biases relative to human aesthetic judgment: high-scoring images may be stiff and uncreative in composition, while some highly artistic works score lower. CLIP is therefore better suited as a screening tool than a final judge, and needs calibration with human spot checks and business rules. In addition, different CLIP model versions may score the same image significantly differently, so the evaluation model version must be fixed when comparing across batches.

Practical Recommendations

If you are preparing to build image generation capabilities, start with a minimal closed loop: fix one model and one set of parameters, get the single-round "prompt—generate—filter" flow working first, then gradually introduce ControlNet and agent orchestration. Don't chase full automation from the start; making each step observable and rollback-able is more reliable than pursuing an end-to-end black box. Diffusion models set the quality ceiling, and an engineered agent determines whether that ceiling can be reliably delivered.