Suppressing Image-to-Text Feedback: SFG Improves Generation Alignment in Diffusion Models

Multimodal Diffusion Transformers (MMDiTs) have become the dominant architecture in text-to-image generation. One of their key characteristics is that during image generation, the evolving image can update internal text representations through a joint attention mechanism — and those updated representations then continue to guide subsequent image generation. This raises a question worth examining: does this Image-to-Text (I2T) feedback help the model correct deviations from the prompt, or does it actually reinforce those deviations?

A recent study provides a clear answer: I2T feedback tends to preserve objects and attributes already formed in the image, even when those elements are incorrect. Based on this finding, the researchers propose Semantic Feedback Guidance (SFG), a method validated across multiple mainstream image generation backbones.

The Core Problem: Is Feedback Actually Working Against Us?

MMDiTs were designed to enable bidirectional interaction between image and text during generation — in theory, this should make outputs better aligned with the prompt. But the research team's intervention experiments revealed a more complex picture.

They ran a straightforward experiment: transplanting I2T information from one generation run into another. The results showed this transplant could either rescue a failing generation or sabotage a successful one. In other words, I2T feedback doesn't consistently push toward "better prompt adherence" — it behaves more like a mechanism that reinforces the image's current state.

A controlled ablation made this even clearer: under the same intervention intensity, suppressing I2T feedback improved generation quality, while amplifying it made results worse. This symmetric result essentially proves that I2T feedback frequently reinforces errors rather than correcting them.

Why Does This "Entrenchment" Happen?

Mechanistically, when an image has already "drawn" a particular object or attribute in early sampling steps, I2T feedback writes that information back into the text representation — effectively allowing the already-formed image to "hijack" the subsequent Text-to-Image (T2I) conditioning. What should be a prompt-driven generation process ends up being driven by the evolving image itself. Once an error appears early, this feedback loop continuously reinforces it.

MMDiTs' joint attention mechanism differs fundamentally from the traditional cross-attention used in earlier diffusion models (e.g., Stable Diffusion 1.x/2.x). In those earlier models, text conditioning acted on image features unidirectionally via cross-attention, and the text representations themselves were never updated by the image. In MMDiTs (such as SD3 and FLUX), image tokens and text tokens are concatenated and jointly processed through the same self-attention computation, enabling bidirectional modification. This design theoretically allows the model to capture richer image-text correspondences and is one key reason MMDiTs generally outperform earlier architectures in image quality and semantic alignment. But this very bidirectionality is what enables I2T feedback to entrench errors: text representations are no longer static conditioning anchors but dynamic variables that drift passively as the image evolves.

SFG's Solution: Guiding Away from a Feedback-Amplified Auxiliary Branch

Since the problem stems from the entrenchment effect of I2T feedback, the natural solution is to weaken its influence. SFG works by constructing an auxiliary branch that amplifies I2T feedback and weakens T2I conditioning, then nudges the main generation process to move away from this auxiliary branch's direction. This negative guidance logic mirrors Classifier-Free Guidance (CFG), except the target of guidance shifts from text conditioning to the feedback mechanism itself.

The study also introduces a more efficient in-place variant, iSFG (in-place SFG). It directly suppresses I2T feedback and strengthens T2I conditioning, with the key advantage of requiring no additional model evaluations — meaning zero added inference cost (NFE, Number of Function Evaluations). This matters greatly for real-world deployment, where inference overhead is often the central practical concern.

Intervening Only in the First 20% of Steps

Analysis along the time dimension yields a practical insight: the SFG branch only needs to run during the first 20% of sampling steps. This aligns with the earlier finding that errors are entrenched early — since deviations primarily form in early steps, intervention should also be concentrated there. Without CFG, this temporal restriction keeps SFG's overhead to roughly 1.2× the base NFE — a very manageable cost.

Classifier-Free Guidance (CFG) is an important reference point for understanding SFG's design. The core idea of CFG is to run both a conditional and an unconditional branch simultaneously, then amplify the conditional signal by extrapolating in the direction of "conditional minus unconditional" — strengthening prompt adherence at roughly double the inference cost. This mechanism has become standard practice in diffusion model inference. SFG borrows this "negative guidance" paradigm: its auxiliary branch plays the role of "the worse direction" — amplifying I2T feedback and weakening T2I conditioning, making it easier for erroneous image content to hijack the generation. The main branch explicitly moves away from this direction during updates, effectively suppressing the interference of I2T feedback on generation. Unlike CFG's contrast between "with and without text conditioning," SFG contrasts "whether the feedback mechanism is amplified" — and the two can be used together without conflict.

Experimental Results: Consistent Gains Across Three Backbones

The study validated SFG across three image generation backbones, achieving improvements in both alignment metrics and preference model scores.

Key numbers:

• On FLUX, stacking SFG on top of CFG improved GenEval from 0.6661 to 0.7479 — a substantial gain;
• On SD3.5M, iSFG improved HPSv2.1 preference scores by 1.33 points over CFG with zero additional NFE.

These two results represent distinct value propositions: one demonstrates significant alignment improvement when a small additional cost is acceptable (FLUX with SFG), and the other shows stable quality gains with absolutely no added inference cost (SD3.5M with iSFG).

Transfer to Video Generation

This control method is not limited to static images. The researchers found that the same feedback-suppression approach transfers to text-to-video and image-to-video generation tasks, yielding improvements on VBench scores. This suggests that I2T feedback entrenchment may be a general problem in multimodal diffusion architectures rather than a task-specific artifact.

VBench is a comprehensive evaluation benchmark designed specifically for text-to-video models, covering multiple dimensions including semantic consistency, temporal coherence, motion quality, and visual clarity. It is one of the most widely used standardized evaluation tools in the video generation field. Unlike GenEval (which focuses on compositional semantic alignment such as object counting and attribute binding) and HPSv2.1 (an aesthetic scoring model based on human preferences), VBench emphasizes cross-frame temporal stability and the plausibility of dynamic content. SFG's improvements on VBench indicate that the I2T feedback entrenchment effect not only impacts single-frame semantic accuracy but may also cause cross-frame content drift or semantic inconsistency by interfering with conditioning signals in video generation. This cross-task transfer provides cross-modal evidence for the argument that "I2T feedback entrenchment is a systemic issue at the MMDiT architecture level."

Implications for Multimodal Generation Research

The value of this work lies not only in proposing a method that improves benchmark scores, but in revealing a previously overlooked mechanism flaw within MMDiTs. The bidirectional interaction enabled by joint attention has long been assumed to be beneficial by default — but this research uses rigorous intervention experiments to demonstrate that one direction of that bidirectional feedback (I2T) is, in many cases, actually a source of generation error.

For researchers, this suggests that when designing multimodal architectures, we should not assume "more interaction equals better results." The directionality of feedback loops, the timing of intervention, and the dominance of conditioning signals are all design dimensions worth carefully tuning. SFG addresses this through negative guidance away from a feedback-amplified branch, offering a plug-and-play improvement paradigm that requires no retraining.

The research team has open-sourced their code at https://github.com/yyc1215lll-creator/SFG, making it easy for others to validate and extend this method across more diffusion models. For developers focused on text-to-image and text-to-video generation quality, SFG — and especially the zero-cost iSFG variant — is an optimization option well worth trying.