SFG
Semantic Feedback Guidance, an inference guidance method designed for Multimodal Diffusion Transformers. It improves alignment between generated outputs and text prompts by suppressing the feedback solidification effect arising from image-to-text (I2T) attention flows within the MMDiT architecture.
Timeline (last 90 days)
The researchers propose Semantic Feedback Guidance (SFG), a method that constructs an auxiliary branch to amplify I2T feedback and weaken T2I conditioning, then steers the generation process away from that auxiliary branch direction during guidance.
SFG provides an in-place variant, iSFG, which directly suppresses I2T feedback and strengthens T2I conditioning without requiring additional model evaluations, adding no inference cost (NFE).
The SFG branch only needs to run during the first 20% of image sampling steps, and without using CFG its overhead is only approximately 1.2× the base NFE.
On the FLUX model, stacking SFG on top of CFG improves the GenEval score from 0.6661 to 0.7479.
SFG's feedback suppression approach can be transferred to text-to-video and image-to-video generation tasks, achieving improvements on VBench scores.
The study was validated on three image generation backbone networks, with improvements observed in both alignment and preference model scores.
SFG is a plug-and-play improvement paradigm that requires no model retraining.
The research team has open-sourced the SFG code on GitHub (github.com/yyc1215lll-creator/SFG).
All Facts (8)
The researchers propose Semantic Feedback Guidance (SFG), a method that constructs an auxiliary branch to amplify I2T feedback and weaken T2I conditioning, then steers the generation process away from that auxiliary branch direction during guidance.
50%UnverifiedSFG provides an in-place variant, iSFG, which directly suppresses I2T feedback and strengthens T2I conditioning without requiring additional model evaluations, adding no inference cost (NFE).
50%UnverifiedThe SFG branch only needs to run during the first 20% of image sampling steps, and without using CFG its overhead is only approximately 1.2× the base NFE.
50%UnverifiedOn the FLUX model, stacking SFG on top of CFG improves the GenEval score from 0.6661 to 0.7479.
50%UnverifiedSFG's feedback suppression approach can be transferred to text-to-video and image-to-video generation tasks, achieving improvements on VBench scores.
50%UnverifiedThe study was validated on three image generation backbone networks, with improvements observed in both alignment and preference model scores.
50%UnverifiedSFG is a plug-and-play improvement paradigm that requires no model retraining.
50%UnverifiedThe research team has open-sourced the SFG code on GitHub (github.com/yyc1215lll-creator/SFG).
50%