Spectral Feedback

Spectral Feedback is an alignment method for discrete diffusion models that re-masks and re-samples already-generated tokens, selects positions to edit within a feedback loop, and allows the model to iteratively correct its own generated results.

Spectral Feedback treats the selection of editing positions (which tokens to revisit) rather than label assignment as the central issue of the alignment problem.

Editing effects are mutually dependent: the impact of modifying one token depends on which other tokens are edited simultaneously, so editing sets must be considered rather than evaluating individual positions in isolation.

The study finds that in the protein inverse folding task, the value function of editing sets exhibits a sparse Fourier representation.

The sparse Fourier structure means that despite the enormous space of editing combinations, the effective components determining their value are sparse, enabling Spectral Feedback to efficiently learn the value function without exhaustively enumerating all combinations.

Spectral Feedback is model-agnostic and can be applied to pretrained models, test-time alignment models, and fine-tuned diffusion models without modifying the underlying generation process.

In an inverse folding task targeting a protein stability reward oracle, Spectral Feedback increases the number of stable proteins by 32.3% for pretrained models.

Spectral Feedback improves the Best-of-10 sampling strategy by 24.8%.

Spectral Feedback still delivers a 5.8% improvement over the current state-of-the-art RL fine-tuned diffusion models.