Martini
A surrogate-reward mixing method for guiding sampling in generative models, aimed at using surrogate rewards to steer sampling toward outcomes desired in downstream applications.
Timeline (last 90 days)
Martini's goal is: given only limited access to the oracle reward r* through a small feedback set D, to approximate the tilted distribution q* of r* using a combination function G of multiple cheap surrogate rewards.
If only the best single surrogate is used with ever-increasing guidance strength, reward hacking occurs and performance degrades; using multiple surrogates simultaneously provides a natural safeguard.
In language modeling experiments, using the generation perplexity of Qwen 2.5 1.5B as the oracle reward and BERT, distilled BERT, token entropy, etc. as surrogates, with a feedback set of only 256 values or rankings, the KL between the q_G learned by Martini and q* is significantly lower than that of uniform weighting or Pareto-type methods.
In protein-ligand cofolding experiments, using the Boltz flow map version with binding affinity as the oracle reward, the success criterion (RMSD less than 2 angstroms and PB valid above threshold) improved from 61.8% to 64.5%.
All Facts (4)
Martini's goal is: given only limited access to the oracle reward r* through a small feedback set D, to approximate the tilted distribution q* of r* using a combination function G of multiple cheap surrogate rewards.
50%UnverifiedIf only the best single surrogate is used with ever-increasing guidance strength, reward hacking occurs and performance degrades; using multiple surrogates simultaneously provides a natural safeguard.
50%UnverifiedIn language modeling experiments, using the generation perplexity of Qwen 2.5 1.5B as the oracle reward and BERT, distilled BERT, token entropy, etc. as surrogates, with a feedback set of only 256 values or rankings, the KL between the q_G learned by Martini and q* is significantly lower than that of uniform weighting or Pareto-type methods.
50%UnverifiedIn protein-ligand cofolding experiments, using the Boltz flow map version with binding affinity as the oracle reward, the success criterion (RMSD less than 2 angstroms and PB valid above threshold) improved from 61.8% to 64.5%.
50%