Training LLMs with Set-Level Reinforcement Learning to Generate Diverse Scientific Ideas

The first step in the research process is often "coming up with an idea." LLMs are increasingly being used to automate this step, but most existing work only optimizes the quality of a single idea while ignoring diversity. A new paper proposes a document-anchored idea generation task, where the goal is not only to make the model produce high-quality ideas, but also to make ideas within the same batch sufficiently different from one another.

The Problem: High-Quality Ideas That All "Look the Same"

In the document-anchored idea generation setting, the model needs to propose a set of ideas based on a given paper. Each idea must satisfy two conditions: it must be high quality in itself, and it must be clearly distinguishable from the other ideas. At the same time, all ideas must be grounded in the input document and cannot be made up out of thin air.

There is a tension here that prior research has overlooked: if you train or evaluate a model solely on "how good a single idea is," the model tends to converge on some safest mode of expression. The ideas it generates may have decent average quality, but they are highly similar to one another. Yet the value of science lies precisely in exploring different directions, and a homogeneous set of ideas offers limited reference value to researchers.

Document-grounded means that generated content must be based on a given input document as its factual and contextual foundation, rather than relying on the model's own parametric memory to improvise freely. In such tasks, the input is usually the title, abstract, or full text of a paper, and every idea the model proposes needs to be traceable to the problem setting, methods, or limitations in the document—for example, proposing a validation scheme for some unresolved hypothesis in the paper. The point of document grounding is to suppress hallucination: idea generation without an anchor easily produces content that sounds plausible but is actually unrelated to the source paper, and it also makes different ideas lose comparability. It simultaneously constrains the boundaries of diversity—ideas can differ from one another, but they must all fall within the problem space delineated by the same paper. This means "diversity" and "relevance" need to be optimized together rather than treated as an either-or choice.

The Method: Set-Level Reinforcement Learning That Pushes New Ideas "Away" from Old Ones

The core proposal of the paper is a set-level reinforcement learning objective. Specifically, in a single generation pass, the model writes K structured "idea cards" all at once, and all of these cards are anchored to the source paper.

The key lies in the scoring method—the entire set is scored jointly, rather than each card being scored independently. The quality score of each idea is weighted according to its distance from previously generated ideas: ideas closer to the preceding content have their weights reduced, while more distant ones retain more weight. This design teaches the model during training to actively push each new idea away from the previous ones, forming a kind of "self-deduplication" generation strategy.

This mechanism differs from the common "generate first, then filter" approach. It does not rely on post-processing to filter out duplicates; instead, it weaves diversity directly into the generation process, making the model aware of the directions already occupied by preceding content when it writes the next idea.

Reinforcement learning (RL), when used to train language models, typically requires an automatically computable reward signal to replace manual item-by-item annotation. The model samples multiple outputs for the same prompt and adjusts the probability of each output being selected according to the reward: trajectories with high rewards are reinforced, and those with low rewards are suppressed. Rewards can come from verifiable answers, rule-based scoring, or judgments given by another model. Compared with supervised fine-tuning, the advantage of RL is that it can optimize objectives that are difficult to directly supervise with standard answers—for example, "the degree of difference among a set of outputs." Such objectives have no unique correct answer and can only be characterized indirectly through a scoring function. The set-level objective in this paper precisely treats the entire set of K cards as a single trajectory to be scored, with the reward jointly determined by quality and within-set distance, so the model can perceive previously generated content during sampling rather than filtering it afterward.

Results: An 8B Model Surpasses Larger Models in Creative Utility

The paper reports several noteworthy results:

• On a contamination-free benchmark, the trained 8B model surpasses Claude Opus 4.8 and GPT-6 Astra in "creative utility," a metric that combines quality and diversity.
• In a preliminary human study with four experts, the set of ideas generated by this model was judged to be as promising as those from Claude Opus 4.8, and it produced slightly more unique ideas.
• When a coding agent executes proof-of-concept experiments on these ideas, the model's success@k is comparable to Claude Opus 4.8, indicating that improved diversity does not come at the cost of executability.

The last point is especially critical. Diversity and executability are often believed to involve a trade-off: the more wildly imaginative an idea is, the harder it is to implement. But in this work, the two do not conflict.

Contamination-free benchmark refers to evaluation data that has not been included in the model's training corpus, or at least for which it can be demonstrated that the model has not seen the answers during training. A common hidden risk in large model evaluation is data leakage: if the questions or answers of a test set have appeared in the pretraining data, the model may score based on memorization rather than reasoning, leading to inflated results. For tasks like idea generation, contamination also manifests as the model "recalling" relevant discussions from some paper rather than genuinely reasoning based on the input document. Therefore, only comparisons on contamination-free benchmarks can more credibly reflect the model's actual generation and generalization ability. This paper emphasizes this point to show that the result of an 8B model surpassing larger models is not due to test set leakage.

success@k is a batch evaluation metric: given k candidate ideas, if at least some number of them are judged successful by downstream experiments or human evaluation—for example, proof-of-concept experiments run successfully and conclusions hold—it is counted as a hit. It measures "whether a feasible solution can be found within a set of outputs," rather than the average performance of a single output. When candidate ideas are automatically executed by a coding agent, success@k can simultaneously reflect the executability of ideas and the coverage of the set: as long as one idea in the set can be implemented and verified, researchers have reason to keep investing. This complements creative utility—creative utility examines the balance between quality and diversity, while success@k tests whether that diversity remains within an actionable range. Together they support the paper's conclusion that "diversity does not sacrifice executability."

A Few Observations

This method elevates "diversity" from an evaluation metric to a training signal, and the idea is worth learning from. The design of set-level scoring can be transferred to other scenarios that require batch generation with mutual distinctness, such as marketing copy, test cases, or dimensions of competitive analysis.

On the other hand, the human study involved only four experts, so the sample size is limited, and the conclusions are better treated as directional evidence rather than definitive. The conclusion that an 8B model surpasses larger models in creative utility also needs to be reproduced on more benchmarks and tasks.

Overall, this work reminds us that when LLMs are used for "batch output," the unit of evaluation and training should perhaps not be a single output, but the entire output set.

One-line Summary

The paper uses set-level reinforcement learning to train an LLM to generate high-quality scientific ideas in batches that are grounded in documents, mutually distinct, and executable.

Paragraph Summary

This article introduces work that upgrades diversity from an evaluation metric to a training signal. To address the problem in document-anchored idea generation where "individual ideas are high quality but resemble one another," the paper proposes a set-level reinforcement learning objective: the model generates K idea cards anchored to the source paper in a single pass, the entire set is scored jointly, and each idea's quality score is weighted by its distance from previously generated ideas—the closer it is, the lower the weight. This achieves self-deduplication during generation rather than in post-processing. Results show that the trained 8B model surpasses Claude Opus 4.8 and GPT-6 Astra in creative utility on a contamination-free benchmark; in a human study with four experts, its idea sets were considered equally promising and slightly more unique; and when a coding agent executes proof-of-concept experiments, success@k is also comparable, indicating that diversity does not sacrifice executability. The authors also note that the human study sample is limited, and the conclusions still require more reproduction.