LLM Scientific Discovery Ability Is Determined by 'Monofacts' in Training Data

Can large language models make genuine scientific discoveries? This question has long remained unresolved. A new study offers a surprising answer: the upper bound of a model's discovery ability may have been quietly set, early on, by something called a "monofact" in its training data.

What Is a 'Monofact'?

The study defines monofact as factual information that appears only once in the training corpus. Conventional wisdom holds that models gain generalization ability from massive, diverse data; but this study, from the perspective of "missing probability mass," proposes that monofacts are the key that triggers scientific discovery.

An intuitive way to understand this: if a piece of knowledge is repeatedly reinforced during training, the model tends to treat it as common-sense memorization; whereas a fact that appears only once forces the model to integrate and transfer it during reasoning, thereby producing new combinations. This is precisely the ability to "connect disparate pieces of information" that scientific discovery requires.

"Missing probability mass" is a concept from information theory and statistical learning, referring to the fact that in a training distribution, certain events (such as specific fact combinations or reasoning paths) occur so infrequently that their probability mass is nearly zero, and thus they are ignored or cannot be adequately modeled during learning. When a large number of facts in the training corpus appear only once, the probability mass corresponding to these facts is severely missing from the empirical distribution, and the model cannot reliably memorize them through frequency estimation. This absence forces the model to rely on parametric knowledge for composition and generalization during reasoning, rather than simple pattern matching. Thus, the missing-probability-mass perspective explains why monofacts can stimulate a model's integration and transfer abilities instead of being stored directly as common sense.

Experimental Design: Directly Manipulating the Proportion of Monofacts

The authors did not stop at theoretical reasoning but directly manipulated the proportion of monofacts in training sets across two domains: materials science and regulatory genomics.

• Covering two open-weight model families
• Covering multiple sampling strategies
• Covering different inference budgets

The results were highly consistent: the more monofacts, the more discoveries the model made. This pattern held across different models, different sampling methods, and different inference budgets, indicating that it is not an incidental product of some training trick.

Open-weight model families refer to pretrained language models whose parameters and architectures are publicly downloadable (such as LLaMA and Mistral), which researchers can fine-tune, run inference on, and experimentally manipulate locally without relying on closed-source APIs. Sampling strategies refer to methods for selecting the next word during generation or inference, such as greedy search, beam search, nucleus sampling (top-p), and temperature sampling; different strategies affect the diversity and determinism of outputs. Inference budget refers to the computational resources a model is allowed to consume during inference, such as the number of chain-of-thought steps, the number of samples, or the maximum generation length. Covering these dimensions was intended to verify whether the monofact effect is universal rather than an incidental phenomenon under specific configurations.

It's Not Diversity but 'Rarity' That Is at Work

A natural counterargument is that datasets with many monofacts are often inherently richer and more diverse, so the improvement might simply be due to diversity. The study conducted further experiments specifically to confirm that this relationship cannot be explained by training-set diversity alone.

In other words, what really matters is not "how many different things the model has seen" but "how many things it has seen only once." Monofacts provide a unique information structure: they give the model clues without providing sufficiently strong repeated patterns, thereby preserving room for reasoning.

Cross-Domain Transfer: Only Effective When Paired with In-Domain Data

The study also tested whether this relationship can transfer to other scientific domains. The answer is that it holds conditionally: when monofacts are paired with in-domain data, the improvement in cross-domain discovery ability can transfer.

This suggests a practical strategy: if you want a model to perform better in a new scientific domain, simply adding repetitive data in that domain may be less effective than deliberately introducing more factual content that appears only once, while retaining foundational in-domain corpora.

Cross-domain transfer in machine learning generally refers to whether a model capability trained or adjusted in one domain (such as materials science) can be applied to another domain (such as regulatory genomics). "Holds conditionally" here means that the discovery-ability improvement brought by monofacts is not a general meta-ability but requires combination with the target domain's foundational corpus to be activated. This is because in-domain data provides the necessary terminology, concepts, and reasoning paradigms, while monofacts provide novel compositional clues; both are indispensable. If monofacts are increased without domain anchors, the model may be unable to effectively associate rare information with domain knowledge, thereby limiting transfer effectiveness.

Implications for Model Training and Data Engineering

The value of this work lies in reducing the vague concept of "scientific discovery ability" to a data attribute that can be measured and manipulated.

For data engineering teams, this means that when building training sets, in addition to focusing on scale, quality, and diversity, they should also pay attention to the monofact rate as a metric. For researchers, it provides a new experimental variable: by adjusting the proportion of monofacts, one can systematically study the boundaries of model reasoning and discovery behavior.

Of course, a higher monofact rate is not always better. The study identifies it as one of the "key factors," not the only factor. How to strike a balance between monofacts and repeated reinforcement remains an open question.

One-line Summary

Research finds that the proportion of monofacts in training data determines the upper bound of LLM scientific discovery ability.

Paragraph Summary

A new study proposes the concept of "monofact"—factual information appearing only once in the training corpus—and finds that its proportion is a key factor determining large language models' scientific discovery ability. By directly manipulating the proportion of monofacts in training sets across materials science and regulatory genomics, the study consistently shows that more monofacts lead to more discoveries, and this pattern holds across model families, sampling strategies, and inference budgets. Further experiments rule out training-set diversity as an explanation, indicating that rarity itself has an independent effect. Cross-domain transfer holds conditionally: monofacts must be paired with in-domain data to transfer discovery ability. This provides data engineering with a new metric—the monofact rate.