Nearest-Neighbour Baselines Challenge Deep Learning: A Rigorous Benchmark for MS/MS Fingerprint Prediction

An Underestimated Simple Approach

Deep learning has long been regarded as the mainstream solution for molecular fingerprint prediction — a core task in computational mass spectrometry. Yet a recent arXiv report (arXiv:2610.02249v1) raises a noteworthy point: simple nearest-neighbour (NN) retrieval methods can match or even outperform current deep learning models when predicting molecular fingerprints from MS/MS spectra.

This is not an isolated finding. The report cites several recent studies (Khoo and Barzilay; Liu et al.; Gupta et al.) that all point in the same direction — nearest-neighbour retrieval represents a consistently underestimated strong baseline. For a field that has invested heavily in complex neural network architectures, this is a sobering observation.

Why Molecular Fingerprint Prediction Matters

Molecular fingerprints are fixed-length vector representations that encode chemical structures, widely used in compound identification, similarity search, and virtual screening. MS/MS (tandem mass spectrometry) is the key experimental technique in metabolomics and natural product research for inferring molecular structures.

Predicting molecular fingerprints directly from MS/MS spectra means structural clues can be extracted for unknown compounds without reference standards. This is precisely the problem that well-known tools like CSI:FingerID aim to solve. Deep learning methods have attracted substantial research investment in this task in recent years — which makes the question all the sharper: if a simple retrieval baseline can keep pace, how much genuine progress have these complex models actually delivered?

Molecular fingerprints typically take the form of binary or real-valued vectors, where each dimension encodes the presence or absence of a specific substructure or chemical feature. Common types include ECFP (Extended Connectivity Fingerprints) and MACCS Keys. MS/MS works by fragmenting a precursor ion and measuring the mass-to-charge ratio (m/z) and abundance of the resulting fragments, producing a "fragmentation spectrum." Different compounds fragment in characteristic ways, in principle encoding the information needed to infer the original molecular structure.

CSI:FingerID, developed by the Böcker group, uses machine learning to map MS/MS spectra into molecular fingerprint space, then retrieves candidate structures from a database via fingerprint similarity. This two-stage pipeline (spectrum → fingerprint → structure) has become the field's standard paradigm, which is precisely why the intermediate step of fingerprint prediction is so critical: the quality of fingerprint prediction directly determines the success of final structure identification.

"Nearest Neighbour" Is Not a Single Method

The report's central insight is that "nearest neighbour" is not a single algorithm but a family of methods. The key difference between variants lies in what information they assume is available at inference time.

Inference Assumptions Drive Performance

For example, some variants may assume that the retrieval library contains reference spectra highly similar to the query; others impose stricter or more lenient constraints on what reference information is available. These seemingly subtle differences in assumption can significantly affect reported performance metrics.

In other words, if a paper claims its deep learning model surpasses a "nearest-neighbour baseline" without clearly specifying which variant and which assumptions that baseline uses, the comparison may lack validity. A weak baseline artificially inflates the apparent advantage of the new model.

The concept of "information assumptions at inference time" essentially refers to which reference data a model or baseline is permitted to "see" during prediction. For nearest-neighbour retrieval, the strongest assumption is that the retrieval library contains reference spectra from the same compound — or a structurally very similar one — as the query spectrum. Under this assumption, the retrieval method can essentially "copy the answer," yielding naturally high performance. A stricter assumption requires the retrieval library to contain only structurally non-overlapping compounds — the standard cross-compound generalization evaluation scenario.

A common problem in deep learning papers is evaluating the proposed model under strict assumptions while comparing it against a nearest-neighbour variant that operates under looser, information-rich assumptions with implicit data leakage. This asymmetric comparison systematically overstates the relative advantage of deep learning. Ensuring that all methods share the same information constraints during evaluation is therefore a prerequisite for valid comparison.

Building More Rigorous Benchmarks

The report's goal is clear: systematically compare multiple nearest-neighbour variants, reveal how different assumptions affect performance, and thereby establish more rigorous baselines. The broader significance for the field is straightforward — only when baselines are sufficiently robust can we genuinely measure the progress that deep learning models deliver.

This approach echoes a recurring theme of self-reflection in the machine learning community: many claimed "breakthroughs" shrink considerably when faced with carefully tuned simple baselines. Strong baselines are not meant to dismiss deep learning, but to ensure that benchmarking is honest and rigorous, giving subsequent research a reliable reference point.

Criticism of overlooked simple baselines has a long history in machine learning. Lipton and Steinhardt (2019), in Troubling Trends in Machine Learning Scholarship, systematically catalogued common pitfalls in research papers, with inappropriate baseline selection among the most prominent. In natural language processing and recommender systems, similar patterns have emerged repeatedly: carefully tuned linear models or BM25 retrieval have matched or outperformed complex neural networks in head-to-head comparisons.

This phenomenon is particularly worth watching in cheminformatics, where training datasets (such as the GNPS public spectral library) are relatively limited in scale and variable in quality. When data is scarce, memory- and retrieval-based methods enjoy a natural advantage — they directly leverage known data rather than attempting to generalize from it, and may therefore outperform deep models that rely on statistical generalization for in-distribution samples.

Implications for Research Practice

For researchers working on mass spectrometry data analysis or molecular representation learning, this report offers several practical reminders:

• When reporting performance, always explicitly state the information assumptions used by baselines at inference time;
• Before evaluating a new model, establish a reference point using a well-tuned retrieval method;
• Be alert to spurious impressions of progress arising from poorly constructed baselines.

It should be noted that, as a report summary, the specific experimental numbers and detailed definitions of each variant have not yet been made public, so we cannot provide quantitative conclusions here. Nevertheless, the methodological question it raises — how to fairly compare retrieval methods and deep learning — has universal value beyond this single task.

Conclusion

This work does not win points with a flashy new model. Instead, it returns to the foundation of scientific evaluation: a fair and rigorous comparison framework. In an era of proliferating deep learning methods, reminding the community to take simple baselines seriously is itself a valuable contribution. For researchers who genuinely want to advance the field of MS/MS fingerprint prediction, rigorous benchmark construction may be more urgently needed than yet another new architecture.