When Do Real Connectomes Win? Three Evaluation Protocols Reveal the Fragility of Structure-Function Claims

Connectome models' structural advantages may stem from tuning bias; only some conclusions hold across evaluation protocols.
The article discusses a study that systematically tests the robustness of connectome-constrained models. The authors note that the common claim that real-connectivity models outperform null models may conflate connectivity structure with tuning, since physiological parameters are set by the modeler and often repeatedly adjusted. The study distinguishes three evaluation protocols—P1 (tune on real connectivity, transfer to null models), P2 (tune each independently), and P3 (physiological parameters constrained by independent data)—and quantifies biases using synthetic ground truth, placebo null models, and three Drosophila circuit models. Results show that conclusions such as segregated reward and punishment dopamine pathways and head-direction topography are robust across protocols, but the mushroom body Kenyon cell output connectivity advantage reverses under P3, and the ranking of the mirrored connectome in flyvis varies with retraining duration and checkpoint. The authors propose that credible structural claims require matched optimisation, data-constrained physiology, and models that still work under those conditions.
Connectome-constrained models are often used to argue that measured connectivity maps outperform random or rewired controls, thereby demonstrating that structure shapes function. However, such comparisons may conflate connectivity structure with the way models are tuned—the connectome only fixes who connects to whom, while parameters such as excitability, gain, and plasticity rates must still be set by the modeler and are often adjusted repeatedly until the real-connectivity model "performs well." A recent study systematically tested the robustness of such claims by distinguishing three evaluation targets (estimands).

Three Evaluation Protocols: From P1 to P3
The study breaks evaluation down into three levels. P1 refers to tuning physiological parameters on the real connectivity and then transferring them to null models (shuffled/rewired) for testing; P2 refers to tuning each connectivity independently; and P3, building on P2, further constrains physiological parameters to be consistent with independent experimental data. These three protocols correspond to different scientific questions and introduce different biases.
The paper notes that these biases are familiar in machine learning: undertuned baselines, inherited weights, and the winner's curse. The problem is that their actual impact on biological conclusions has never been quantified. Using synthetic ground truth, placebo null models, and three Drosophila circuit models, the authors measured how biases distort inference.
Null model in connectomics refers to a control network obtained by randomizing the real connectivity matrix, commonly via edge rewiring (keeping each node's in-degree and out-degree unchanged but shuffling connection targets) or edge shuffling (randomly exchanging connection weights or targets). Its role is to provide a baseline in which "structural information is destroyed"; if the real-connectivity model significantly outperforms the null model, this is taken as evidence that the connectivity structure confers a functional advantage. However, models on null networks are typically not subjected to the same parameter tuning as the real connectivity, so performance gaps may stem partly from differences in tuning effort rather than the connectivity structure itself.
Winner's curse arises from multiple comparisons or selective reporting: when researchers select the best performer among many candidate models or parameter configurations, the selected model's observed performance tends to be systematically higher than its true generalization performance, because the selection process itself exploits noise. In connectome modeling, if parameters are repeatedly tuned on the real connectivity until it "performs well" and then compared against an untuned null model, the real connectivity's advantage may stem partly from this selection bias. The related inherited weights problem refers to null models directly reusing parameters trained on the real connectivity, causing their performance to be overestimated or underestimated, depending on how well the parameters match the structure.
Which Conclusions Hold Up?
Not all structure-function claims are equally fragile. Findings that are robust across protocols include: segregated reward and punishment dopamine pathways improving learning; and head-direction topography being necessary to maintain a persistent bump. These conclusions hold under P1, P2, and P3, and are relatively trustworthy.
But other conclusions flip with the protocol. In the mushroom body, the real Kenyon cell output connectivity beats cross-subtype identity shuffles by 0.09 under P1 and P2, but loses to that shuffle by 0.09 under P3 (95% confidence intervals exclude zero; held-out runs confirm). The decisive factor is the proportion of alpha'beta' cells among active Kenyon cells: P1 tends to select a lower proportion, whereas imaging data suggest a proportion of about 44%—at which the real-connectivity model fails to learn, revealing model inadequacy rather than a flaw in the fly connectome itself.
In the flyvis dataset, P1 penalizes the mirrored connectome more heavily than the spatial shuffle (50 out of 50 members), even though the mirrored connectome admits a constructed solution whose median error is only 0.07 above the true error. This ordering persists after tenfold longer retraining but becomes checkpoint-dependent at the fiftyfold checkpoint.
Kenyon cells are the principal intrinsic neurons of the Drosophila mushroom body, whose axons form synapses in the mushroom body calyx and receive olfactory input from projection neurons. The mushroom body is the core brain region for learning and memory in Drosophila, and Kenyon cell output connectivity describes the projection patterns of these neurons onto downstream output neurons. Kenyon cells can be divided into subtypes such as alpha/beta, alpha'beta', and gamma, which differ in anatomical projections and functional roles. In the study, "cross-subtype identity shuffle" refers to randomly exchanging cell identity labels while keeping the number of cells in each subtype unchanged, thereby destroying subtype-specific connectivity patterns, and is used to test whether the real connectivity exploits subtype identity information.
Methodological Implications: Three Conditions for a Credible Structural Claim
The authors conclude that a credible structural claim requires three conditions: matched optimisation, data-constrained physiology, and the model still working under those conditions. Without any one of these, an observed "structural advantage" may simply be a product of tuning bias.
This work raises a methodological warning for connectome-constrained modeling, which is widely used in computational neuroscience: when a model is finely tuned on the real connectivity but the null models receive no comparable tuning, the comparison itself may produce spurious structural effects. Future studies should report results under multiple protocols and clearly distinguish "connectivity contributions" from "tuning contributions."
Background Notes
Mirrored connectome refers to a control network obtained by flipping the real connectivity matrix along an axis of symmetry, commonly used to test whether function depends on the directionality of connections rather than merely the distribution of connection strengths. Constructed solution refers to a parameter configuration designed specifically, via analytical or optimization methods, for a given control network so that it can perform the target task. If a mirrored connectome admits a constructed solution whose performance approaches that of the real connectivity, this shows that the control network is not functionally unattainable, and the large penalty observed under P1 may reflect insufficient tuning rather than a genuine functional disadvantage of the structure itself.
Related articles

Gemini 4 Hydrogen Hands-On: Google's Fastest and Cheapest AI Model
Hands-on with Gemini 4 Hydrogen: the fastest, cheapest model in the Gemini 4 family. We break down the Carbon/Argon/Hydrogen split, API pricing traps, free-tier limits, and the cost trade-offs of four thinking levels—with a real bug-debugging demo.

Gemini 4 Hydrogen vs Sonnet Comparison Test: Uses Only 1% of Weekly Quota, Performance on Par
Hands-on test: Gemini 4 Hydrogen vs Sonnet on the same CRUD task—Hydrogen used only 1% weekly quota with comparable performance and better UI details, while Anthropic's model used 7%. Developer says daily tasks don't require expensive models.

Gemini 4 Neon Released: Beats Opus 5.5 on Multiple Benchmarks, Google Returns to the Top Tier
Google DeepMind releases Gemini 4 Neon: 61 on the Artificial Analysis intelligence index (3rd) and 1st on the coding index, beating Opus 5.5 on multiple benchmarks. $3/$15 per million tokens; available in the Gemini API and AI Studio.