SCRIBE: Moving Brain Semantics Research from Bag-of-Words to Context

Language neuroscience has long faced an awkward problem: experiments study semantics using isolated words or sentences, but real-world linguistic meaning is almost entirely determined by context. A word activates different semantic features in different contexts, yet traditional semantic models tend to produce fixed word vectors that cannot capture this dynamic variation. SCRIBE attempts to solve this problem—it uses an open-source LLM ensemble to generate interpretable word-level semantic representations, and validates these representations against human semantic judgments and brain activity at both the behavioral and neuroimaging levels.

Semantics Is Not a Dictionary Entry, but Something That Drifts with Context

The core premise of SCRIBE is that semantic feature judgments shift systematically with context. The research team recruited 921 participants for a behavioral experiment, asking them to rate the semantic features of words. Results showed that the same word's semantic feature ratings exhibited stable and predictable shifts across different contexts. SCRIBE's context-dependent representations were able to track these shifts, reaching 75%–81% of the human noise ceiling. The noise ceiling is an upper-bound estimate of measurement reliability in behavioral experiments; reaching this level means the model has captured most of the explainable variance in human semantic judgments.

This result alone is not surprising—LLMs performing well on semantic tasks is no longer news. The key is that SCRIBE's design goal is not prediction accuracy but interpretability. It outputs not black-box vectors, but word-level representations that can be mapped to specific semantic features. This allows researchers to ask: when the brain understands language, which semantic dimensions is it actually encoding?

Noise ceiling is a key statistical concept in behavioral and neuroimaging research. Its logic is: any measurement contains noise, so the correlation between model predictions and observed data cannot infinitely approach 1. Researchers typically estimate the internal reliability of a measurement by repeatedly measuring the same participant or the same stimulus, thereby calculating the maximum theoretically achievable prediction level. If a model's prediction accuracy has already approached the noise ceiling, it means it has captured all explainable variance in the data, and further improvement in accuracy can only come from reducing measurement noise rather than improving the model. SCRIBE's report of "reaching 75%–81% of the human noise ceiling" means its representations can already explain most reliable variance in human semantic judgments, with the remaining gap mainly stemming from measurement error in the experiment itself rather than model deficiencies. This metric is widely used in cognitive neuroscience for fair comparison of different models' predictive power.

Validation with Brain Activity: Contextual Representations Really Are Better

Behavioral experiments can only show that SCRIBE aligns with human judgments, not that it reflects the brain's actual computational mechanisms. The research team further used brain activity data from a natural narrative story listening task to test SCRIBE's ability to predict neural responses in language-selective brain regions.

The result: SCRIBE significantly outperformed non-contextual semantic representations. In other words, to predict brain responses while listening to stories, considering context is more effective than providing only fixed word meanings. This provides neural-level evidence that "the brain's semantic processing is inherently context-dependent."

More subtly, there is the variance partitioning analysis. Researchers decomposed brain prediction ability into different components and found that SCRIBE could still explain unique brain prediction information beyond syntactic features. This means its representations captured semantic dimensions that syntax cannot cover. Of course, the "syntactic features" here are a specific measurement set chosen by the researchers and do not represent all possible syntactic information, so this conclusion should be understood as: beyond the syntactic dimensions already measured, SCRIBE has additional contributions.

Variance partitioning is a multivariate statistical method used to quantify the independent contributions of multiple predictor variables to a dependent variable and their shared portions. In neuroimaging analysis, researchers often regress brain activity time series simultaneously onto multiple feature sets (such as semantic, syntactic, phonological, etc.), then calculate the unique and shared variance explained by each feature set by comparing the predictive performance of different model combinations. In the SCRIBE study, researchers used this method to decompose brain prediction ability and found that the unique contribution of semantic representations remained significant after controlling for syntactic features. It should be noted that variance partitioning results are highly dependent on the chosen feature sets and model forms, so "unique contribution" can only be interpreted as incremental information within a specific measurement framework, and does not equal complete anatomical or functional separation of semantics and syntax in the brain.

Why "Interpretable" Is More Noteworthy Than "More Accurate"

SCRIBE's methodological choices are noteworthy. It did not train a new model, but instead used an ensemble of open-source large language models to generate semantic representations. This approach lowers the barrier to replication and makes the source of semantic representations more transparent. For brain semantics research, interpretability is more valuable than mere prediction improvement, because the research goal is not to build a better brain activity predictor, but to understand how the brain represents meaning.

Of course, SCRIBE has its boundaries. It relies on the semantic space of an LLM ensemble, and whether this space itself fully aligns with the brain's semantic organization remains an open question. The noise ceiling estimates in behavioral experiments also depend on the specific experimental design and participant population. Additionally, while the natural narrative listening task is closer to real language use than traditional experiments, it is still a controlled scenario.

LLM ensemble refers to combining the outputs of multiple large language models with different architectures or training data to improve the robustness and generalization of representations. SCRIBE did not train a single model, but instead had a group of open-source large models each generate semantic feature predictions for words in the same context, then formed final representations through an aggregation strategy. The advantage of this approach is that individual model biases can be diluted by other models, yielding a more stable semantic space; at the same time, since all member models are open-source, any researcher can replicate or replace components. But ensembles also bring new interpretive burdens—the nature of the final representation depends on the choice of member models and aggregation weights, and if member models share similar training biases, the ensemble cannot eliminate these systematic biases. Therefore, SCRIBE's interpretability is built on the intermediate layer of a "semantic feature space," rather than directly dissecting the internal neurons of any single model.

Implications for AI and Neuroscience Cross-Disciplinary Research

SCRIBE's significance may extend beyond brain semantics research itself. It demonstrates a path: using large models to generate interpretable intermediate representations, then using neural data to validate whether these representations align with brain computations. This methodology can be transferred to other cognitive domains—as long as researchers can define an interpretable feature space and test its correspondence to human cognition using behavioral or neural data.

For researchers concerned with AI interpretability, SCRIBE also offers a reverse perspective: interpretability need not come from dissecting a model's internal mechanisms, but can also come from constraining model outputs to a human-understandable semantic space, then validating its effectiveness with external data. This "external alignment" style of interpretability is especially feasible in fields like brain science that have rich measurement tools.

SCRIBE is currently a research framework, not a product, but its open-source approach and interpretable design philosophy offer valuable reference for building more transparent language models that are closer to human cognition.

One-line Summary

SCRIBE uses an open-source LLM ensemble to generate interpretable word-level semantic representations, and validates their consistency with human semantic judgments and brain activity in behavioral and neuroimaging studies.

Paragraph Summary

SCRIBE aims to address the problem of semantic representations dynamically changing with context in language neuroscience. It uses an open-source large language model ensemble to generate interpretable word-level semantic feature representations, rather than fixed word vectors. In a behavioral experiment with 921 participants, SCRIBE tracked semantic feature shifts of the same word across different contexts, reaching 75%–81% of the human noise ceiling. In brain activity data from natural narrative stories, SCRIBE significantly outperformed non-contextual semantic representations, and variance partitioning showed it could explain unique brain prediction information beyond measured syntactic features. This methodology emphasizes interpretability over mere prediction accuracy, providing a new path for AI-neuroscience cross-disciplinary research by constraining model outputs to a human-understandable semantic space and validating them with external neural data.