LitArc-Bench: Disentangling Position and Lexical Sentiment to Identify Literature-Specific Reading Responses

New LitArc-Bench benchmark decomposes the sources of model emotion-prediction alignment: position priors are predictive, but text-specific signals remain.
The study built LitArc-Bench using 39 Chinese literary excerpts and VAD ratings from 81 readers, decomposing model alignment with human emotional trajectories into narrative position priors, lexical sentiment cues, and text-specific response dynamics. Position priors alone predicted arousal at r=.470, yet after subtracting position and lexical contributions, models still showed text-specific advantages for valence and arousal transitions; identity matching significantly beat mismatched texts and direction recovery exceeded chance. However, arousal robustness was lower, and high shape consistency coexisted with scale bias, indicating that shape fitting, calibration, and individual differences must be assessed separately.
When a model predicts an emotion curve and seems to "get it," what is it actually tracking?
When reading a literary work, a reader's emotions rise and fall as the narrative unfolds. If an AI model can predict this emotional trajectory, does that mean it truly understands the text? The answer may not be so simple.
A new study points out that a model's alignment with human emotional trajectories may come from three fundamentally different sources: regularities inherent to narrative position, hints from sentiment words in the text, and reader-response dynamics genuinely triggered by a specific text. The first two are "shallow explanations"; only the third touches the core of literary reading.
To disentangle these three, the research team built LitArc-Bench—a benchmark specifically designed to separate these components.

Experimental design: how to decompose the sources of "alignment"
Data composition
LitArc-Bench is based on 39 Chinese literary excerpts, each divided into seven narrative checkpoints. 81 adults (from two class cohorts) completed a total of 430 questionnaires, yielding 9,020 VAD (valence–arousal–dominance) ratings. This forms a human emotional trajectory dataset with a temporal structure.
The VAD (valence–arousal–dominance) model is a three-dimensional emotion representation framework commonly used in affective computing. Valence refers to the positive or negative orientation of emotion, from pleasant to unpleasant; arousal refers to the level of activation or intensity of emotion, from calm to excited; dominance refers to an individual's sense of control over the emotional state, from being dominated to dominant. Compared with using only discrete emotion labels (such as joy, anger, sadness), VAD can more finely characterize continuously changing emotional states, and is especially suitable for analyzing emotional trajectories over time during literary reading. In this study, participants rated each narrative checkpoint on VAD precisely to capture dynamic changes in emotion across the three dimensions rather than simple categorization.
Three comparison models
The study set up three "future-blind" model runs:
- Position prior: predicts emotional trends based solely on the relative position of the text in the narrative (using leave-one-text-out);
- Lexical sentiment: makes judgments based solely on sentiment words appearing in the text;
- Human–human reference: uses other readers' responses to the same text as a reference.
By comparing model performance against these baselines, the researchers tried to answer: where does model alignment actually come from?
Key findings: position effects are significant, but not enough to explain everything
Position alone has predictive power
One result worth watching: based solely on narrative position, the model's prediction of arousal reached r = .470. This means that if one looks only at raw alignment scores, it is easy to overestimate the model's text-specific understanding—position alone can "guess" a considerable portion correctly.
Leave-one-text-out is a cross-validation strategy often used to evaluate a model's ability to generalize to unseen texts. Specifically, when training the position prior model, all data from one text is excluded each time, and only data from the remaining texts are used to estimate the "narrative position–emotion" mapping, after which predictions are made on the held-out text. This prevents the model from memorizing the emotional patterns of specific texts and ensures that the predictive power of the position prior reflects general regularities of narrative structure rather than overfitting to the training texts. This method is a key design choice for separating position effects from text-specific effects.
But text-specific signals do exist
The study did not stop there. For transition predictions of valence (V) and arousal (A), the text-bootstrap confidence intervals for the advantage of all six model–dimension combinations over the joint "position + lexical" control were all above zero. In other words, after subtracting the contributions of position and lexical sentiment, the models still captured human response dynamics associated with specific texts.
Text bootstrap is a resampling technique used to estimate confidence intervals for statistics. In this study, resampling with replacement is performed at the text level to generate multiple pseudo-datasets, and the model's advantage over the baseline is recomputed on each dataset. Through repeated resampling, an empirical distribution of the advantage values can be obtained, from which confidence intervals are constructed. If the confidence interval does not include zero (i.e., is above zero), it indicates that the model's performance beyond the baseline is not accidental. This method accounts for text-level variation and is more suitable for hierarchical data (such as readers nested within texts) than traditional sample bootstrap.
Identity recognition test
More compelling evidence comes from an identity-matching experiment with cross-fitted models: the model's match to the correct human text was better than to 38 mismatched texts in all nine model–dimension units (p < .0001). A text-invariant negative control, by contrast, fell back to chance level (p = 1). This shows that the model is indeed distinguishing response patterns elicited by different texts rather than relying on some generic bias.
Direction recovery ability
On samples where human emotional transitions deviated from the positional trend by at least 0.5 points, the model recovered 69–76% of valence directions and 53–68% of arousal and dominance directions. Direction recovery rates were clearly above chance, further supporting the recoverability of text-specific dynamics.
Robustness checks and open questions
Stability under resampling
After participant–text cross-resampling, all nine identity-recognition intervals remained above zero. The joint-control advantage for valence and directionality remained robust, but the advantage for arousal was less certain—suggesting that different emotional dimensions may differ in their sensitivity to text-specific signals.
The shape may match, but the scale may not
The study also found that high shape consistency can coexist with scale bias. In other words, a model may accurately depict the "shape" of emotional ups and downs while systematically deviating from human ratings in absolute values. This reminds us that shape fitting, calibration accuracy, and individual human differences need to be evaluated separately and should not be conflated.
What this means
The core contribution of LitArc-Bench is not to claim that models "understand" literature, but to provide an operational methodology: testing position effects, lexical effects, and text-specific responses separately. For affective computing and computational narratology, this kind of decomposition is necessary—otherwise, a model that has only learned that "stories start flat and peak at the climax" might also perform decently on aggregate metrics.
The study also exposes a deeper problem: differences among human readers are themselves a non-negligible source of variance. Alignment with the "average reader" is not the same as alignment with real reading experience. A three-part evaluation framework of shape, calibration, and individual differences may be a more honest direction for the next step.
One-line Summary
The new LitArc-Bench benchmark decomposes the sources of model alignment in emotion prediction and finds that position priors have predictive power while text-specific signals still remain.
Paragraph Summary
The study built the LitArc-Bench benchmark based on 39 Chinese literary excerpts and VAD ratings from 81 readers, decomposing model alignment with human emotional trajectories into three sources: narrative position priors, lexical sentiment cues, and text-specific response dynamics. Results showed that position priors alone predicted arousal at r=.470, but after subtracting the contributions of position and lexical sentiment, models still showed text-specific predictive advantages for valence and arousal transitions; identity-matching experiments significantly outperformed mismatched texts, and direction recovery rates were above chance. However, robustness in the arousal dimension was lower, and high shape consistency could coexist with scale bias, suggesting that shape fitting, calibration accuracy, and individual differences need to be evaluated separately.
Related articles

ClawDOS: Bringing AI Coding Capabilities to DOS
ClawDOS is an AI coding agent project for DOS, exploring the possibility of bringing AI coding assistance into resource-constrained retro computing environments. This article analyzes its positioning, use cases, and community response.

Baseten Releases Bison-1 and Groundhog-1, Topping the Decision Model Leaderboard
Baseten releases open-weight decision models bison-1 and groundhog-1 built on Gemma 4; bison-1 scores 76 on the AA Decision Index v2.1 to top the leaderboard. Both support 128k context and image input—here we analyze their capability combination, benchmark significance, and selection advice for developers.

Cline CLI v3.0.75 Released: Yolo Mode Now Default for One-Shot Runs
Cline CLI v3.0.75: one-shot runs now default to yolo mode, --timeout and Ctrl+C can interrupt the first round, read_files is no longer truncated to 8k chars, plus reasoning parameter fixes and 94 new providers.