SAGE Framework: Detecting AI-Generated Peer Review Comments Using Subjectivity-Based Linguistic Features

SAGE detects AI-generated review comments via semantic and pragmatic subjectivity features—lightweight, effective, and interpretable.
To tackle academic integrity challenges from LLM-generated peer reviews, an arXiv paper proposes SAGE, which uses subjectivity features: semantic-level evaluative language and modality, plus pragmatic-level speech acts. A lightweight classifier reaches 0.9566 AUROC; fused with Sem-Detect, it raises TPR at 1% FPR from 0.874 to 0.985 on APRDB. Features are traceable and open-sourced.
Why AI-Generated Review Comments Need Specialized Detection
With the growing adoption of large language models in academic writing, AI-generated peer review comments are emerging as a new challenge to academic integrity. Traditional AI-text detectors often rely on statistical patterns or watermarking techniques, but these methods tend to fail when faced with carefully tuned generated text. A paper on arXiv proposes the SAGE (Subjectivity-Aware Generation Estimator) framework, which starts from the linguistic notion of "subjectivity" and opens an interpretable new path for detecting AI-generated review comments.

The core insight of SAGE is that human reviewers naturally express subjective judgments, hedging expressions, and specific speech acts when evaluating papers, whereas AI-generated text tends to show systematic differences along these dimensions. The researchers decompose subjectivity into two levels—evaluative language and modality at the semantic level, and speech acts at the pragmatic level—and build a lightweight classifier accordingly.
How SAGE Quantifies "Subjectivity"
Semantic Level: Evaluative Language and Modality
At the semantic level, SAGE focuses on evaluative vocabulary and modal verbs in review comments. Human reviewers often use modal words expressing uncertainty such as "might," "seem," and "I think," as well as evaluative adjectives such as "insufficient," "strong," and "convincing." These linguistic features reflect the reviewer's subjective stance, whereas AI-generated text may be overly neutral or formulaic along these dimensions. SAGE uses model-predicted subjectivity scores as features to capture these subtle linguistic differences.
Modality is the grammatical category in linguistics that expresses a speaker's attitude toward a proposition (e.g., possibility, necessity, certainty). In English, modality is typically realized through modal verbs (may, might, could, should, must) and modal adverbs (possibly, probably, certainly). Systemic functional linguistics (Halliday) divides modality into modalization (involving possibility and usuality) and modulation (involving obligation and inclination). The use of modality in review comments reflects the reviewer's degree of certainty about judgments of paper quality: human reviewers often soften criticism with low-value modals such as "might" and "seems," preserving room for negotiation; AI-generated text may overuse high-value modals or exhibit abnormally uniform modality distributions. By using modality word frequency and types as semantic-level features, SAGE exploits exactly this systematic difference.
Pragmatic Level: Speech Act Annotation
At the pragmatic level, SAGE introduces speech act theory to annotate sentences in review comments, such as "suggestion," "criticism," "question," and "summary." Review comments are a highly structured sequence of speech acts; human reviewers typically follow a "summary–evaluation–suggestion" pragmatic pattern, whereas AI-generated text may deviate in the distribution and transitions of speech acts. SAGE extracts features from these annotations to train the classifier.
Speech Act Theory was proposed by Austin (1962) and systematized by Searle (1969); its core idea is that "saying is doing"—language not only describes the world but also performs actions (such as promising, requesting, and criticizing). Searle classifies speech acts into five types: assertives, directives, commissives, expressives, and declarations. In the peer review context, typical speech acts include summarizing the paper's contributions (assertives), pointing out shortcomings (expressives/assertives), proposing revisions (directives), and asking questions (directives). The pragmatic structure of review comments is genre-conventional: experienced reviewers usually organize discourse in the order of "summary—criticism—suggestion," and the type distribution and sequence transition patterns of speech acts can serve as stylistic fingerprints. By annotating sentences with speech acts and extracting distributional features, SAGE turns this pragmatic framework into computable signals.
Detection Performance: Lightweight but Effective
SAGE trains only a lightweight classifier using the above linguistic features and achieves an AUROC of 0.9566 on independent tests. This result shows that subjectivity-based linguistic features alone have strong discriminative power, without relying on large-scale pretrained models or complex ensemble methods.
In more detail, SAGE's output can be combined with the current state-of-the-art detector Sem-Detect. On the APRDB benchmark, after fusing SAGE with Sem-Detect, Sem-Detect's true positive rate at a 1% false positive rate increases from 0.874 to 0.985, surpassing the current state of the art. This plug-and-play property allows SAGE to enhance existing detection systems at low cost.
AUROC (Area Under the Receiver Operating Characteristic Curve) is the most commonly used threshold-independent metric in binary detection tasks, with 0.5 indicating random guessing and 1.0 indicating perfect separation. An AUROC of 0.9566 means that when randomly drawing one AI-generated sample and one human sample, the model has a 95.66% probability of assigning a higher AI-probability score to the AI sample. In high-stakes scenarios such as academic integrity detection, the true positive rate (TPR) at a fixed false positive rate (FPR) is more practically meaningful: a 1% FPR means only a 1% chance of misclassifying human text as AI, and under this strict constraint, the higher the TPR, the more usable the system. After fusing SAGE with Sem-Detect, the TPR at 1% FPR increases from 0.874 to 0.985, indicating that its incremental value is mainly concentrated in the low-false-alarm operating region—precisely the most critical region for real-world deployment.
Interpretability and Open-Source Contributions
Another advantage of SAGE is interpretability. Unlike black-box detectors, each feature of SAGE corresponds to a specific linguistic concept, such as the frequency of a particular modal word or the proportion of a certain speech act. When SAGE determines that a passage is AI-generated, researchers can trace which linguistic features triggered the judgment, which is especially important for appeals and reviews in academic integrity investigations.
The research team has released the code and models on GitHub (Arys02/generated_text_detection_hub) and Hugging Face (Arys02) to facilitate reproduction and integration by other researchers. This openness helps promote standardized comparison and continuous improvement in the field of AI-generated text detection.
Limitations and Future Directions
SAGE currently focuses on the specific genre of peer review comments, and whether its linguistic features apply to other types of academic text (such as paper bodies and grant applications) remains to be verified. In addition, subjectivity features may be influenced by individual reviewer style, field conventions, and cultural background, and cross-domain generalization capability requires further study. Nevertheless, SAGE provides an interpretable and integrable detection approach, offering a new tool for addressing the academic integrity challenges posed by AI-generated content.
One-line Summary
SAGE detects AI-generated review comments using semantic and pragmatic subjectivity features—lightweight, effective, and interpretable.
Paragraph Summary
To address the academic integrity challenges posed by peer review comments generated by large language models, an arXiv paper proposes the SAGE framework, which approaches AI-generated text detection from the linguistic notion of "subjectivity." SAGE decomposes subjectivity into the semantic level (evaluative language and modality) and the pragmatic level (speech acts), extracts features to train a lightweight classifier, and achieves an AUROC of 0.9566 on independent tests. After fusion with the state-of-the-art detector Sem-Detect, the true positive rate at a 1% false positive rate on the APRDB benchmark increases from 0.874 to 0.985. Each feature of SAGE corresponds to a specific linguistic concept, making judgments traceable and suitable for appeals and reviews in academic integrity investigations; the code and models are open-sourced.
Related articles

Why Microsoft CEO Nadella Still Backs Nvidia After Jensen Huang's Departure
Microsoft CEO Nadella publicly expresses confidence in Nvidia interim CEO Colette Kress. This article analyzes Nvidia's power transition after Jensen Huang's departure, Microsoft's compute dependency, and the trust logic of the AI supply chain.

Nadella: Critical Infrastructure Is 'Unpatchable' as AI Cybersecurity Arms Race Escalates
Microsoft CEO Nadella addresses AI cyberattack fears, saying much critical infrastructure is unpatchable, downplaying the Outlook outage, and analyzing AI's dual role in cybersecurity.

Musk Backs Jensen Huang: Firing Him Is Like Cutting Jordan
Musk says Jensen Huang is loyal to the U.S. and likens firing him to cutting Michael Jordan—highlighting politicization anxieties in the AI chip industry and defending talent value.