[KongchangAI]
· 2 min read· 1,140 words

SAGE Framework: Detecting AI-Generated Peer Review Comments Using Subjectivity-Based Linguistic Features

SAGE Framework: Detecting AI-Generated Peer Review Comments Using Subjectivity-Based Linguistic Features

SAGE detects AI-generated review comments via semantic and pragmatic subjectivity features—lightweight, effective, and interpretable.

To tackle academic integrity challenges from LLM-generated peer reviews, an arXiv paper proposes SAGE, which uses subjectivity features: semantic-level evaluative language and modality, plus pragmatic-level speech acts. A lightweight classifier reaches 0.9566 AUROC; fused with Sem-Detect, it raises TPR at 1% FPR from 0.874 to 0.985 on APRDB. Features are traceable and open-sourced.

Why AI-Generated Review Comments Need Specialized Detection

With the growing adoption of large language models in academic writing, AI-generated peer review comments are emerging as a new challenge to academic integrity. Traditional AI-text detectors often rely on statistical patterns or watermarking techniques, but these methods tend to fail when faced with carefully tuned generated text. A paper on arXiv proposes the SAGE (Subjectivity-Aware Generation Estimator) framework, which starts from the linguistic notion of "subjectivity" and opens an interpretable new path for detecting AI-generated review comments.

SAGE framework diagram

The core insight of SAGE is that human reviewers naturally express subjective judgments, hedging expressions, and specific speech acts when evaluating papers, whereas AI-generated text tends to show systematic differences along these dimensions. The researchers decompose subjectivity into two levels—evaluative language and modality at the semantic level, and speech acts at the pragmatic level—and build a lightweight classifier accordingly.

How SAGE Quantifies "Subjectivity"

Semantic Level: Evaluative Language and Modality

At the semantic level, SAGE focuses on evaluative vocabulary and modal verbs in review comments. Human reviewers often use modal words expressing uncertainty such as "might," "seem," and "I think," as well as evaluative adjectives such as "insufficient," "strong," and "convincing." These linguistic features reflect the reviewer's subjective stance, whereas AI-generated text may be overly neutral or formulaic along these dimensions. SAGE uses model-predicted subjectivity scores as features to capture these subtle linguistic differences.

Modality is the grammatical category in linguistics that expresses a speaker's attitude toward a proposition (e.g., possibility, necessity, certainty). In English, modality is typically realized through modal verbs (may, might, could, should, must) and modal adverbs (possibly, probably, certainly). Systemic functional linguistics (Halliday) divides modality into modalization (involving possibility and usuality) and modulation (involving obligation and inclination). The use of modality in review comments reflects the reviewer's degree of certainty about judgments of paper quality: human reviewers often soften criticism with low-value modals such as "might" and "seems," preserving room for negotiation; AI-generated text may overuse high-value modals or exhibit abnormally uniform modality distributions. By using modality word frequency and types as semantic-level features, SAGE exploits exactly this systematic difference.

Pragmatic Level: Speech Act Annotation

At the pragmatic level, SAGE introduces speech act theory to annotate sentences in review comments, such as "suggestion," "criticism," "question," and "summary." Review comments are a highly structured sequence of speech acts; human reviewers typically follow a "summary–evaluation–suggestion" pragmatic pattern, whereas AI-generated text may deviate in the distribution and transitions of speech acts. SAGE extracts features from these annotations to train the classifier.

Speech Act Theory was proposed by Austin (1962) and systematized by Searle (1969); its core idea is that "saying is doing"—language not only describes the world but also performs actions (such as promising, requesting, and criticizing). Searle classifies speech acts into five types: assertives, directives, commissives, expressives, and declarations. In the peer review context, typical speech acts include summarizing the paper's contributions (assertives), pointing out shortcomings (expressives/assertives), proposing revisions (directives), and asking questions (directives). The pragmatic structure of review comments is genre-conventional: experienced reviewers usually organize discourse in the order of "summary—criticism—suggestion," and the type distribution and sequence transition patterns of speech acts can serve as stylistic fingerprints. By annotating sentences with speech acts and extracting distributional features, SAGE turns this pragmatic framework into computable signals.

Detection Performance: Lightweight but Effective

SAGE trains only a lightweight classifier using the above linguistic features and achieves an AUROC of 0.9566 on independent tests. This result shows that subjectivity-based linguistic features alone have strong discriminative power, without relying on large-scale pretrained models or complex ensemble methods.

In more detail, SAGE's output can be combined with the current state-of-the-art detector Sem-Detect. On the APRDB benchmark, after fusing SAGE with Sem-Detect, Sem-Detect's true positive rate at a 1% false positive rate increases from 0.874 to 0.985, surpassing the current state of the art. This plug-and-play property allows SAGE to enhance existing detection systems at low cost.

AUROC (Area Under the Receiver Operating Characteristic Curve) is the most commonly used threshold-independent metric in binary detection tasks, with 0.5 indicating random guessing and 1.0 indicating perfect separation. An AUROC of 0.9566 means that when randomly drawing one AI-generated sample and one human sample, the model has a 95.66% probability of assigning a higher AI-probability score to the AI sample. In high-stakes scenarios such as academic integrity detection, the true positive rate (TPR) at a fixed false positive rate (FPR) is more practically meaningful: a 1% FPR means only a 1% chance of misclassifying human text as AI, and under this strict constraint, the higher the TPR, the more usable the system. After fusing SAGE with Sem-Detect, the TPR at 1% FPR increases from 0.874 to 0.985, indicating that its incremental value is mainly concentrated in the low-false-alarm operating region—precisely the most critical region for real-world deployment.

Interpretability and Open-Source Contributions

Another advantage of SAGE is interpretability. Unlike black-box detectors, each feature of SAGE corresponds to a specific linguistic concept, such as the frequency of a particular modal word or the proportion of a certain speech act. When SAGE determines that a passage is AI-generated, researchers can trace which linguistic features triggered the judgment, which is especially important for appeals and reviews in academic integrity investigations.

The research team has released the code and models on GitHub (Arys02/generated_text_detection_hub) and Hugging Face (Arys02) to facilitate reproduction and integration by other researchers. This openness helps promote standardized comparison and continuous improvement in the field of AI-generated text detection.

Limitations and Future Directions

SAGE currently focuses on the specific genre of peer review comments, and whether its linguistic features apply to other types of academic text (such as paper bodies and grant applications) remains to be verified. In addition, subjectivity features may be influenced by individual reviewer style, field conventions, and cultural background, and cross-domain generalization capability requires further study. Nevertheless, SAGE provides an interpretable and integrable detection approach, offering a new tool for addressing the academic integrity challenges posed by AI-generated content.

One-line Summary

SAGE detects AI-generated review comments using semantic and pragmatic subjectivity features—lightweight, effective, and interpretable.

Paragraph Summary

To address the academic integrity challenges posed by peer review comments generated by large language models, an arXiv paper proposes the SAGE framework, which approaches AI-generated text detection from the linguistic notion of "subjectivity." SAGE decomposes subjectivity into the semantic level (evaluative language and modality) and the pragmatic level (speech acts), extracts features to train a lightweight classifier, and achieves an AUROC of 0.9566 on independent tests. After fusion with the state-of-the-art detector Sem-Detect, the true positive rate at a 1% false positive rate on the APRDB benchmark increases from 0.874 to 0.985. Each feature of SAGE corresponds to a specific linguistic concept, making judgments traceable and suitable for appeals and reviews in academic integrity investigations; the code and models are open-sourced.

#AI-generated text detection#peer review#SAGE framework#subjectivity features#speech acts#AUROC
Share:

Related articles