PISA 2025 Evidence: How Do AI Text Detectors Perform on Real Student Writing?

Real PISA writing data show AI detectors are reliable at the text level but clearly fragile at the sentence level and under adversarial rewriting.
A study based on real PISA 2025 writing data used keystrokes, clipboard, and chat logs to trace every character of essays and evaluate AI text detectors in real student writing scenarios. Both feature-based and Llama-3.1-8B embedding-based detectors performed reliably at distinguishing AI from human text at the text level, with macro F1 up to 0.96 and sentence-level up to 0.95, unaffected by spelling errors. But robustness differed markedly: the feature detector relies on lexical complexity, and simple student-style rewriting reduced AI recall from 0.92 to 0.60 or even 0.41; the embedding detector stayed robust on full texts but had sentence-level recall as low as 0.59. Mixed-text results fell in between, and about 95% of chatbot-used-but-trace-free text was classified as human. The study suggests detectors should not be the sole basis for academic integrity, and provenance process data offers an alternative approach.
AI text detectors often approach perfection in lab evaluations, but real student writing scenarios are far more complex—AI use is voluntary, sporadic, and may involve nothing more than polishing a few words. A study based on real PISA 2025 writing data provides the most detailed answer yet to this challenge.

Study Design: Tracing the Origin of Every Character
The study used the PISA 2025 FLA (Foreign Language Assessment) dataset, with a sample of English argumentative essays written by 15-year-old German students. Students wrote under two conditions: one involving pure human writing, and another where they could use assistance from a chatbot powered by gpt-4o-mini.
The key lies in the data collection method. The research team recorded students' keystrokes, clipboard activity, and chat logs, enabling them to trace every character in the final essay back to its source. Based on this "provenance" information, researchers labeled both texts and sentences as: purely human-written, AI-generated, mixed text, and "chatbot-used-but-no-AI-trace" text.
This represents a major step forward from traditional detection research. Most detector evaluations use clean "all-human vs. all-AI-generated" comparisons, whereas real classroom AI use tends to be partial and fragmented—a student might ask AI to rewrite just one sentence, or reference AI suggestions and then rewrite in their own words.
PISA 2025 FLA (Foreign Language Assessment) is a newly added foreign language assessment module in the OECD's 2025 cycle of the Programme for International Student Assessment, focusing on 15-year-olds' foreign language production in authentic task contexts. The subset used in this study consists of argumentative essays written in English by German students—semi-authentic writing data with uniform task constraints that preserves real students' language proficiency and writing habits (including spelling errors and colloquial expressions) while offering comparability and a traceable collection environment. Compared with "clean" corpora generated by adults or models in laboratories, this type of data is closer to classroom reality, but also brings limitations such as annotation relying on process logs and restricted sample language and genre—precisely the direction of the open questions noted at the end.
Detection Performance: Reliable at the Text Level, Fragile at the Sentence Level
The study evaluated two types of detection systems: one based on traditional feature-based detectors, and another using classifiers based on frozen Llama-3.1-8B embeddings. Both were adapted from top-ranked systems in shared tasks.
In distinguishing AI text from human text, both performed reliably: text-level macro-averaged F1 reached as high as 0.96, and sentence-level up to 0.95, unaffected by spelling errors. This result indicates that even in the noisy environment of real writing, detectors retain considerable discriminative ability.
However, robustness tests revealed significant differences. The feature-based detector relies on lexical complexity as a signal, and a simple "student-style rewriting" prompt could substantially weaken it—after rewriting with gpt-4o-mini, AI recall dropped from 0.92 to 0.60; after rewriting with Qwen3, it fell further to 0.41. In other words, as long as AI-generated content is deliberately "dumbed down" into a student voice, the feature detector becomes nearly ineffective.
The embedding-based detector was more robust on full texts, maintaining AI recall above 0.94; but it was equally fragile when facing individual sentences, with AI recall as low as 0.59.
Macro-averaged F1 is a commonly used comprehensive metric in classification tasks: first, the F1 score (the harmonic mean of precision and recall) is calculated separately for each class (here, "AI text" and "human text"), and then the arithmetic mean is taken across class F1 scores. Unlike micro-averaging, macro-averaging gives equal weight to each class, so when the two classes are imbalanced in sample size (e.g., human text far outnumbers AI text), macro-averaging more sensitively reflects detection quality on the minority class. The reported text-level macro F1 of 0.96 means the detector achieves balanced and high precision and recall on both classes, rather than inflating scores by merely predicting the majority class.
AI recall refers to the proportion of all samples that are truly AI-generated (or contain traceable AI content) that are correctly identified as AI by the detector. A drop in recall means more AI text is missed and misclassified as human-written—precisely the most dangerous type of error in academic integrity scenarios. Therefore, the study uses the change in AI recall before and after rewriting (0.92→0.60→0.41) to quantify the extent to which the detector is circumvented by "student-style rewriting," which better illustrates failure risk under adversarial conditions than accuracy alone.
Feature-based detectors and embedding-based detectors represent two different technical approaches. The former relies on manually designed interpretable features, such as lexical complexity (proportion of rare words, word length distribution), syntactic complexity, punctuation and function word frequencies, which are then fed to traditional classifiers (e.g., logistic regression, gradient boosted trees). Its advantages are lightweight and interpretable, but its weaknesses are also obvious: once rewriting tools deliberately lower lexical complexity and mimic a student voice, these surface signals are flattened out. The latter uses a pretrained language model (here, a frozen Llama-3.1-8B) to encode text into dense vectors, then trains a classification head on the embedding space. "Frozen" means the encoder parameters are not updated and only the top-level classifier is trained, which is computationally cheap and less prone to overfitting on small samples. Embeddings capture deeper semantic and distributional patterns, hence greater robustness on full texts; but individual sentences carry limited information, reducing embedding discriminability and leading to fragile sentence-level detection.
Mixed Text and Trace-Free Text: The Gray Zone of Detection
The study also examined two special types of text. Detection results for mixed text (alternating human and AI writing) fell between the two classes, consistent with intuition—it is neither purely human nor purely AI.
More noteworthy is "no-AI-trace" text: students used the chatbot, but no traceable AI-generated content could be found in the final essay. Such text was classified as human-written in about 95% of cases. This is actually reasonable—if students reference AI suggestions and then completely rewrite in their own words, the final product is indeed closer to human writing. But it also reminds us that a detector's "human" judgment does not equal "no AI use."
The classification logic for "no-AI-trace" text deserves clarification at the level of detection objectives. Records for such text show that students did call the chatbot, but no character in the final draft can be traced to AI output—students may have read AI suggestions and completely rewritten in their own wording. The detector classifying about 95% of these as human-written is less a "misjudgment" than an exposure of the inherent goal mismatch of the detection task itself: the detector judges "whether the surface text was generated by AI," not "whether AI was used during the writing process." These two are not equivalent in real classrooms. This also explains why the study emphasizes the value of provenance data (keystrokes, clipboard, chat logs)—process evidence can answer "whether AI was used," while post-hoc text detection can only answer "whether traces were left." The two are complementary, not substitutes.
Implications for Education and Detection Practice
This study conveys several signals that educators and tool developers should take seriously.
First, detectors perform well on the binary classification of "AI vs. human," but the forms of AI use in real scenarios are far more complex than a binary classification. Mixed writing, reference-then-rewrite, and localized polishing all blur the boundaries of detection.
Second, detector robustness depends heavily on the signals it relies on. Surface features such as lexical complexity are easily bypassed by rewriting prompts; embedding-level features are more reliable on long texts, but sentence-level judgments remain unreliable. A single detector should not be the sole basis for academic integrity determinations.
Third, provenance data (keystrokes, clipboard, chat logs) offers an approach different from "post-hoc detection": rather than guessing whether text was generated by AI, it records the creative trajectory during the writing process. This has direct reference value for the design of assessment tools.
The study also leaves open questions: current results are based on English argumentative writing by 15-year-old German students, and performance on other languages, other genres, and other age groups remains to be verified. The adversarial contest between detectors and rewriting tools will not stop here either.
One-line Summary
Real PISA writing data show AI detectors are reliable at the text level but clearly fragile at the sentence level and under adversarial rewriting conditions.
Paragraph Summary
A study based on real PISA 2025 writing data used keystrokes, clipboard, and chat logs to trace every character of essays and evaluate AI text detectors in real student writing scenarios. Results show that two types of detectors—feature-based and based on Llama-3.1-8B embeddings—performed reliably in distinguishing AI from human text at the text level, with macro F1 up to 0.96 and sentence-level up to 0.95, unaffected by spelling errors. However, robustness differences were significant: the feature detector relies on lexical complexity, and simple student-style rewriting could reduce AI recall from 0.92 to 0.60 or even 0.41; the embedding detector remained robust on full texts but had sentence-level recall as low as 0.59. Mixed-text results fell in between, and about 95% of text with chatbot use but no AI traces was classified as human. The study suggests detectors should not be the sole basis for academic integrity, and provenance process data offers an alternative approach.
Related articles

Robinhood's Hybrid AI Reviewer: 60% of Data PRs Auto-Merge
Robinhood's data platform team shares a hybrid AI PR reviewer: deterministic scripts verify facts, AI Agents judge contextual risk, and humans keep key decisions. About 60% of data pipeline PRs merge without human review, cutting review workload ~75%, saving ~132 engineering hours monthly, and reducing related incidents ~30%.

Making Marketing Videos with Claude Code Is More Efficient Than AI Video Generation
Generate marketing videos programmatically with Claude Code—more flexible and cheaper than AI video generators. Reference a style, generate a storyboard, output content as code, and make edits instantly.

Open-Weight Models Are Falling Behind: Three Structural Reasons
A veteran developer uses his platform's usage data and public benchmark experiments to argue why open-weight models are falling behind closed-source frontier models.