[KongchangAI]
· 2 min read· 1,264 words

Commercial AI Detector Breached by 'Style Mimicry': Evidence of Score Reduction via Rewriting

Commercial AI Detector Breached by 'Style Mimicry': Evidence of Score Reduction via Rewriting

Gemini style mimicry of 55 authors cuts QuillBot AI detector median score from 100% to 0%.

A study found that author-conditioned style transfer using Gemini to rewrite 991 AI answers in the styles of 55 authors dropped QuillBot AI detector scores from a 100% median to 0%. The mean fell from 67.9% to 11.2%, with 66% of rewrites scoring exactly 0%. Effects varied by author—10 authors' rewrites all scored 0%, while 6 authors' rewrites still averaged above 50%. The authors caution results apply only to the QuillBot-Gemini combination.

Article Content

A recent study reveals a significant weakness in commercial AI text detectors: when AI-generated text is rewritten through "Author-Conditioned Style Transfer," the AI probability scores given by detectors drop dramatically. The study targeted the QuillBot AI detector, using Gemini to rewrite 991 AI-generated answers in the style of 55 specified authors. The median score dropped from 100% directly to 0%, while the mean fell from 67.9% to 11.2%. This means that merely changing the writing style may be enough to bypass the judgment of current commercial detection tools.

Study source screenshot

Experimental Design: Driving Style Transfer with 'Author Samples'

The research team's core approach was not complex: first, they prepared a batch of AI-generated answers (991 in total), then asked Gemini to rewrite these answers while referencing the writing samples of a specific author. The prompts included text fragments from 55 named authors, and the model needed to imitate the sample's word choice, sentence structure, and rhythm. Each original answer and its rewritten version were scored by the QuillBot AI detector, forming paired comparisons.

This design treats "style" as the sole manipulated variable, rather than simply swapping synonyms or rearranging paragraphs. The author-conditioned approach makes the rewritten text more closely resemble a specific human writing exemplar in surface features, thereby interfering with the detector's discrimination logic based on stylistic statistics.

Style Transfer is a class of tasks in natural language processing that aims to change the expressive style of text (e.g., formal/casual, different authorial tones, different genres) while preserving semantic content. Traditional style transfer mostly targets coarse-grained attributes such as sentiment or politeness level, whereas author-conditioned style transfer further concretizes "style" into a specific author's writing fingerprint—word preferences, syntactic complexity, punctuation habits, sense of rhythm, etc. Such methods typically use a small number of author samples as conditioning to guide large models to imitate the target style during generation. Because large models possess powerful in-context learning capabilities, a single passage of original author text is sufficient to achieve fairly realistic style imitation. This greatly lowers the barrier to style transfer, making it a potential means of evading AI detectors based on stylistic statistics.

Score Cliff: Median Drops to Zero, Mean Falls Sharply

The detection results show an extreme distribution shift. The median score of original AI answers was 100%, meaning the detector was almost certain they were AI-generated; the median after rewriting dropped directly to 0%, a complete reversal. The mean fell from 67.9% to 11.2%, indicating that even accounting for outliers, the overall shift remains enormous.

More detailed statistics show that 66% of rewritten versions received an exact 0% score. Among the 699 samples with original scores of at least 50%, 71% of rewritten versions scored below 10%. The paired Wilcoxon signed-rank test yielded a p-value below 1e-100, indicating that this decline is not random fluctuation.

The Wilcoxon signed-rank test is a non-parametric statistical test used to compare whether two paired samples differ significantly, without requiring the data to follow a normal distribution. It calculates ranks based on the magnitude and sign of paired differences, making it particularly suitable for this study's paired design of "the same original answer vs. its rewritten version." A p-value below 1e-100 means that if the true situation were no difference in scores before and after rewriting, the probability of observing such a consistent and large score decline would be extremely low, thus supporting that style transfer indeed systematically lowered detector scores. This test is robust to outliers and is more suitable than the paired t-test for detecting skewed data such as scores.

The Influence of Author Identity: Some 'Vanish' Successfully, Others Still Detected

The study further found that different author samples produced significantly different evasion effects. Among the 55 authors, all rewritten versions associated with 10 authors scored 0%, equivalent to complete "invisibility"; while for another 6 authors, the average score of their corresponding rewrites remained above 50%, with the detector still maintaining high vigilance. The Kruskal-Wallis test yielded a p-value below 1e-80, indicating that differences among authors are statistically significant.

This result suggests that the effectiveness of style transfer is not uniform; certain writing styles may naturally be closer to the feature space the detector judges as "human."

The Kruskal-Wallis test is a multi-group extension of the Wilcoxon test, used to determine whether three or more independent samples come from the same distribution, and is likewise a non-parametric method. In this study, it treats the 55 authors as 55 groups and tests whether there are overall differences in the distribution of rewrite scores corresponding to different author samples. A p-value below 1e-80 indicates that author identity has a highly significant effect on evasion effectiveness, i.e., certain authors' styles are indeed more likely to cause the detector to "misjudge" the text as human writing. This result suggests that the detector's decision boundary is not uniform in style space, and there may exist certain "human style blind spots" triggered by specific author samples.

Vocabulary Reuse Correlates with Score Decline, but Is Not the Whole Story

The study also examined the degree to which rewritten text reused five-word sequences from author samples. The data show that the more five-word sequences from author samples were reused, the lower the detection score. However, even for rewritten versions with a reuse rate below 5%, the average score was still 15.2%. This indicates that the score decline cannot be attributed solely to direct copying of fragments; deeper imitation at the style level also plays a role.

Limitations and Implications: One Detector, One Rewriting System

The authors explicitly caution that these results describe only the combination of one detector (QuillBot AI detector) and one rewriting system (Gemini) and cannot be generalized to all detectors. In addition, the study lacks a generic rewriting baseline (i.e., ordinary rewriting without specifying author style), so it is impossible to determine how much additional advantage style conditioning provides over ordinary rewriting. The study also did not measure the degree of semantic preservation before and after rewriting, and the rewritten text is noticeably longer than the original, which may itself affect detector judgment.

These limitations do not diminish the cautionary significance of the core finding: AI text detectors that rely on a single stylistic feature may be very fragile when faced with targeted style transfer. For scenarios that need to identify AI-generated content, detection tools need to integrate signals from more dimensions rather than drawing conclusions based solely on stylistic statistics.

One-line Summary

Research shows that using Gemini to rewrite AI text in the style of specified authors can reduce QuillBot detector median scores from 100% to 0%.

Paragraph Summary

A recent study on the QuillBot AI detector found that when AI-generated text is rewritten through "author-conditioned style transfer," detector scores plummet. The study used Gemini to imitate the writing styles of 55 specified authors and rewrote 991 AI-generated answers. The median score dropped from 100% to 0%, the mean fell from 67.9% to 11.2%, and 66% of rewritten versions received an exact 0% score. Different author samples produced significantly different effects: all rewrites associated with 10 authors scored 0%, while the average score for 6 authors remained above 50%. Vocabulary reuse correlated with score decline, but versions with a reuse rate below 5% still averaged 15.2%, indicating that deeper style imitation also plays a role. The authors caution that the conclusions apply only to the QuillBot and Gemini combination and that the study lacks a generic rewriting baseline and semantic preservation measurement, but the core finding warns that detectors relying on a single stylistic feature may be very fragile.

#AI text detector#style transfer#QuillBot#Gemini#author-conditioned rewriting#AI detection evasion
Share:

Related articles