AI Text Watermarks Struggle to Meet EU Rules? OpenAI's textGrain Testing Exposes Weaknesses

Title

AI Text Watermarks Struggle to Meet EU Rules? OpenAI's textGrain Testing Exposes Weaknesses

Meta Description

The EU AI Act requires watermarks on AI-generated text. OpenAI's textGrain tests show detection rates drop sharply on short texts and after editing. Experts note the lack of independent verification and measurable standards, and OpenAI may face fines of up to 7% of annual revenue.

Article Content

The transparency obligations under Article 50 of the EU AI Act have taken effect, requiring systems that generate synthetic audio, images, video, or text to mark their outputs in a machine-readable format. OpenAI has introduced a text watermarking scheme called textGrain for this purpose, but third-party testing and expert review indicate that the technology is fragile when dealing with short texts and post-editing, and whether it can meet legal requirements remains uncertain.

textGrain's Technical Approach: Treating "Entropy" as a Budget to Spend

According to a technical report published by OpenAI, when a large language model generates each token (a word or word fragment), it first computes the probability distribution over candidate words and then samples from it. The watermarking approach uses a secret key to generate pseudorandom values that subtly influence the model's token selection, thereby leaving a statistical pattern in the text that only a detector holding the same key can identify.

What sets textGrain apart is that it introduces "entropy" as a budgetable resource. Entropy measures the degree of uncertainty in the model's word choices: developers can specify how much uncertainty to allow, and textGrain adjusts the watermark strength accordingly. Xiang Li, a postdoctoral researcher at the University of Pennsylvania and co-author of the report, describes entropy as a sum of money, and this method can precisely determine how much to spend. This gives developers control over the trade-off between "detectable signal" and "output diversity."

The watermark is invisible to readers, but the detection results themselves have interpretive limits: a positive result cannot indicate the proportion of human contribution in a given piece of text, and a negative result cannot prove that the text was written by a human.

Tokens and Watermark Embedding

When a large language model generates text, it predicts one token at a time as the basic unit. Each token corresponds to a word or word fragment in the vocabulary; the model outputs a probability distribution over the entire vocabulary and then samples accordingly. Text watermarking exploits precisely this sampling step: a secret key generates a pseudorandom sequence that applies a slight bias to the probabilities of candidate tokens during sampling, so that the generated text exhibits a statistical pattern that only key holders can detect. Because the bias is small, human readers can barely perceive it, but a detector can use statistical tests to determine whether the text contains a watermark. The detection capability of this method depends heavily on text length—the more tokens there are, the more statistical evidence can accumulate.

Short Texts and Editing: The Two Weak Spots of Watermark Signals

Test data published by OpenAI reveals the practical limitations of the watermark. In psychological question tests based on the ELI5 dataset (a collection of Q&A from Reddit's /r/explainlikeimfive), a 200-token text of about 150 words had a watermark detection rate of about 80%, while a 400-token text of about 300 words reached a detection rate of 95%. In other words, the shorter the text, the less evidence is available for detection and the more reliability declines.

Post-editing is even more damaging. In evaluations of 400-token paragraphs, replacing only 10% of the words with synonyms dropped the detection rate from 92% to 66%; when the replacement ratio rose to 25%, the detection rate fell to 17%. Li explains that the watermark depends on preceding tokens, so changing one word disrupts signals at multiple positions. This is a "fundamental limitation" of text watermarking technology that is difficult to fully overcome even with more compute.

OpenAI also acknowledges that watermarking alone is insufficient to determine whether content was AI-generated, and that textGrain will complement other provenance tools, such as Content Credentials in metadata and Google DeepMind's SynthID watermark for images and audio.

The Statistical Nature of Watermark Detection Rates

Watermark detection is essentially a statistical hypothesis testing problem: the detector needs to accumulate enough token-level evidence to determine, with a low false-positive rate, that a text contains a watermark. The shorter the text, the smaller the sample size, the weaker the statistical power, and the naturally lower the detection rate. The detection rates reported in OpenAI's paper—about 80% for 200-token texts and about 95% for 400-token texts—reflect exactly this statistical regularity. Similarly, synonym substitution directly destroys the watermark signal on the substituted tokens and indirectly affects the detection statistics of subsequent tokens, so a small amount of editing can cause a sharp drop in detection rates. This is not a flaw unique to textGrain but a fundamental limitation shared by all token-sampling-based text watermarking schemes.

The Ambiguity of EU Regulation: What Counts as "Effective"?

There is obvious room for interpretation at the regulatory level. Article 50 requires technical solutions to be "effective, interoperable, robust, and reliable" within the limits of technical feasibility, but it does not set a quantifiable threshold. The EU's Code of Practice provides a voluntary compliance path, which stipulates that watermarking is not mandatory for outputs of fewer than 200 tokens—an exemption that happens to cover the range where textGrain performs worst.

Thomas Souverain, a postdoctoral researcher in AI ethics at the Paris School of Advanced Studies, believes the legislative wording leaves room for interpretation. He notes that textGrain's early methodology appears sound and may be better than competitors at identifying AI-generated text. OpenAI claims that in internal tests it matches or outperforms Google's SynthID Text, but these results have not yet been independently verified.

Article 50 of the EU AI Act and the Code of Practice

Article 50 of the EU AI Act specifically sets out transparency obligations for generative AI, requiring providers of synthetic audio, images, video, and text to mark outputs in a machine-readable format so that content can be identified as AI-generated. The provision uses qualitative terms such as "effective, interoperable, robust, and reliable" but does not set quantitative thresholds, thus leaving room for interpretation. The EU Code of Practice is an accompanying voluntary compliance tool that provides companies with specific operational guidance; it exempts outputs of fewer than 200 tokens from watermarking requirements on the grounds that short-text watermarking is technically difficult to implement reliably. This exemption overlaps with the range where textGrain performs worst, meaning that in short-text scenarios companies may be able to claim compliance without mandatory watermarking.

Lack of Independent Verification Leaves Compliance Risk Unresolved

The current evidence base is quite limited. Souverain emphasizes that the outside world has only one technical study, no peer-reviewed academic paper, and no access to the precise testing protocol. OpenAI is working with a limited group of researchers, including John Thickstun, an assistant professor of computer science at Cornell University, and Dominik Macko, a senior researcher at the Kempelen Institute of Intelligent Technologies (KInIT) in Slovakia. Both say they have just received access and have not yet begun their evaluations; Macko's team plans to test textGrain's robustness across different topics, languages, and text lengths.

If textGrain is ultimately found not to meet EU requirements, Souverain estimates that OpenAI could face fines of up to 7% of annual revenue. He also calls on OpenAI to open-source textGrain so that researchers outside the team can try to break the system and discover flaws in steps such as paraphrasing and translation. "I understand corporate interests, and major vulnerabilities can be found behind closed doors," he says. "But I would like to see a more open phase, like SynthID today, where we can access the method."

SynthID Text and Content Credentials

SynthID is a cross-modal watermarking family developed by Google DeepMind, of which SynthID Text is used for text watermarking, while SynthID also covers images, audio, and video. Like textGrain, SynthID Text embeds watermarks by introducing key-controlled bias during token sampling, and Google has open-sourced it for external research. Content Credentials, based on the C2PA standard, record provenance and editing history in file metadata and are a provenance method complementary to watermarks: watermarks are embedded in the content itself and are difficult to simply strip away, while metadata credentials provide richer provenance information but can be removed. Combining the two can cover different attack scenarios, which is why OpenAI says textGrain will complement other provenance tools.

Beyond Watermarks: The Long Game of Detecting AI Text

The deeper meaning of this debate is that as model capabilities improve, distinguishing AI text from human text is becoming increasingly difficult, and watermarking is only one of many provenance tools. textGrain's entropy-budget design is conceptually novel, but its failure on short texts and fragility under editing mean it is difficult for it to bear the compliance burden alone. For content platforms and enterprise users, combining watermarks, metadata credentials, and human review remains a more realistic strategy. For regulators, how to determine "effectiveness" in the absence of measurable benchmarks will become the core controversy in the next phase.

One-line Summary

OpenAI's text watermark textGrain is fragile on short texts and after editing, raising doubts about whether it can meet the EU AI Act.

Paragraph Summary

OpenAI has introduced a text watermarking scheme called textGrain, using an "entropy budget" to balance watermark strength and output diversity in an attempt to meet the transparency obligations of Article 50 of the EU AI Act. However, tests published by OpenAI show that detection rates decline significantly on short texts (about 80% for 200 tokens and about 95% for 400 tokens), and replacing 10% of words with synonyms drops the rate from 92% to 66%, while replacing 25% leaves only 17%. EU regulations require watermarks to be "effective and robust" but set no quantitative benchmark, and the Code of Practice exempts outputs of fewer than 200 tokens from watermarking—precisely the range where textGrain is weakest. textGrain has not yet been independently verified, and collaborating researchers have only just gained access. Experts are calling for open-sourcing to enable external auditing; if found non-compliant, OpenAI could face fines of up to 7% of annual revenue. Watermarks need to be used in combination with other provenance tools such as metadata credentials.