Naver Blog Gets 128 Google Results but Zero AI Citations — The robots.txt Truth

A Neglected Fault Line: Search Visibility and AI Citation Rates Are Completely Decoupled

Naver Blog, South Korea's largest blogging platform, appears frequently in Korean-language Google queries but has never been cited by generative AI. Daehyeon Nam, founder of Trina Inc., led a team that designed 120 queries around Korean dental care topics, covering 10 regions, 10 treatment categories, and 6 search intents, and deployed them to ChatGPT and Gemini, collecting 2,435 citation data points.

The control group consisted of the top results for the same set of queries in Google organic search. Without this control, the model could only learn to distinguish between meaningless categories like "dental pages" and "random pages." The real question they wanted to answer was: Which pages can appear in search results but are never cited by AI, and why?

Finding 1: Two Blog Platforms, Completely Inverted Fates

Channel	Google Organic Appearances	AI Citations
Naver Blog	128 (12.0%)	0
Tistory	13 (1.2%)	91

Naver Blog appeared in Google organic results roughly 10 times more often than Tistory, yet Tistory captured nearly all the AI citations.

If you combine the entire Naver suite — Blog, Cafe, Post, Knowledge-iN — only 1 out of 2,435 citations came from Naver-affiliated sources, and it came from Knowledge-iN.

This isn't a case of a channel "performing slightly worse." It's a channel that exists on one surface and completely vanishes from another.

Finding 2: The Root Cause Is Crawler Policy, Not Content Quality

The reason is written in the first few lines of blog.naver.com/robots.txt:

# BOT ACCESS FOR THE PURPOSES OF AI TRAINING AND
# RETRIEVAL-AUGMENTED GENERATION (RAG) IS STRICTLY PROHIBITED.
User-agent: GPTBot          Disallow: /
User-agent: OAI-SearchBot   Disallow: /
User-agent: PerplexityBot   Disallow: /
User-agent: Google-Extended Disallow: /

Anyone can verify this with a single command:

curl -s https://blog.naver.com/robots.txt | head -20

A Distinction Almost Everyone Conflates

GPTBot and Google-Extended are training crawlers. OAI-SearchBot and ChatGPT-User are retrieval crawlers — they read your content at the moment a question is answered, determining whether your document can be cited by AI.

Blocking the first type means you don't want your content used to train models. Blocking the second type means ensuring your content is never cited in AI answers. These are two completely different decisions, and robots.txt lets you specify them separately. A large number of websites made the second choice while believing they only made the first. Naver Blog blocked both categories and explicitly prohibited RAG in its comments. No matter how good the content quality is, it can't pass through this gate.

robots.txt is a protocol file that search engines and crawlers read first when accessing a website, located in the site's root directory (e.g., example.com/robots.txt), using User-agent to specify crawler names and Disallow to specify paths that must not be crawled. It is not legally enforced but rather an industry self-regulation convention — mainstream crawlers (Google, OpenAI, Perplexity, etc.) comply with it, though malicious crawlers may ignore it.

The key point is that different AI companies use multiple different User-agents for different purposes. Take OpenAI as an example: GPTBot trains foundation models, OAI-SearchBot indexes pages for ChatGPT search, and ChatGPT-User fetches in real time when a user initiates a query. Google's Googlebot serves regular search, while Google-Extended controls whether content is used for Gemini training and AI summaries. Therefore, a single robots.txt can precisely control "allow search indexing but prohibit training" — this is the technical foundation behind the original article's emphasis on "two independent decisions."

Finding 3: Domains Blocking AI Crawlers Convert Search to Citations at Only One-Fifth the Rate

The team extracted the top 60 domains by citation count and the top 60 domains by search appearances (90 after deduplication), scraped each robots.txt, and parsed the allow/deny status for six AI crawlers.

Domain Group	Count	Search Queries	Citation Queries	Conversion Rate
Blocks AI crawlers	8	171	29	0.17×
Open	82	559	455	0.81×

After appearing in search results, if you block AI crawlers, your probability of being cited is roughly one-fifth that of open sites.

Finding 4: An Accidental Natural Experiment

Because different domains had different blocking policies, the network as a whole ran an experiment on the researchers' behalf. The test logic: If which crawler is blocked predicts which engine doesn't cite the domain, that's mechanism, not correlation.

Blocking Pattern	Search	ChatGPT	Gemini
Blocks training crawlers only (OAI-SearchBot, ChatGPT-User remain open)	4	7	0
All OpenAI crawlers blocked, Google-Extended remains open	22	0	42
Only GPTBot blocked	5	0	10
All six crawlers blocked (blog.naver.com)	69	0	0

Across all 8 domains with blocking policies, the blocked crawlers matched the engines that actually didn't cite the domain 8 out of 8 times.

Limitations That Must Be Explicitly Stated

The sample size is only 8 domains, and no significance testing was performed. The reported fact is that all 8 matched, which is difficult to explain by coincidence, but this does not equal measuring effect size. Finding 4 proves the mechanism exists, not quantifies it. This is also an observational study — no intervention experiment was conducted on the same page under "blocked vs. open" conditions. The query set covers only one vertical (dental) in one country, and specific percentages should not be generalized to other markets — but the mechanism itself is not Korea-specific; robots.txt directives are read the same way globally.

The core difference between an observational study and an intervention experiment is that the former only observes naturally occurring data, while the latter actively manipulates variables. Observational studies can identify correlations but struggle to rule out confounding factors, and thus cannot directly establish causality. The "8 out of 8 match" in the original article is observational evidence — although the match rate is extremely high and difficult to explain by coincidence, the study did not conduct a controlled "blocked vs. open" experiment on the same page, so it cannot quantify "how much blocking a crawler reduces citation rates."

Significance testing is used to determine whether an observed difference could be due to random fluctuation. With a sample size of only 8 domains, even if all 8 match, statistical power is extremely low and reliable p-values cannot be calculated. This does not weaken the conclusion that "the mechanism exists" — that crawlers comply with robots.txt directives is a known technical fact — but it means specific percentages (such as 0.17× and 0.81×) cannot be directly extrapolated to other markets or verticals. The researchers' explicit distinction between "proving the mechanism exists" and "measuring effect size" is key to avoiding overinterpretation.

They Personally Overruled Their Own First Model

This was the team's second study. In the first, they collected approximately 95,000 AI citation content pieces, decomposed "what causes citations" into 189 factors, and built independent models for each engine.

During pre-launch testing, most results were overturned:

• After re-measuring with query-level out-of-fold, R² = −0.31, worse than simply predicting the mean, with no predictive power at all.
• They fell into the p > n trap — more variables than data points, a situation where a model can fit anything.
• Evaluation used in-sample data, equivalent to grading yourself on problems you've already done.
• They used globally random negative samples, causing the model to only distinguish between "dental pages" and "arbitrary pages."

These four failures were each named and redesigned to avoid: query-level out-of-fold, per-query controls, permutation tests, and seed reproducibility all came from that retrospective.

Out-of-fold evaluation is the core operation of cross-validation: data is split into K folds, each time training on K-1 folds and validating on the held-out fold, cycling K times and aggregating validation results. It ensures the model is evaluated on "unseen data," avoiding the optimistic bias of self-evaluation on the training set. The "query-level out-of-fold" mentioned in the original article further requires that all samples generated by the same query must be assigned entirely to the same fold, otherwise the model may memorize query features rather than truly generalize.

The p > n trap refers to the number of variables (p) exceeding the number of samples (n). In this situation, the model has infinitely many solutions that can perfectly fit the training data but will almost certainly collapse on new data — this is a classic pitfall in statistical learning and one of the root causes of the original article's first model having a negative R². Globally random negative samples refers to randomly drawing pages from the entire site as "irrelevant" controls, causing the model to only learn to distinguish "dental pages vs. arbitrary pages" — an overly simple classification that cannot answer the real question: among equally relevant pages, why are some cited and others not?

Another Trap: Most "More Is Better" Metrics Are Channel Proxies

List item counts, image counts, body length, and external link counts all show positive correlations in the full sample — more quantity, more citations. But when comparing only clinic homepages against other clinic homepages, the direction flips (list items 36 vs. 61, images 11 vs. 21).

Long lists and numerous links are characteristics of portal sites. What gets cited is the channel, not the feature. If you read correlations without holding the channel constant, you'll get a content checklist that teaches you to "act like a portal site." The only features that maintain stable direction within one's own site are: image alt ratio, heading tag count, location information, and practitioner information.

Actionable Checklist

1. Scrape the robots.txt of every channel you publish on — not just your own domain, but platforms too.
2. Separate the two decisions. To refuse training but remain citable: block GPTBot and Google-Extended, keep OAI-SearchBot and ChatGPT-User open.
3. Don't infer AI visibility from search visibility. Naver vs. Tistory is a 10-to-1 inversion between two surfaces.
4. Measure citations directly. Ranking doesn't equal citation; some pages occupy the top of organic search for extended periods yet have never been cited once in an AI answer.

What This Research Changed

Trina Inc. now publishes content only on clients' own domains and does not accumulate core content on platforms where crawler policies cannot be controlled. This isn't a preference — it's a direct conclusion from the measurements above.

Background Supplement

Channel proxy refers to when a feature correlates with an outcome, but what actually matters is the channel or platform attribute behind the feature, not the feature itself. For example, pages with more list items and external links are cited more often by AI, seemingly suggesting "long lists drive citations," when in fact such pages often come from portals or aggregation platforms, and those platforms themselves are more open and easier to crawl.

Simpson's Paradox is the classic statistical manifestation of this phenomenon: in aggregate data, variable A is positively correlated with outcome B, but after grouping by channel, the within-group correlation may disappear or even reverse. The original article's "when comparing only clinic homepages against other clinic homepages, the direction flips" is a textbook example of Simpson's Paradox. To avoid being misled by channel proxies, researchers must hold the channel constant before comparing features, or use hierarchical models that include channel as a random effect. This also explains why only features like image alt ratio, heading tag count, location information, and practitioner information maintain stable direction within one's own site — they reflect the page's inherent quality, not platform attributes.