llms.txt Log Analysis: Who's Actually Reading It?

Introduction: The Practical Question of llms.txt

llms.txt and llms-full.txt have been deployed on many sites as site declaration files aimed at large language models. But a more practical question follows: is any client actually requesting them? Tech blogger richardlemon offers a verification method that doesn't rely on any third-party dashboard by analyzing Nginx access logs, and uses it to discuss what this really means for GEO (Generative Engine Optimization).

Log Forensics: Starting from Nginx access.log

In a typical Nginx + VPS setup, access logs are usually located at /var/log/nginx/access.log, alongside rotated files like access.log.1, access.log.2.gz, etc. To verify llms.txt access, you only need to focus on two fields: the request path and the User-Agent. The former tells you which file was requested; the latter tells you who requested it.

The original article emphasizes that this is a raw forensic approach—no dashboard, no intuition—just reading the logs directly and letting the facts speak.

The Nginx combined log format is Nginx's default access log format. Each line consists of several space-separated parts, some of which contain quoted fields, typically shaped as: $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent". Understanding its field layout is a prerequisite for interpreting logs—because the request, referer, and user_agent fields are each wrapped in double quotes and may contain internal spaces, splitting directly on spaces will misalign fields. The correct approach is to use double quotes as the delimiter to locate fields. Under this format, the 1st segment is the client IP, the 2nd is the request line (including method, path, protocol), the 3rd is the status code and other info, and the 6th is the User-Agent. This also explains why the awk command below uses -F '"' instead of the default space delimiter. If your site uses a custom log_format, field positions will differ, so confirm the format definition in nginx.conf before analysis.

Filtering and Aggregation: Extracting Requests with grep and awk

The first step is to extract the lines involving llms.txt and llms-full.txt from the logs:

cd /var/log/nginx
grep -hE "llms\.txt|llms-full\.txt" access.log* > llms-requests.log

Here -h hides filenames, -E enables extended regex, and access.log* covers the current and uncompressed rotated logs; if there are .gz files, use zgrep instead.

The second step is to extract the User-Agent from each line and count frequencies. In the standard combined log format, the User-Agent is the last quoted string (field 6):

awk -F '"' '{print $6}' llms-requests.log \
  | sort | uniq -c | sort -nr > llms-user-agents.txt

This step outputs a raw list of "who requested and how many times."

Bucket Classification: Identifying Major Crawlers Like GPTBot and ClaudeBot

Raw UA strings are messy and need to be bucketed by known crawler families. Common targets listed in the original article include:

• GPTBot (OpenAI)
• ClaudeBot (Anthropic)
• PerplexityBot
• Googlebot (as a control, not a comparable LLM crawler)
• Other (browsers, random crawlers, SEO tools, etc.)

The classification itself can be done with a single awk script: extract the UA, then case-insensitively match keywords in sequence, defaulting to Other, and finally sort | uniq -c | sort -nr to get the request volume ranking for each family. This script can be run periodically to observe how the request composition changes over time.

Common LLM crawler User-Agent identifiers each have fixed keywords, and matching usually just takes the product token in the UA string: GPTBot's UA looks like Mozilla/5.0 ... GPTBot/1.x, and some OpenAI crawls also use OAI-SearchBot or ChatGPT-User; ClaudeBot corresponds to Anthropic's crawling, with historical identifiers like Claude-Web and anthropic-ai; PerplexityBot is Perplexity's indexing crawler, while user-triggered real-time fetches may show Perplexity-User; Googlebot is used for Google search indexing and is unrelated to LLM training, serving only as a control. There are also Bytespider (ByteDance), CCBot (Common Crawl, the corpus source for many models), Amazonbot, Applebot, etc. Bucket regexes should be case-insensitive and cover these variants, otherwise real traffic will be misclassified as Other, underestimating actual interest.

What Logs Can and Cannot Prove

This is the core warning of the original article. Seeing GPTBot request llms.txt makes it easy to conclude "my content has been used for training," but the logs don't support that leap. Logs can only prove: at a certain time, from a certain IP, a client claiming a certain UA requested a certain URL and received a certain HTTP status code.

Specifically, there are several layers of disconnect:

• Identity is self-reported. UA can be spoofed, and without reverse DNS and IP range verification, you cannot confirm the request truly came from OpenAI or Anthropic.
• Crawling does not equal storing. A crawler may simply read the policy and decide not to crawl or retain content; llms.txt may serve purely as a routing or filtering hint.
• Storing does not equal training. Caching doesn't mean entering the training set; vendors typically separate crawling, indexing, and training pipelines.
• Training does not equal citation. Even if content enters the training set, there's no guarantee it will be cited or displayed in generated output.

So "GPTBot in the logs" is an observable fact, while "GPTBot is training on my content" and "GPTBot will cite my content" are vendor-side policy and implementation details, with decreasing levels of certainty.

What It Really Means for GEO Practice

There's much discussion around llms.txt and LLM-specific robots directives, but access logs have limited power to demonstrate GEO policy enforcement.

What you can safely say from logs: a certain crawler family seems aware of llms.txt and does crawl it; a certain crawler accesses from certain IP ranges at a certain frequency. What you cannot say: a certain vendor fully complies with your GEO directives; your content has been excluded from all future training in all regions; changes to llms.txt will take effect globally on a specific date.

At best, what you can do is correlation observation: publish or modify llms.txt, see a bot crawl it, then assume the other party's policy of mentioning that file has taken effect. The actual enforcement logic and timeline are in someone else's API and release notes.

Why llms.txt Is Still Worth Maintaining

Despite the limitations above, the original author still believes llms.txt and log analysis are valuable:

• They provide a clear, machine-readable place to declare preferences.
• Access logs can at least verify that some crawlers are paying attention.
• When policies or vendors change, there's a concrete file to update rather than guessing which meta tag or header is working this month.

But don't treat it as magic. If you care about how your content is used in LLMs, logs are just the most basic low-tech feedback loop—they tell you who knocked on the door, but not what happened after the data left your server.

Background Notes

The relationship between GEO and llms.txt/robots declarations needs clarification. GEO (Generative Engine Optimization) refers to optimizing content visibility for generative retrieval systems like ChatGPT, Perplexity, and AI Overviews. Its goal is to make content more easily crawled, understood, and cited in answers by AI engines—different from traditional SEO's focus on blue-link rankings. llms.txt is a convention proposed in 2024 that lists key pages and summaries for LLMs in Markdown form at the site root (llms-full.txt provides full content), allowing models or agents to efficiently obtain site structure; but it is not a formal standard from W3C or any standards body, and major vendors have not publicly committed to following it. Traditional robots.txt with User-agent sections and Disallow remains the widely recognized crawling control mechanism, while LLM-specific extensions (such as prohibiting training) likewise lack enforceability. Therefore, the actual effectiveness of GEO declaration files ultimately depends on whether each crawler actively reads and follows them, which is precisely the root reason why access logs can only provide correlational evidence and cannot prove compliant enforcement.