llms.txt Best Practices: Give AI Agents a Map They Won't Get Lost In

AI crawlers and agents are becoming the new visitors to websites, but most sites haven't prepared a decent "guide map" for them. llms.txt is a convention format created for exactly this purpose—it doesn't decide who gets through the door, but it determines whether the reader who enters can find answers within three steps.

What llms.txt Is and Isn't

Let's clear up a common misconception first: llms.txt is not a submission mechanism. There's no registry, no queue, no ping endpoint. This convention was proposed by AnswerDotAI in September 2024 and still lives in a deliberately small GitHub repository. The core structure has only three things—an H1 site name, a blockquote summary, and link lists under several H2 groupings.

It solves a retrieval problem, not an indexing problem. By the time anything comes to read your file, a decision about "whether to crawl your site" has long been made: it might be a browsing assistant, a coding agent, or a RAG pipeline that has whitelisted your domain. This reader arrives with a task and a token budget, and /llms.txt is the map you hand them.

Google's official guidance on AI search features is clear: pages don't need any special files or markup to appear in AI-driven results. Crawler permissions are a separate matter, governed by robots.txt, and OpenAI has also documented in detail how its crawlers work. You can't "opt in" to AI visibility with a text file, just as a sitemap alone won't get you into search indexes.

A fitting analogy: llms.txt is the media kit sitting on the reception desk. It tells visitors where the archive room is and who to ask, but it doesn't decide who gets through that door.

robots.txt is a plain text file in the website root directory that uses the Robots Exclusion Protocol to tell crawlers which paths can be crawled through user-agent, allow, disallow, and other directives. It governs "whether you can enter," which is a permissions layer; llms.txt governs "how to find answers after entering," which is a navigation layer. The two are often confused, but their purposes are completely different.

sitemap is a URL list for search engines to help crawlers discover site pages, usually provided in XML format. It solves the discovery problem of "what pages exist," but doesn't care about page semantics and priority, nor does it describe the purpose of each link—this is exactly the gap that llms.txt fills with short descriptions and task groupings.

RAG pipeline refers to the retrieval-augmented generation process: first retrieving relevant snippets from a knowledge base or web pages, then handing them to a large model to generate answers. Whitelisting a domain means that pipeline's retrieval phase will prioritize crawling and trusting your content, so a clear llms.txt can directly influence which pages it selects.

What a File That Actually Gets Read Looks Like

Most files we've examined are just link dumps—sixty URLs, no descriptions, no ordering. Readers scan for a domain match and leave. Files that actually get used share six characteristics:

• A single H1 that matches the name people search for
• A single blockquote that explains what this thing is and who it serves
• Two or three lines of plain text context—base URL, auth model, where each section lives
• H2s named by task rather than by department, so readers searching for "how to cancel subscription" can recognize the heading
• A description of ten words or fewer next to each link
• An Optional section at the bottom for secondary material

Here's an example you can reference directly:

# Northwind Analytics

> Northwind is a billing API for subscription software teams. Docs, SDKs, and pricing live here.

Northwind exposes a REST API, four official SDKs, and a hosted webhook relay.
Base URL: https://api.northwind.example/v2

## Core documentation
- [Quickstart](https://docs.northwind.example/quickstart): auth, first charge, five minutes
- [API reference](https://docs.northwind.example/api): every endpoint, request and response
- [Webhooks](https://docs.northwind.example/webhooks): events, retries, signature checks

## SDKs
- [Python](https://docs.northwind.example/sdk/python)
- [TypeScript](https://docs.northwind.example/sdk/typescript)

## Pricing and limits
- [Plans](https://northwind.example/pricing): tiers, overage, annual discounts
- [Rate limits](https://docs.northwind.example/limits): per-key quotas and backoff

## Optional
- [Changelog](https://northwind.example/changelog)
- [Status](https://status.northwind.example)

Two details do most of the work. Absolute URLs, because readers may parse your file in a context-free setting with no reliable base to resolve relative paths. Short descriptions, because they're the only signal telling readers "which of these six links under Core documentation answers the question I'm holding."

There's an excellent precedent for this genre—Hugging Face's model cards: a small piece of structured text for humans to scan and machines to parse, placed alongside something much larger. Your llms.txt is your site's model card.

Hand-maintained files rot. Endpoints get deprecated, pricing pages move, products get renamed, and the map still points to old addresses. A more reliable approach is generating it at build time from your sitemap and frontmatter, wiring it into the same pipeline that builds your docs—so the map never drifts more than one deployment cycle behind.

Agent Traffic Is the Real Payoff Scenario

Chatbots are the least demanding readers in your audience: grab a page, synthesize, done. Agents are completely different—they hold a goal across multiple steps, choose tools mid-task, and MCP gives them a standard way to invoke tools. An agent that just received the instruction "help this customer upgrade their plan" needs to find your endpoints, auth scheme, and error codes in sequence, not crawl twelve pages to judge which matters.

This is exactly the tradeoff that determines file shape. Cover everything, and you burn context you can't get back. llms-full.txt is for readers who genuinely want the entire corpus in one shot, which for a documentation site might be a forty-thousand-word payload, often truncated before reaching the useful section. Default strategy: use compact llms.txt as the map, use llms-full.txt to serve readers who specifically ask for everything.

What does the payoff look like concretely? A customer service agent handling a billing question reads your file, sees [API reference](...), grabs that one page, finds the endpoint, stops. Three requests instead of a full crawl—that's the entire benefit.

MCP stands for Model Context Protocol, an open protocol proposed by Anthropic in late 2024 for letting large models invoke external tools and data sources in a standardized way. It defines the communication format between clients and servers, so agents don't need to write custom adapter code for every service.

Token budget refers to the total token limit available in a single model call or task, including both input and output. An agent consumes context for every page it reads during multi-step execution, and once it exceeds the window, earlier content may be truncated or discarded. So llms.txt's value lies in letting agents hit the target page with minimal crawling, spending limited tokens on actual answers rather than scanning irrelevant links. This also explains why short descriptions matter more than simply listing URLs—they're a direct means of reducing retrieval cost.

Three Things It Can't Do

It can't control crawlers. That's robots.txt's job, and Google's documentation on getting started with that protocol remains the right starting point for understanding what user-agent directives can and can't do. A file that says "please cite us" while your robots.txt simultaneously blocks crawlers is a note left in a locked room.

It can't establish identity. Entity identity comes from structured data—Organization, Product, and article types on schema.org, which give machines a canonical name, logo, and sameAs relationships. Structured data for agents is what lets models resolve your brand as a real entity rather than a string appearing across eleven unrelated pages.

It can't fix thin content. If your docs are locked behind a login, or the best answers are buried in a 2019 PDF, even a tidy link map points to nothing. The webmaster guidance published by the Bing search team explains how Copilot surfaces content, and the answer rhymes with Google's: crawlable, well-structured, plainly described pages. No file can circumvent this.

What it can do is small, real, and cheap: save a few requests from the retrieval loop for readers who were going to arrive anyway.

schema.org is a structured data vocabulary co-founded by Google, Microsoft, Yahoo, and others, defining types like Organization, Product, Article, and FAQPage along with their properties. Webmasters can annotate page content as these types using JSON-LD or microdata, and search engines and AI systems can then extract entity information from these rather than just guessing from text.

sameAs is a schema.org property used to declare an entity's official external links, such as Wikipedia entries, social media profiles, or other authoritative records. It helps machines confirm that multiple URLs point to the same real-world entity, upgrading a brand name from "a string appearing on several pages" to a formal identity that can be disambiguated. Without structured data and sameAs, even the tidiest links in your llms.txt just point to a pile of pages unconfirmed by machines.

How to Measure and Maintain

Server logs are the only conclusive evidence. Count GET /llms.txt hits, look at the user agent distribution, observe quarterly trends. Everything else is inference—there's currently no public data on how many sites publish this file or which engines consume it, so anyone citing "improvement numbers" is citing something they made up.

Of the five-step maintenance loop we run, only the first step is about the file itself:

1. Log the crawls. If nothing has requested this file in ninety days, it's not your bottleneck.
2. Track citations. Answer engine citation tracking tells you which URLs are actually being cited, which often differs from your own list of what you think matters.
3. Spot decay. Pages losing citations over time are usually pages that lost accuracy—a price change, a deprecated endpoint, a version upgrade. Content decay detection and refresh catches this before competitor pages replace you as the source.
4. Patch the gaps. Run a GEO / AI visibility audit on your domain to see which questions you're absent from, then check whether the page that should answer that question exists, is crawlable, and is described in your file.
5. Regenerate on every deploy. This file is an artifact, not a document.

The order matters. Teams that start with step five spend a quarter drawing a beautiful map for a site that "can't answer anything."

GEO stands for Generative Engine Optimization—the practice of optimizing content visibility for AI-driven search and Q&A systems. It differs from traditional SEO in both goals and methods: SEO focuses on rankings and clicks, while GEO focuses on whether you're cited, summarized, and treated as an answer source by AI.

A GEO / AI visibility audit typically checks: whether your domain is cited by AI systems for specific questions, which competitors replaced you, which pages the cited links point to, and whether the page answering your core question exists and is crawlable. This kind of audit outputs exactly the inputs needed for steps three and four of the maintenance loop—where's the decay, where are the gaps. It's worth emphasizing that llms.txt is an optional component in a GEO workflow, not a shortcut; the audit comes before the file, and the order can't be reversed.

FAQ

Can llms.txt improve my ranking in AI summaries or ChatGPT?

No. Google's AI features guidance explicitly states pages don't need special files or markup to appear, and the file has zero effect on ranking. Treat it purely as a retrieval aid for readers who arrive anyway.

Do OpenAI, Anthropic, and Google actually read it?

Neither OpenAI's crawler documentation nor Google's AI features guidance lists this file as an input. However, Anthropic published one for its own developer documentation, which shows the format is comprehensible to the people building these systems. Comprehensible doesn't equal consumed—the honest position is: support varies by engine and isn't publicly disclosed.

Should I publish llms.txt, llms-full.txt, or both?

Publishing both is a defensible default. The short file serves agents needing navigation, the full file serves pipelines wanting the corpus in a single fetch. If you can only maintain one, maintain the short one—an outdated map survives, outdated content doesn't.

My docs are on a subdomain—where should the file go?

At the root of the host serving the content, so docs.example.com/llms.txt covers that documentation site. If you have multiple subdomains, a root domain file can serve as an index to subdomain files. Absolute URLs in the body resolve most ambiguity anyway.

How often should I update it?

Whenever the underlying navigation changes—new sections, retired products, moved pricing pages. In practice this means auto-generating on every deploy and manually reviewing diffs when structure changes, rather than when paragraphs change.

Next Steps

Before writing the first line of the file, figure out how engines currently rate you. A GEO audit will show which pages get cited, which questions you're absent from, and whether a map helps people navigate or just describes an empty building.

One-line Summary

llms.txt is a site guide map for AI crawlers and agents—it doesn't decide crawl permissions, it just helps readers find answers within three steps.

Paragraph Summary

llms.txt is a convention proposed by AnswerDotAI in September 2024, whose core structure is just an H1 site name, a blockquote summary, and link groups under task-named H2s. It solves a retrieval problem rather than an indexing problem, has no effect on AI search rankings, and crawl permissions fall under robots.txt's jurisdiction. A file that actually gets read needs absolute URLs, short descriptions, task-oriented H2 groupings, and an Optional section at the bottom, and should be auto-generated on deploy rather than hand-maintained. The real payoff scenario is multi-step AI agents, which can compress a full crawl into two or three requests. The article also emphasizes it can't control crawlers, establish entity identity, or fix thin content, and closes with a five-step maintenance loop based on server logs and citation tracking.