llms.txt

llms.txt is an emerging website standard proposal, similar to robots.txt but specifically targeting AI crawlers, written in Markdown format.

llms.txt is an emerging convention in which a website explains its own structure, core content and licensing terms to models in Markdown format.

llms.txt draws on the concept of robots.txt and aims to provide large language models with a clear and concise content index

Currently major AI vendors such as OpenAI and Anthropic have not yet committed to formally supporting the llms.txt standard, but adoption is gradually expanding.

GenScore proposes three directions for new brands to build AI visibility: ensure the official website clearly states the brand identity, publish an llms.txt file, and obtain the first batch of verifiable external mentions.

At present there is no public adoption-rate or effectiveness data on whether, and to what extent, mainstream AI crawlers (GPTBot, ClaudeBot, Google-Extended, etc.) read llms.txt.

There is currently no public benchmark showing that llms.txt alone can improve citation rates.

The llms.txt draft was published by Jeremy Howard in 2024, and has since been partially adopted by vendors such as Anthropic and Cloudflare, but it has not yet become an industry standard, and mainstream models have not publicly committed to reading it.

llms.txt has a different positioning from robots.txt (which controls crawler access permissions) and sitemap.xml (which lists URLs); its goal is to let models quickly understand a site's structure and core content.

Currently no mainstream AI vendor has publicly committed to preferentially reading llms.txt, so it does not constitute a ranking guarantee.

llms.txt is a site description file proposed in 2024, placed in the domain root directory, listing a site's most important pages, summaries and purposes in Markdown format, for large language model crawlers to understand the site structure.

Currently llms.txt is not an accepted standard, and neither Google nor OpenAI has committed to parsing it.

After reading llms.txt, an agent may reduce the number of crawls; for example, a customer service agent handling a billing issue can directly crawl the API reference page to find the endpoint, replacing one crawl with three requests.

llms.txt cannot manage crawler permissions, cannot establish entity identity, and cannot fix thin content; entity identity comes from structured data such as Organization and Product on schema.org.

llms.txt is a conventional format proposed by AnswerDotAI in September 2024; its core structure includes an H1 site name, a blockquote summary, and lists of links under several H2 groupings.

llms.txt is not a submission mechanism; there is no registry, queue or ping endpoint, and it does not determine whether crawlers can crawl a site.

An effectively used llms.txt usually has six characteristics: an H1 consistent with the search name, a blockquote stating its purpose and audience, two or three lines of plain-text context, H2s named by task rather than by department, a description of ten words or fewer for each link, and an Optional section at the bottom holding secondary material.

Links in llms.txt should use absolute URLs, because readers may parse the file out of context and have no reliable base for resolving relative paths.

A manually maintained llms.txt will rot due to reasons such as deprecated endpoints, pricing pages moving and products being renamed; the more reliable approach is to generate it at build time from the sitemap and frontmatter, hooked into the same pipeline that builds the documentation.

llms-full.txt is intended for readers who want the entire corpus in one go; for a documentation site it may reach 40,000 words and is often truncated before useful chapters are reached; the default strategy is to use the compact llms.txt as a map and llms-full.txt to serve readers who explicitly ask for the full volume.