What Is llms.txt and How to Add It to Your Site

What exactly is llms.txt?

llms.txt is a proposed standard, introduced in September 2024, for a single Markdown file that lives at the root of your domain — for example, yoursite.com/llms.txt. Its job is to hand language models a clean, curated summary of what your site offers and which pages carry the most value, without forcing them to wade through navigation menus, ads, cookie banners and JavaScript.

The reasoning is practical. AI models work within limited context windows, and a full HTML page is noisy and expensive to parse. A concise Markdown file solves both problems at once: it is cheap to tokenize and easy to read. People often describe it as "robots.txt for AI," but the analogy is loose. Robots.txt tells crawlers what they may access; llms.txt instead tells models what your content actually means and where the good stuff lives.

One thing to keep clear from the start: llms.txt does not block, grant or control access to anything. It is a helpful signpost, not a gatekeeper. That distinction matters when you decide how much effort to put into it and what to reasonably expect in return.

Why do AI answer engines and agents look for it?

The honest answer is nuanced. The major AI search crawlers — the ones behind ChatGPT, Claude, Perplexity and Google's AI features — mostly crawl your HTML directly today and rarely request llms.txt. No major provider has formally committed to reading it, and Google has said publicly that it does not plan to support it. So treating the file as a guaranteed ticket to AI citations would be a mistake.

Where llms.txt genuinely earns its keep is the agentic web. AI coding assistants such as Cursor, GitHub Copilot, Claude Code, Windsurf and Cline routinely fetch llms.txt when pointed at a documentation site, using it to pull the right pages with far less token waste. That makes it a strong "business-to-agent" signal, especially for SaaS, developer tools and API products.

There is a smaller secondary upside, too. A well-written llms.txt is itself a clean, keyword-relevant page that can get indexed and occasionally rank. It won't replace solid on-page SEO, but as a low-cost addition with real upside for agent tooling, it's a sensible file to ship.

What should go inside an llms.txt file?

The format is deliberately simple and follows a fixed order. You begin with an H1 containing your project or site name — this is the only strictly required element. Directly beneath it, add a blockquote (a line starting with >) that summarizes what your site does in one or two sentences, packing in the key context a model needs to interpret everything that follows.

After the summary you may add a few plain paragraphs or lists for extra context, but no headings yet. Then come your link lists, each grouped under an H2 heading such as ## Documentation or ## Guides. Every link follows one exact pattern: a dash, the title in square brackets, the URL in parentheses, a colon, then a short description. Agents parse these lines literally, so deviating from the format quietly breaks things.

Two conventions are worth knowing. Descriptions should carry concrete facts, because agents often use them as standalone context without ever fetching the page. And an optional ## Optional section signals links that can be skipped when context is tight — a polite way to mark your lower-priority pages.

What does a real llms.txt example look like?

Picture a simple SaaS site. The file opens with # Acme Analytics, followed by a blockquote such as > Acme Analytics is a privacy-first product analytics tool for small teams, with self-serve dashboards and a REST API. That single line tells a model who you are, what you sell and who it's for.

Below that you'd add H2 groups. Under ## Documentation you might list - [Getting Started](https://acme.com/start): Install the SDK and send your first event in five minutes, then an API reference line, then a billing guide. Under ## Guides you'd link tutorials and best-practice articles. Anything low-signal — legacy docs, changelogs — goes under ## Optional so agents can drop it under pressure.

The structure below shows the required order at a glance. Keep the whole file focused: roughly 15 to 25 strong links with specific descriptions tends to outperform an exhaustive dump that overwhelms the context window.

How do you add llms.txt to your site?

Start by listing your genuinely important pages — core docs, key guides, pricing, product overviews — and writing a specific one-line description for each. Then assemble them into the Markdown structure above: H1 title, blockquote summary, H2 groups with correctly formatted link lines, and an Optional section for the rest. Keep descriptions concrete and jargon-free.

Save the file as llms.txt and upload it to your web root so it resolves at yoursite.com/llms.txt, served as plain text or Markdown. Many documentation platforms — including Mintlify, GitBook, Fern and others — can generate it automatically, and WordPress plugins now do the same, so check whether your stack already offers it before hand-rolling one.

If your site is documentation-heavy, consider shipping a companion llms-full.txt that concatenates every linked page into one file for agents that want everything in a single request. Finally, test it: feed the file to a couple of models and ask them questions about your content to confirm they can answer accurately.

Is llms.txt worth the effort right now?

Set expectations honestly and it's an easy yes for the right sites. Adoption sits at roughly one in ten domains, and the crawlers that drive AI search citations largely skip the file today. If your goal is immediate AI answer-engine visibility, your effort is better spent on clear, well-structured HTML content and strong internal linking that crawlers actually read.

But the file is cheap to create and carries real upside where it counts. For SaaS, dev tools and API products whose audience uses AI coding assistants, llms.txt meaningfully improves how those agents navigate your docs. And as the agentic web matures, having a clean machine-readable surface already published puts you ahead rather than scrambling later.

The pragmatic move is to treat llms.txt as one small layer in a broader organic strategy — not a silver bullet. Publishing genuinely useful, well-organized content remains the foundation for both Google rankings and AI citations. That's exactly the kind of work an organic-marketing autopilot like Artiql is built to keep running for you.

Frequently asked questions