llms.txt Explained: Do AI Crawlers Actually Use It?

What is llms.txt, and where did it come from?

llms.txt is a plain-text Markdown file you place at the root of your domain, at yourdomain.com/llms.txt. It was proposed in late 2024 by Jeremy Howard, co-founder of Answer.AI, as an open convention to help large language models read and use a website more easily. Think of it as a curated index: instead of leaving an AI to crawl every menu, script and layout, you hand it a tidy summary of what matters most and where to find it.

The motivation is genuinely technical. AI models work within limited context windows, and pulling clean meaning out of cluttered HTML — navigation, ads, cookie banners, JavaScript — is hard and wasteful. By offering a concise, machine-friendly map in Markdown, llms.txt is meant to reduce that friction and point models toward your canonical, high-value content rather than an outdated blog post buried three clicks deep.

It's often compared to robots.txt or an XML sitemap, and the analogy is roughly fair in spirit. But the intent is different. robots.txt tells crawlers what they may access; llms.txt tries to tell models what's worth reading and how your site is organized. It's a suggestion about meaning, not a rule about permission.

What does an llms.txt file actually look like?

The format is deliberately simple, which is part of its appeal. The official specification asks for an H1 with your project or site name — the only strictly required element — followed by a blockquote summary that captures the essentials in a sentence or two. After that you can add a few paragraphs or lists for context, then organize links into H2 sections, each link written as a standard Markdown link with an optional note describing the page.

Order matters more than you might expect. Models read top-down and tend to weight earlier sections more heavily, so lead with your most authoritative, canonical pages. There's also a special convention: a section titled "Optional" signals content that can be safely skipped when a shorter context is needed. Where possible, the spec suggests linking to clean Markdown versions of pages, since they parse more reliably than full HTML.

Keep it lean — most useful files stay well under 500 words. A companion file, llms-full.txt, can hold the complete documentation library, while llms.txt itself stays a slim navigational index. After publishing, confirm it loads as plain Markdown at your root with a 200 status, and check that every link resolves cleanly.

Do ChatGPT, Claude and Perplexity actually use llms.txt?

This is where expectations meet reality. Despite the buzz in marketing circles, independent server-log studies through 2025 found that the major AI crawlers essentially don't request the file. In one analysis spanning mid-August to late October 2025, the llms.txt page received zero visits from GPTBot, PerplexityBot, ClaudeBot or Google's AI crawler. A separate audit of CDN logs across 1,000 domains saw no LLM-specific bots fetching it at all, while Google's ordinary desktop crawler accounted for the overwhelming majority of hits.

A 90-day study reinforced the pattern: even with a correctly implemented file, dedicated AI bots used the llms.txt entry point in roughly 0.1% of their visits. Adoption on the publishing side is tiny too — only around a thousand domains had published one by mid-2025. No major provider has confirmed it reads these files when generating answers, though a handful, including Anthropic, have published their own, which hints at openness rather than active use.

The honest summary: implementing llms.txt is harmless and low-effort, but there is no solid evidence the big answer engines treat it as a meaningful signal today.

Is llms.txt a Google ranking or GEO lever?

No — and this is the most common misunderstanding. Many marketers reframed llms.txt as a fresh GEO trick, a way to muscle into AI Overviews or get cited more often by ChatGPT and Perplexity. That was never the proposal's purpose. It was designed to help models read your site, not to manipulate rankings or citations. Treating it as a visibility hack mostly leads to disappointment.

Google has been unusually blunt here. In mid-2025, Gary Illyes stated that ranking in AI Overviews just needs normal SEO and that Google does not support llms.txt and has no plans to. John Mueller went further, comparing it to the long-ignored keywords meta tag — a signal search engines abandoned precisely because the site owner controls it, which makes it trivial to game and therefore untrustworthy.

That comparison is the crux. Any signal a publisher fully controls and can stuff in their favor tends to get discounted. Real authority in AI answers is earned through content quality, third-party mentions and verifiable expertise — not through a self-declared file.

What structured data actually influences AI citations today?

If llms.txt isn't moving the needle, what is? The strongest evidence points to structured data — specifically JSON-LD schema markup that describes your content in machine-readable terms. Unlike a self-declared index, schema annotates content the user can actually see, which makes it harder to fake and more trustworthy to consumers. Platforms have confirmed they use it: a Microsoft engineering lead stated publicly in early 2025 that schema markup helps its models understand content, and independent tests have shown ChatGPT, Claude, Perplexity and Gemini all parse schema when they fetch a page.

The schema types with the clearest citation evidence are FAQPage, Article, Product/Offer, Organization and Person. Just as important is connecting them into a coherent entity graph using @id and sameAs, so engines understand that the author, the company and the product are related, identifiable things. This entity clarity — not keyword density — is what answer engines increasingly reward.

Schema is context, not a magic switch. It helps AI parse and attribute your content accurately, but it can't manufacture credibility. Pair it with concise answer capsules of 40–60 words under question-style headings, visible authorship and dates, and anchor IDs so engines can cite specific sections.

Should you publish an llms.txt file anyway?

For most sites, yes — with realistic expectations. The file is cheap to create, won't harm your SEO, and gives you a tidy place to declare your most important content. There's also a real, if narrow, use case today: if your audience includes developers using tools like Cursor, Windsurf or Claude Code with your API documentation, a well-structured llms.txt acts as a genuine entry point. That's close to Howard's original intent, and it works.

There's a modest future-proofing argument too. Because the same web crawlers that index your pages can surface them in AI-driven search, a clean, indexable llms.txt occasionally ranks for organic terms on its own. And if providers do adopt the convention more seriously, you'll already be ready. The risk isn't the file itself — it's mistaking it for the work that actually drives visibility.

So treat it as a low-priority hygiene item, not a strategy. Publish it, keep it accurate, and spend your real effort on content, schema and entities.

How should you actually optimize for AI answer engines?

The durable playbook is less exotic than a single magic file, and that's good news. Start with genuinely useful content that answers real questions directly — clear, well-structured pages with question-style headings and concise summaries that an engine can lift and quote. Layer JSON-LD schema on top to make that content machine-readable, and connect your entities so engines know who wrote it and what brand stands behind it. Then earn trust the old-fashioned way, through quality and credible third-party mentions.

Crucially, remember that Google rankings and AI citations are separate games. You can rank first and still go uncited, because answer engines weigh structure, entity clarity and trustworthiness differently. Building for both means writing for humans, marking up for machines, and keeping everything fresh and accurate across the languages your audience actually uses.

This is exactly what Artiql automates: multilingual SEO- and GEO-ready articles with built-in structure and schema, an AI video per piece for YouTube and social, a review queue, and publishing to your own domain. If you'd like to see it work on your content, book a demo and we'll walk you through it.

Frequently asked questions