Artiql

llms.txt Explained: Help AI Crawlers Find You

Quick answer: llms.txt is a plain markdown file you place at yoursite.com/llms.txt that curates links to your most important content so AI models can find and parse it efficiently. Proposed in September 2024, it remains a community convention, not an official standard. Today it mainly helps AI coding agents and documentation tools rather than search rankings — but it's cheap insurance worth shipping.

What exactly is llms.txt?

llms.txt is a single markdown file that lives at the root of your domain — yoursite.com/llms.txt — just like robots.txt or sitemap.xml. Instead of telling crawlers what they can't touch, it does the opposite: it hands AI models a clean, curated map of the pages you most want them to read and understand. Think of it as a friendly briefing note written specifically for machines that read language.

The format was proposed by Jeremy Howard of Answer.AI on September 3, 2024. The motivation was practical. Large language models lean on website content to answer questions, but their context windows are too small to swallow an entire site, and raw HTML is cluttered with navigation, scripts, and ads. A short markdown file points straight to your best, cleanest content.

Crucially, llms.txt is curation, not a data dump. It is not a sitemap listing every URL you own. It is a hand-picked shortlist — the documentation, guides, and canonical pages you'd most want an AI assistant to quote when someone asks about your space.

How is an llms.txt file structured?

The file uses plain markdown — the same lightweight syntax behind GitHub README files — because the people reading it are language models and agents, and markdown is trivially easy for them to parse. The structure follows a fixed order, and only one part is strictly required.

You begin with an H1 heading: the name of your project, company, or site. That single H1 is the only mandatory element. Everything after it is optional but strongly recommended. Next comes a blockquote summary — one or two sentences explaining what you do. Then you can add a short paragraph of context, followed by organized link sections under H2 headings.

Under each H2, you list bullet-point links, and each link gets a brief description telling the model what that page answers. Many teams also add an 'Optional' H2 at the end for secondary resources a model can skip when context is tight. Keeping the whole file tight — often under 500 words — keeps it scannable for both machines and humans.

There's also a sibling file, llms-full.txt, which inlines your entire documentation into one large markdown document. Most sites only need the standard llms.txt; the full version suits developer docs and structured ingestion.

How do you create an llms.txt file step by step?

You need no special tools — any text editor works. Create a file named exactly llms.txt, write your markdown, and save it. Start with your H1 brand name, add a blockquote summary, then sort your most valuable links into logical H2 sections like Getting Started, Guides, or Product. Lead with the pages you most want AI engines to cite.

Place the raw file in your site's root directory so it resolves at yoursite.com/llms.txt. Then verify it: visit the URL in a browser and confirm it returns a 200 status and displays as plain markdown, not a download or an error. If you run a large documentation set, optionally generate llms-full.txt as well.

A few habits keep it useful. Curate ruthlessly rather than listing everything. Link to clean markdown versions of pages when you can, since models parse them more easily than heavy HTML. Validate your syntax before publishing — even small formatting slips can confuse parsers. And keep it fresh: a stale llms.txt is worse than none, because it actively misinforms.

If keeping these files current across multiple languages sounds like a chore, that's exactly the kind of upkeep an organic-marketing autopilot can handle for you.

What does a copy-ready llms.txt template look like?

Below is a minimal structure you can adapt in minutes. Replace the placeholders with your own brand, summary, and curated links, then save it as llms.txt at your root. The pattern is deliberately simple: one H1, one blockquote, optional context, and grouped link sections — each link followed by a short description of what it answers.

The example mirrors the official specification. Notice how every link earns its place with a plain-language note. That description is doing real work: it tells a model not just where a page is, but when to reach for it. The clearer your descriptions, the more accurately an assistant can match your content to a user's question.

Element	Markdown syntax	Required?	Purpose
Title	# Your Brand	Yes	Names the project or site
Summary	> One-line description	Recommended	Tells models what you do
Context	Plain paragraph	Optional	Adds background detail
Link section	## Guides	Recommended	Groups related pages
Link item	- [Title](url): note	Recommended	Points to a key page
Optional block	## Optional	Optional	Resources models may skip

A minimal llms.txt template: required H1 plus recommended sections, in the order the spec expects.

Do AI crawlers actually read llms.txt in 2026?

This is where honesty matters more than hype. After roughly eighteen months of industry chatter, the major AI crawlers largely do not fetch the file during normal crawling. GPTBot, ClaudeBot, PerplexityBot, and Google-Extended overwhelmingly skip /llms.txt and crawl your HTML directly. One 90-day test logged tens of thousands of AI bot visits, of which only a tiny fraction touched the llms.txt file at all.

Google has been explicit. In 2025 its representatives confirmed Google does not support llms.txt and isn't planning to, with one comparing it to the long-discredited keywords meta tag. OpenAI, Anthropic, and Perplexity have not publicly committed to reading it automatically either. So if your goal is an immediate ranking boost, llms.txt is not that lever.

Reports aren't unanimous — some GEO trackers have observed Microsoft and OpenAI crawlers occasionally fetching the files. But occasional fetching is not the same as confirmed influence over what gets cited.

~10%

Site adoption

Roughly one in ten domains in a large 2026 study

0.1%

AI bot fetches

Share of AI crawler traffic that targeted llms.txt in one test

2024

Year proposed

Introduced by Jeremy Howard in September

Snapshot of llms.txt reality in 2026: adoption is climbing, but crawler usage remains tiny.

So when does llms.txt genuinely help?

The real value today is in developer experience and the emerging agentic web, not classic search. AI coding assistants and IDE agents like Cursor and Claude Code can use llms.txt when pointed at it, and it measurably improves how those tools reason about your documentation. MCP servers and doc frameworks such as VitePress, Docusaurus, and Nuxt can generate or consume it too.

If your audience includes developers who lean on AI-assisted editors, shipping an llms.txt is a small, concrete win. It also positions your brand for what some call the business-to-agent future — the first standardized way to publish a machine-readable surface that autonomous agents can route on as the ecosystem matures.

The pragmatic 2026 stance is to treat llms.txt as low-cost insurance and optionality. The file costs almost nothing to create and maintain. If the major crawlers do start respecting it, you'll already be correct — and that head start is cheap to buy today.

Pros

+Near-zero cost to create and host
+Helps AI coding agents and IDE tools today
+Positions you for an agent-driven web
+Easy to keep updated as docs change

Cons

−No confirmed search-ranking benefit
−Major crawlers rarely fetch it yet
−Google has declined to support it
−A stale file can misinform models

Should you ship an llms.txt now? A balanced view for busy operators.

How does llms.txt fit a broader SEO and GEO strategy?

llms.txt is a tidy finishing touch, not a foundation. Generative engine optimization still rests on the fundamentals: genuinely useful, well-structured content that answers real questions, clear headings, quotable summaries, and clean markup that any crawler can parse without help. Get those right and you earn visibility whether or not a model ever reads your llms.txt.

Where llms.txt shines is as part of a system. Pair it with a consistent publishing rhythm, internal links that connect related articles into authoritative clusters, and content written natively in each language you target rather than machine-translated. The file then becomes a clean index pointing AI engines at work that already deserves to be cited.

That full stack — multilingual SEO and GEO articles, a review queue, a headless CMS on your own domain, and machine-readable files like llms.txt generated automatically — is a lot to run by hand. If you'd rather put it on autopilot, book a demo and see how it fits together.

Frequently asked questions

Is llms.txt the same as robots.txt?

No. They sit in the same place — your domain root — but serve opposite purposes. robots.txt tells crawlers which paths they may not access, acting as a gatekeeper. llms.txt instead invites AI models in, handing them a curated markdown map of your most valuable content so they can parse it efficiently. robots.txt is widely respected by crawlers today; llms.txt remains an optional convention that most AI crawlers don't yet fetch automatically.

Will adding llms.txt improve my Google rankings?

There's no evidence it does. Google has stated on the record that it doesn't support llms.txt and has no plans to, with one representative comparing it to the obsolete keywords meta tag. Treat llms.txt as a developer-experience and future-proofing measure, not a ranking tactic. Your search and AI visibility still depend on genuinely useful, well-structured content and clean markup — the file is a finishing touch, not a shortcut to higher positions.

What's the difference between llms.txt and llms-full.txt?

llms.txt is a short, curated index linking to your most important pages with brief descriptions of each. llms-full.txt is a larger file that inlines your full documentation into one big markdown document, so a model can ingest everything in a single fetch. Most websites only need the standard llms.txt. The full version suits developer documentation or structured data ingestion, where having all content in one place outweighs the bulk.

How often should I update my llms.txt file?

Update it whenever the pages it references change meaningfully — new flagship guides, renamed sections, or retired URLs. A stale llms.txt is worse than having none, because it actively points AI models at outdated or broken content and misrepresents your site. Build a quick review into your normal content workflow, or use a platform that regenerates the file automatically as you publish, so the index always reflects your current, canonical pages across every language you serve.

Do I need technical skills to create an llms.txt file?

Not really. The file is plain markdown — the same simple syntax used in GitHub README files — and you can write it in any text editor. The main steps are: create a file named exactly llms.txt, add an H1 title, a short summary, and grouped links with descriptions, then upload it to your site's root directory. The only mild technical part is placing it at the root so it resolves at yoursite.com/llms.txt and returns a 200 status.