Structured Data for AI Search: llms.txt vs Schema

What's the real difference between llms.txt and schema markup?

They solve different problems for different readers, which is why pitting them against each other confuses so many teams. Schema markup is per-page JSON-LD embedded in your HTML. It labels entities — this is an organization, this is a product, this is a question and its answer — so machines can parse meaning without guessing from prose. It powers traditional rich results and gives crawlers an explicit, structured answer key sitting inside each page.

The llms.txt file is something else entirely. It's a single plain-text Markdown document at the root of your domain, written for AI retrieval pipelines rather than link-following crawlers. Think of it as a hand-curated map pointing to your highest-value pages and clean content versions. It doesn't grant or deny access like robots.txt, and it doesn't describe individual entities like schema. It recommends priority across your whole site.

So they're complementary, not rivals. One labels meaning at the page level; the other suggests importance at the domain level. The mistake worth avoiding is treating llms.txt as a replacement for schema, or assuming either one guarantees you a citation in an AI answer. Neither claim holds up once you look at what crawlers actually fetch and parse in practice.

Do AI crawlers actually read llms.txt?

Mostly, no — and the server logs are blunt about it. A widely-discussed audit reviewed 30 days of raw CDN traffic across 1,000 large enterprise domains to see who requested the file. Nearly 95% of all hits to llms.txt came from Google's general crawler, while the AI-specific bots everyone is optimizing for — GPTBot, ClaudeBot, PerplexityBot — were essentially absent. The file existed; the intended audience simply wasn't showing up to read it.

The platform signals point the same way. No major model provider has formally committed to reading llms.txt in their crawlers, and Google has stated plainly that no AI system currently uses it. Google's own guidance goes further, telling teams they don't need to create special machine-readable files, AI text files, or Markdown indexes to appear in generative search — and it names llms.txt directly in that list.

There's one real exception worth keeping in mind. When a human pastes your URL into an AI tool, several assistants will happily fetch and use an llms.txt file to navigate your site. That's a genuine use case, just a narrow one. It means the file can help on demand, but it isn't quietly feeding the autonomous crawlers that build most AI answers today.

Does JSON-LD schema actually move AI citations?

Here the evidence is messier and more interesting. Controlled tests where product details lived only inside JSON-LD found that ChatGPT, Claude, Perplexity, Gemini and Copilot all missed the data — strong proof that schema alone isn't reliably read as a standalone signal. Yet correlation studies tell a friendlier story: a large share of pages cited in AI answers happen to carry structured data, and well-implemented schema correlates with meaningfully higher citation rates than no schema at all.

Both things can be true. Schema rarely works as a secret machine channel, but it tends to travel with the qualities AI engines reward — clear entities, tidy question-and-answer structure, and content that's easy to extract. Microsoft is the clearest beneficiary: Copilot inherits schema understanding from Bing, so structured data there has a confirmed path to influence. For Google's AI surfaces the mechanism is unconfirmed, though the pattern suggests it plays some supporting role.

The crucial caveat is that bad schema can underperform no schema. Audits found minimally-populated markup correlating with notably lower citation rates than pages with none, while malformed JSON or content that doesn't match the markup can make engines distrust your data outright. Treat these vendor figures as directional correlation, not proven cause — but the direction is consistent: complete and accurate helps, half-finished hurts.

Why do AI crawlers ignore markup they don't render?

It comes down to how these systems read a page. ChatGPT, Claude and Perplexity generally work with rendered or extracted text, not the raw HTML source. When markup sits in a script tag the model never executes, it can be skipped entirely — which is exactly what the standalone-schema tests revealed. The crawler grabs the visible words, strips the chrome, and reasons over that. Anything not surfaced as readable text is at risk of being left on the floor.

There's a practical reason this happens. A modern page can be hundreds of kilobytes to several megabytes of HTML, scripts, cookie banners, navigation and ads. Models either render that through an expensive browser-like pipeline or flatten it to text — and flattening tends to mangle tables, drop semantics, and discard tucked-away markup. The cheapest, most reliable path is to read the prose a human sees.

The takeaway flips the usual instinct. Instead of betting visibility on hidden machine signals, put the meaning where the model is guaranteed to look: in clear headings, direct answers near the top of sections, and self-contained sentences that survive being copied out of context. Schema and llms.txt become reinforcement, not the foundation. This is the core of structured data AEO — structure the visible content first, then label it.

Which structured signals should you prioritize first?

Start with the layer every engine actually reads: the content itself. Lead sections with a crisp, quotable answer, write in self-contained statements, and use real headings that mirror the questions people ask. Make your entity identity unmistakable in visible text — who you are, what you do, what you're known for. These are the signals with the most consistent evidence behind AI citations, and they cost nothing but discipline.

Next, keep schema, but treat it as amplification rather than a growth hack. Maintain accurate Organization markup for entity verification, Article and FAQ-style structure on content pages, and make sure every property is fully populated and matches what's visible. Validate before you ship. The goal is a clean, trustworthy knowledge layer that helps the engines that do parse it — Bing and Copilot today, likely more tomorrow — without ever contradicting your page.

Put llms.txt last, as low-cost insurance. If your stack generates it automatically, ship a curated, navigation-only file and move on. Don't expect it to feed autonomous crawlers yet, and don't sink hours into hand-maintaining it. The honest priority order is content and entity clarity first, validated schema second, llms.txt a distant third — exactly the inverse of where many teams currently spend their attention.

Could llms.txt still pay off down the road?

Quite possibly — which is why writing it off completely would be a mistake too. The problem it targets is real: pages are bloated, context windows are finite, and a clean, curated index of your best content is genuinely useful to any retrieval system. Adoption among documentation tools is climbing, public implementations now number in the thousands, and the web has a long history of early conventions becoming standards once enough sites and vendors lean in.

The smart posture is cheap optionality. If generating llms.txt is near-zero effort in your platform, do it and forget it. You capture the on-demand use case where humans paste your URL into an assistant, and you're positioned if crawlers start honoring the file. What you shouldn't do is reorder your roadmap around an unproven signal or believe it will lift citations on its own. Insurance, not strategy.

This is exactly the kind of layered, evolving work that's painful to manage page by page. Artiql builds multilingual SEO and GEO articles with clean structure, accurate schema and machine-legible content baked in — plus AI video per article and a review queue publishing to your own headless CMS. If you'd rather run organic marketing on autopilot than babysit structured-data trends, book a demo and see it on your own content.

Frequently asked questions