Answer · checked 31 August 2026
llms.txt vs robots.txt — what do AI crawlers actually respect?
robots.txt is respected: it is a thirty-year-old convention that every major crawler including GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended reads and obeys, and it is the only file on this list that changes what a machine is permitted to do. llms.txt is a proposal, not a standard — no major AI vendor has committed to reading it, and publishing one grants and denies nothing. It is worth having as a curated plain-text summary of your site for whatever does read it, and it is worth nothing as an access-control mechanism.
Colour International LLC · 31 August 2026 · colourintl.com/learn/llms-txt-vs-robots-txt
The honest state of llms.txt
We publish an llms.txt. We also think most of what is written about it is overstated, and saying so is more useful to you than selling it. As of this writing no major model provider has published a commitment to fetch or honour llms.txt, and server logs across the sites we have instrumented show it requested far less often than robots.txt or a sitemap.
What it is genuinely good for: it is a single plain-text page, under your control, that states what your organisation does, what it sells, what is true about it, and where the evidence lives — with no markup to parse and no JavaScript to run. When something does read it, it reads your framing rather than inferring one. That is a modest, real benefit and it costs an afternoon. It is not a ranking factor and there is no evidence it is a citation factor.
What robots.txt controls, precisely
Different bots do different jobs and the same company runs several. Blocking the wrong one has consequences people do not expect: GPTBot is OpenAI's training crawler, while OAI-SearchBot is the one that supports ChatGPT's search results — block the second and you remove yourself from ChatGPT citations while leaving training access untouched, which is usually the exact inverse of what was intended.
Likewise Google-Extended governs Gemini training and grounding, not AI Overviews or AI Mode eligibility, which follow Googlebot. If your goal is to be cited but not trained on, the configuration is: allow Googlebot and the search-side bots, disallow the training-side ones, and state the policy in a form a human reviewer can read.
Where the real leverage is
None of these files make a site legible. They make a site *permitted*. What determines whether an assistant can answer a question about your business is whether the answer is in the HTML your server returns, whether the entity is identified unambiguously, and whether there is any machine-readable statement of what you offer and what can be done. A perfect robots.txt in front of a page that renders empty without JavaScript is a well-marked door into an empty room.
Related
How do I make my site work with Google's AI Mode?
Google's AI Mode does not visit your site to answer a question.
What is /.well-known/agent.json and should I publish one?
A capability manifest at /.well-known/agent.json is a small JSON file at a predictable address that states, in machine-readable form, who a site belongs to, what it offers, what an agent is permitted to do, and which operations exist — the thing a site has no other way to say.
Check your own site
Scan any domain at colourintl.com — sixteen weighted checks, free, no account, and the transcript of all eight requests is printed on the result page. Everything it finds is yours to implement at no cost.
This page is also available as Markdown: curl -H "Accept: text/markdown" https://colourintl.com/learn/llms-txt-vs-robots-txt