Answer · checked 1 September 2026
Why do agent-readiness scanners give my site different scores?
Two agent-readiness scanners can score the same site 100 and 31 without either being broken, because they are measuring different things. Most public scanners test whether a site has adopted a slate of emerging protocol endpoints — MCP server cards, OAuth metadata, agent catalogues, DNS discovery records — almost none of which any deployed agent reads yet. Fewer test whether an agent can find, read and understand the site today. Before acting on any score, read the check list rather than the number: a scanner weighted toward files nothing fetches will point your work at the wrong thing, and one that counts a robots.txt rule without reading the rule can pass a site that is closed to every crawler that answers questions.
Colour International LLC · 1 September 2026 · colourintl.com/learn/why-scanners-disagree
The two things being measured
Protocol adoption asks which emerging standards a site has implemented: an MCP server card, an A2A agent card, OAuth discovery metadata, an API catalogue, agent-discovery DNS records, agentic-commerce endpoints. It is a roadmap of where the web may be going, and it is genuinely useful to a platform planning two years out.
Readability asks a narrower question with an answer today: when a crawler that is about to answer somebody's question fetches this site, does it get the content, and can it tell what the business is, what it sells, and how to act? That is structured data, a plain-text summary, content served without requiring JavaScript, and not being turned away at the door.
A site can be excellent at the second and score near zero on the first. That is not a contradiction; it is the two lists disagreeing about what year it is. The mistake is treating a protocol-adoption number as a report card on whether agents can use your site now.
A worked example, and the number to ask about
On 1 September 2026 this site scored 100 out of 100 on our own sixteen weighted checks and 31 per cent on Cloudflare's public scanner at isitagentready.com. Both numbers are correct about what they measure. Ours found a site with a manifest, a plain-text guide, structured data, Markdown negotiation and server-rendered content. Theirs found a site with no MCP server card, no A2A agent card, no OAuth metadata and no agent-discovery DNS records, which is true and which no agent has yet asked for.
The number worth asking about is not the total but the weighting. Their headline is an unweighted tally: passes divided by checks, so having a robots.txt counts exactly as much as running an MCP server. Ours is weighted by our own judgement of impact, which is a different bias and not a neutral one — it is stated on every report for that reason. A score with no published weighting is a number you cannot argue with, which is not the same as one you can trust.
Presence is not permission
There is one failure mode that turns a scanner's advice upside down, and it is worth checking for in any tool you use. A robots.txt check can look for the presence of a rule naming an AI crawler without ever reading whether that rule is an allow or a deny.
Verified on 1 September 2026: nytimes.com returns Disallow: / to ClaudeBot, GPTBot, OAI-SearchBot and PerplexityBot, and Cloudflare's public scanner reports its AI bot rules as a pass, with the note that it found rules for AI bots. It did. The rules say to go away. A site following that green tick believes it is open to the assistants and is invisible to all of them.
The same tool checks fifteen crawler names, and ClaudeBot, Claude-User, Claude-SearchBot and OAI-SearchBot are not among them — its own remediation guide still recommends adding a rule for Claude-Web, a name Anthropic's current crawler documentation does not list. A robots.txt written by following that guide would say nothing about the crawlers actually fetching pages to answer questions about you.
None of this makes the tool useless, and it gets one important thing right that most do not: a domain it cannot reach returns an error rather than a score, instead of reporting a site as failing every check when the real finding is that it was refused. That is the correct behaviour and it is rarer than it should be.
How to read any agent-readiness score
Ask what the checks are before you look at the number. A tool that will not show you its list is asking to be believed rather than checked.
Ask whether the weighting is published. An unweighted tally quietly says every check matters equally, which is never true.
Ask whether it read the rule or only found it. This is the single most consequential difference between two scanners looking at the same robots.txt.
Ask how many pages it looked at. Most scanners read only the homepage. Structured data, canonical tags and server-rendered content are properties of a template, and a site marked up once on its front door can be blank from every other entry point — which a homepage-only scan reports as healthy.
Ask what happens when a site refuses the scanner. If a 403 produces a low score rather than an error, every number that tool gives you about a protected site is noise.
Then check the one thing no scanner can settle, which is whether the files say anything true. A manifest that parses while describing actions that do not work, or a plain-text guide still carrying its template's placeholder text, passes every structural check and misinforms the first customer who asks about you.
Check it yourself
The claim about answering crawlers takes one command and needs no scanner at all. Run it against your own domain and read the status codes: anything other than 200 means the crawler that fetches pages in order to answer a question about you was turned away.
for ua in ClaudeBot Claude-User GPTBot OAI-SearchBot PerplexityBot; do curl -sS -A "$ua" -o /dev/null -w "$ua %{http_code}\n" https://your-site.com/; done
Then scan your domain at colourintl.com. Sixteen weighted checks, free, no account, and the full transcript of all eight requests on the result page so the number can be argued with. Where a check cannot apply it is left out of the total rather than counted as a failure you cannot act on.
Related
llms.txt vs robots.txt — what do AI crawlers actually respect?
robots.txt is respected: it is a thirty-year-old convention that every major crawler including GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended reads and obeys, and it is the only file on this list that changes what a machine is permitted to do.
Agent readiness vs SEO, AEO and GEO — what is actually different?
SEO, AEO and GEO are all about winning a contest for attention: ranking a page, owning a featured answer, or getting mentioned by a model.
Check your own site
Scan any domain at colourintl.com — sixteen weighted checks, free, no account, and the transcript of all eight requests is printed on the result page. Everything it finds is yours to implement at no cost.
This page is also available as Markdown: curl -H "Accept: text/markdown" https://colourintl.com/learn/why-scanners-disagree