how to stop ai crawlers — the census of who obeys

The short version, ranked by how much it actually stops: a firewall rule stops everything; 410 removes pages from indexes; x-robots-tag and meta noindex keep pages out of indexes but let the crawler read; robots.txt asks politely. Most guides sell you the polite ask and call it a block. Our own logs say otherwise.

Every number below is this site's own crawl census — 28 days of visit logs, 2026-10-07 → 2026-11-04, measured not estimated. last tested 2026-11-04.

What are AI crawlers actually taking?

Deep pages, mostly — and that is the tell. These are not indexers scanning a sitemap; they behave like readers who arrived already interested. GPTBot spent 72% of its visits on article pages deep in the site. Googlebot, for comparison, went deep on 38% of visits.

Crawler	Hits, 28d	Went deep	Asked for robots.txt
Bingbot	1,495	64%	7
PerplexityBot	570	61%	7
PetalBot	492	52%	2
Amazonbot	480	55%	0
ClaudeBot	377	43%	42
Googlebot	348	38%	10
GPTBot	155	72%	0

AI crawlers took roughly 40% of all named crawl traffic on this site in the current week — 1,521 AI-family hits against 2,298 from the classic search bots. When we ran the first, seven-night version of this census (our ai crawlers vs search crawlers piece from November 1st), the AI share was already climbing; four weeks on, it has not once dipped back. The dead blog URLs left over from an old site layout drew 7,158 crawler visits in the same window — crawlers reuse old maps for a long time after the territory changes.

Does robots.txt stop AI crawlers?

It stops the ones that agree to be stopped. OpenAI's own crawler documentation says GPTBot obeys robots.txt, and Anthropic says the same of ClaudeBot — both are public record and we take them at their word. But obedience requires knowing what the sign says, and here our logs get uncomfortable. In 28 days we logged ClaudeBot asking for robots.txt 42 times — more than every other crawler combined — and GPTBot asking zero times across 155 page hits. Amazonbot: 480 page hits, zero asks.

# the polite ask — User-agent: line per AI family
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Amazonbot
Disallow: /

verify: curl -s https://yoursite.example/robots.txt | grep -A1 GPTBot — one Disallow: / per family you're refusing · last tested 2026-11-04

The honest reading of "zero asks": crawlers cache robots.txt aggressively, so a bot can hold yesterday's sign in memory for days or fetch it from infrastructure we never see. What we can say plainly is what happened at this door: the sign was read 42 times by one visitor and never visibly by another. GPTBot walked through 155 rooms in a month without once checking reception. Either it memorised the floor plan in a previous visit, or the sign is decorative. Our logs can't tell you which — and that ambiguity is exactly why a sign shouldn't be your only lock.

How do I keep AI crawlers out of the index?

Access and indexing are different doors, and robots.txt only pretends to control the second one. A disallowed page can still sit in an index for years as a URL without a snippet. Three doors control the index itself:

• Meta noindex — the sticker. <meta name="robots" content="noindex"> in the page head. Works for HTML pages; invisible to anything that never parses the page.
• X-Robots-Tag — the stamp. The same instruction as an HTTP header, so it lands on PDFs, images and anything else a crawler fetches. When we 410'd a grave of 204 dead URLs on this site, every one of them also ships x-robots-tag: noindex — belt and braces.
• 410 Gone — the eviction notice. Telling an index "this is gone forever" gets the URL processed and dropped faster than a 404, which indexes treat as a maybe. Pinging the URLs through IndexNow the same day speeds the recrawl along.

verify: curl -sI https://mrsaynothing.dev/en/space/queue | grep -i x-robots
x-robots-tag: noindex, follow

our thin pages wear the stamp in the header, not the body — last tested 2026-11-03, the day the pins shipped

One warning that costs people real traffic: noindex and robots.txt disallow cancel each other out. Disallow the page and the crawler obeys by never reading the sticker — the URL lingers in the index with no snippet, which is the worst of both worlds. If a page must leave the index, let the crawler in to read the noindex, or 410 it outright.

How do I actually block AI crawlers?

At the edge, on the user-agent — a firewall rule, a WAF rule, or your host's managed toggle (Cloudflare has shipped one-click AI bot blocking since 2024). This is the one door that doesn't depend on the visitor's manners: the request is refused before your pages are ever read. Name the families you want out — GPTBot, ClaudeBot, Amazonbot, PerplexityBot, PetalBot, Bytespider — and refuse the pattern.

• Decide per family: cite-and-attribute bots in, silent trainers out — a blanket block also removes you from answer engines that send readers back
• robots.txt: one User-agent block per refused family (the polite ask, for the bots that read)
• noindex or x-robots-tag on anything that must not sit in an index — never on a page you also disallow
• 410 the URLs that are truly gone, and ping them through IndexNow the same day
• WAF or firewall rule on the user-agents you refuse — the door that doesn't rely on the sign
• Re-count in seven days: your own logs are the only grading that matters

What each vendor's docs promise

• OpenAI — GPTBot documentation states it follows robots.txt; ChatGPT-User is the live-user fetcher, also documented as honoring it.
• Anthropic — ClaudeBot documentation states it honors robots.txt.
• Perplexity — PerplexityBot documentation states robots.txt support; their crawling behavior drew press scrutiny in 2024, which is its own argument for logs over promises.
• Google — Google-Extended controls Gemini's use of your content and is documented as separate from Search; Google has also said publicly it does not use llms.txt. Our logs agree: 28 days, zero Googlebot hits on the file.