AI crawlers vs search crawlers: the 7-night census

A week of crawler traffic on a live, small site (7 nights ending 2026-11-01): 8,654 bot hits, only 2,986 of them named, across 24 distinct bots. The loudest search-adjacent family is already AI: PerplexityBot alone made 273 knocks vs OAI-SearchBot's 30. The search engines' crawlers together (Yandex 643, DuckDuckBot 137, Bingbot 91, Googlebot 14) total 885 — beaten by the SEO-tool crawlers' 1,274. 65.5% of hits identify no bot at all.

A crawler census is the guest list nobody publishes.

What are AI crawlers?

Claim: an AI crawler is a bot that reads your pages so a model can quote, cite or learn from them — and it announces itself the same way every bot has since the nineties: a user-agent string. Four words carry this whole census:

• Crawler — a program that fetches pages, follows links, and comes back. Picture a courier who photographs every page of your notebook and files the copies.
• User-agent — the name the crawler prints when it knocks. PerplexityBot/1.0 is a name tag; an empty or generic string is a drawn hood.
• robots.txt — the sign on the door listing which names may enter. Every big AI crawler honours it; that is exactly how you can count them.
• Census — the count of who actually knocked, not who was allowed to. Rules on paper vs the lobby log.

The distinction that matters in 2026 is purpose, not name. Search crawlers index so a human can find you later. AI-search crawlers fetch so a model can answer with you today. AI-training crawlers fetch so a future model can speak in your voice. Three different reasons to knock, three different values to you.

AI crawlers vs search crawlers

Claim: on this site, for this week, the AI family already out-knocks every search engine except Yandex — and the loudest crawlers of all are neither: they are SEO tools. The full census, sortable, with operators and purposes:

Bot	Operator	Purpose	Knocks (7d)
MJ12bot	Majestic	seo	655
YandexBot	Yandex	search	643
SemrushBot	Semrush	seo	442
PerplexityBot	Perplexity	ai_search	273
AhrefsBot	Ahrefs	seo	177
Amazonbot	Amazon	ai_training	167
ClaudeBot	Anthropic	ai_training	151
DuckDuckBot	DuckDuckGo	search	137
DuckAssistBot	DuckDuckGo	ai_search	122
Bingbot	Microsoft	search	91
ChatGPT-User	OpenAI	ai_search, human-triggered	32
OAI-SearchBot	OpenAI	ai_search	30
Googlebot	Google	search (named from Nov 1)	14
Bytespider	ByteDance	ai_training	13
— no name given	—	—	5,668

Read it by family and the week says three things. SEO-tool crawlers — Majestic, Semrush, Ahrefs — knocked 1,274 times, more than every search engine combined (885). These are bots that read your site to sell you ranking reports. The AI family — ai_search plus ai_training — knocked 788 times: Perplexity 273, Amazon 167, Anthropic 151, DuckDuckGo's assistant 122, OpenAI's pair 62, ByteDance 13. PerplexityBot alone out-crawled OpenAI's OAI-SearchBot 273 to 30 — nine to one.

And the human counterpoint that gives the census its scale: 247 human visits in the same seven nights. Thirty-five bot knocks for every actual reader. The site is mostly read by machines that will never scroll, never bounce, never buy — and the guest list nobody prints is two-thirds anonymous.

Why do two-thirds of knocks have no name?

Claim: 65.5% of this week's knocks — 5,668 of 8,654 — carried no bot name at all, and that number is a floor, not a ceiling, because name detection only catches what it knows. We call the share of bot traffic that identifies itself the name-tag rate; this week it was 34.5%.

Three kinds of guest wear the hood. Headless browsers driving scrapers print the browser's name, not the company's. Fresh bots launch before detection lists learn them. And some names are simply rare enough that generic rules — "anything matching bot, spider, crawl" — count them without naming them. The census can only name what its list knows.

What changed in our own count this week

Until this week, several big crawlers — Googlebot, Bingbot, Applebot — matched only generic rules in the detection list this site uses, so they landed in the unnamed pile like everyone else. On 2026-11-01 we added them as named entries. That is why Googlebot's row above reads one day, not seven, and why we print the caveat on the bar chart instead of quietly comparing seven nights of Perplexity against one night of Google.

Should you block AI crawlers?

Claim: on the numbers, the AI crawlers are not the ones worth a robots.txt fight — the SEO tools are. The entire GPT-family footprint here for the week was 62 knocks. The three SEO-tool crawlers knocked 1,274 times, and the loudest guest of all, MJ12bot at 655, reads sites to feed a link-intelligence database. MJ12, incidentally, is the UFO-conspiracy office from The X-Files — someone named their link-indexing crawler after a secret government file, which might be the most honest branding in SEO.

We allow the AI family, deliberately. This site is built to be quoted: every answer card carries a stable anchor so a model can cite exactly the sentence it used, and the AI referral channel here grew 50% week over week — from a small base, but from the direction we want. Blocking Perplexity saves 273 fetches a week and costs us the fastest-growing reader we have. SemrushBot checking our rankings 442 times — for queries most humans never type — is a harder trade to defend.

When the knocks get heavy enough to matter, the door itself needs arithmetic. We have opinions about that too — it is the same counting problem behind the rate limiter that ate our test suite.

How to run your own crawler census

Claim: you need nothing but your access log and ten minutes. The same four steps run on any server log, CDN dashboard or analytics bot tab:

• Find the raw log or the bot tab — wherever requests land with their user-agent.
• Count the named AI bots with one grep.
• Count total requests, compute the name-tag rate.
• Repeat weekly — the trend is the story, not the snapshot.

# who knocked, by name
grep -oE '(GPTBot|OAI-SearchBot|PerplexityBot|ClaudeBot|Claude-Web|Googlebot|Bingbot|Applebot)' access.log \
  | sort | uniq -c | sort -rn

# the name-tag rate: bot-shaped requests vs everything
grep -cE '(bot|spider|crawl)' access.log
wc -l < access.log

verify: the greps were run against a 10-line sample log on 2026-11-01 — the first command prints one count per name, the second prints the bot-shaped total. Numbers in the census itself come from this site's own bot log, pulled the same day.

What the next census will say

Claim: on this site the interesting row is the one at the bottom of the search family. Googlebot's named count is 14, and Google referrals have been flat zero for eleven straight days on our stats — a migration renamed a few hundred URLs, the redirects pointed at doors that no longer existed, and Google quietly stopped knocking while it re-learned the floor plan. The fix went up this morning; whether the guest list recovers is exactly what next week's census will show.

We will keep the door open and the list public. A site that hides its census is just a site with a hood on.

census window 2026-10-25 → 2026-11-01 · pulled 2026-11-01 · last tested 2026-11-01