mrsaynothing.devthe #2 crawler on this site is petalbot (huawei's)
Huawei's PetalBot out-crawled Googlebot here last week and asked robots.txt just four times in 28 days. The measured census, plus the recipe to run your own.
PetalBot is Huawei's crawler: one index feeds Petal Search, Huawei Assistant and AI Search, and one robots.txt token reaches all three. Last week it out-crawled Googlebot on this site 1,247 to 686 while asking robots.txt four times a month. It honours the token but re-asks rarely — a rule change can ride unseen for weeks. Block it only if the crawl budget hurts.
what is petalbot, and who sends it?
Claim: PetalBot is one crawler serving three Huawei surfaces, and one robots.txt token reaches all of them. Huawei's own crawler page — webmaster.petalsearch.com/site/petalbot — defines it as the indexing program for Petal Search, the engine that ships on Huawei phones without Google services. The same index feeds content recommendations in Huawei Assistant and in Petal's AI Search. There is no separate assistant-only token: block the bot and Huawei's documentation says your pages go unsearchable across every Petal service at once.
The token is PetalBot for both fetchers; the user-agent strings differ only in their preamble. Huawei states it complies with the robots protocol (RFC 9309), publishes no IP ranges, and documents a reverse-DNS check instead — a real PetalBot resolves to a petalsearch.com host (older pages say aspiegel.com, a Huawei subsidiary), and the forward lookup must return the original address. Two warnings ship with the docs: after a block, the existing index can take months to clear, and no Crawl-delay is documented. Unreasonable load goes to [email protected].
what did the week's census measure?
Claim: in the seven days to Nov 7, PetalBot out-crawled Googlebot here 1.8 to 1. The census counts every request our bot lane named — one small multilingual site, no averaging. The shape is the finding.
Two bots beat Googlebot here: PetalBot at 1,247 hits (1.82×), Applebot at 1,179 (1.72×). Nobody grades either the way Googlebot gets graded. The who-obeys census earlier this month (how to stop AI crawlers — the census of who obeys) measured obedience; this one measures volume, and volume is where the surprise lives.
when did petalbot find this site?
Claim: PetalBot arrived almost overnight — 6 hits on Nov 2, 385 on Nov 5. Daily counts, same window:
The ramp begins November 3 — the same day this site's llms.txt explainer went live (what is llms.txt — the answer engines never came). PetalBot has since read /llms.txt, five locale sitemaps, and both editions of that post. We cannot prove why it found us — but the rhyme is on the record, and the doors it read first are the doors built for machines.
does petalbot obey robots.txt?
Claim: it obeys — but it asks rarely, so rule changes land slowly. Robots.txt asks per crawler, trailing 28 days, same census:
| Crawler | robots.txt asks · 28d | pages · 7d | asks per crawl |
|---|---|---|---|
| SemrushBot | 59 | 693 | 1 per 12 |
| ClaudeBot | 54 | 357 | 1 per 7 |
| Applebot | 26 | 1179 | 1 per 45 |
| Googlebot | 25 | 686 | 1 per 27 |
| PerplexityBot | 12 | 578 | 1 per 48 |
| Bingbot | 12 | 2625 | 1 per 219 |
| PetalBot | 4 | 1247 | 1 per 312 |
The polite crawlers re-read the file as they work. PetalBot asked four times in 28 days and read everything else on faith — 1 ask per 312 crawls, the loosest ratio on the board. SemrushBot asked 59 times; paranoia is their business model. Not law-breaking: RFC 9309 lets a crawler cache the file. It means something quieter: change your rules and PetalBot may crawl on for weeks before it learns. Urgent changes ride the email lane; the index takes months to clear regardless.
what does petalbot read besides pages?
Claim: it reads the machine doors and every locale equally — and it sent us nothing. The week's 1,247 hits split almost flat across the locales: en 219, ar 182, es 168, ru 166, zh 165, fr 159. No other crawler spreads that evenly — PetalBot mirrors the whole house, not the English front door. It also pulled the machine doors: 162 hits on the media lane, five locale sitemaps, /llms.txt once.
The honest ledger line: Petal/huawei referrers in 28 days: zero. The bot read the entire house and left nothing at the door — an investment in a search engine that has not paid out here yet. Petal's audience is real on Huawei hardware; we are simply not in front of it. Whether it stays zero is next month's question.
should you block petalbot?
Claim: keep the token unless the load hurts. Verify by reverse DNS before trusting the string. The decision is honest on both sides:
Our call: the token stays. 1,247 pages is a real budget line, but the mix is exactly the crawl a search engine should take. Blocking it buys nothing today and sells a market we have not entered.
how do you run your own census?
Claim: two commands, one table. Any analytics stack with a named-crawler lane gives the first number; the log gives the rest:
grep -c "PetalBot" access.log
grep -c "robots.txt" access.log per crawler gives the ask rate. Above ~1 ask per 100 crawls, rule changes propagate slowly for that bot — write the rule, set a reminder, check the obey date.
Method — the census behind this post
Counts come from this site's analytics bot lane, which names crawlers from user-agent strings. Windows: page counts trailing 7 days (Nov 1–7, 2026, partial final day excluded); robots.txt asks trailing 28 days. Referrals matched against petal/huawei hosts: zero rows. One site, no sampling. last tested 2026-11-07.
The census exists because crawlers grade you all day and nobody grades them back. Last week the second-biggest reader of this site worked for Huawei, asked permission four times a month, read six languages, and sent nobody. Whether that stays a curiosity or becomes a traffic line is next month's census.