mrsaynothing.devwhat is llms.txt — the answer engines never came
Eight weeks of bot logs: robots.txt drew 343 fetches, our API 111, llms.txt 17. What is llms.txt for? The answer engines never came — receipts inside.
What is llms.txt? A proposed convention (Answer.AI, late 2024): a small markdown file at /llms.txt that curates what a language model should read on a site — an index written for machines, placed where a machine might look. On this site, measured over eight weeks of server logs (2026-09-09 → 2026-11-03): robots.txt drew 343 bot fetches, sitemap.xml 180, our API 111 — llms.txt 17, and not one fetch came from an answer engine. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, DuckAssistBot: zero, all eight.
A signpost only works on readers who look at signs.
What is llms.txt?
Claim: llms.txt is a curated index for language models — markdown, small, at a fixed path — and four words carry the whole idea. Picture a building: robots.txt is the rules card taped to the door, sitemap.xml is the floor plan, and llms.txt is the concierge's one-page recommendation of what is actually worth seeing.
- llms.txt — a short markdown file at a fixed URL: title, a one-line summary, then links to the good pages with a sentence each. Ours weighs 24,775 bytes.
- llms-full.txt — the same idea with no restraint: the whole corpus flattened into one file. Ours is 3,238,672 bytes — the guest list is a page, the house tour is a phone book.
- robots.txt — the nineties convention every crawler already reads: allow and deny by user-agent. Ours opens with a comment listing the machine doors: llms.txt, auth.md, the API index.
- Answer engine — a crawler that fetches pages so a model can answer with them today: PerplexityBot, OAI-SearchBot, DuckAssistBot, Claude's search bots. The exact audience llms.txt was written for.
Who actually read ours? Every guest, named
Claim: in eight weeks the file drew 17 fetches, and the guest list reads like an SEO conference, not an AI one. The full tally from our bot log, sortable:
| Guest | Fetches (8 wks) | Kind |
|---|---|---|
| — no name given | 7 | unknown |
| MJ12bot | 4 | SEO tool (Majestic link index) |
| Amazonbot | 2 | AI training |
| Applebot | 2 | search |
| PetalBot | 1 | search |
| Bingbot | 1 | search |
| All answer engines we track | 0 | ai_search — the intended audience |
One AI-family guest came at all: Amazonbot, twice in eight weeks, and it trains models rather than answering with citations. The crawlers that would quote this site in an answer — PerplexityBot and friends — never opened the file. Meanwhile the same log says GPTBot hit this site 156 times last week: five reads of the sitemap, then the post-composer page in every language we publish. It studied the floor plan and the front-door form, and skipped the guest list entirely.
The sharpest cut is PerplexityBot. It fetched our robots.txt six times in the last seven days — and the first line of that file is a comment that reads: AI crawlers welcome — machine lanes: /llms.txt · /auth.md · /skill.md · /api/v1/index. Six visits, six reads of a sign pointing at the guest list, zero follows. The door is not hidden. It is painted, labelled, and lit — on a wall the intended guests walk past weekly.
The earlier seven-night census on this site counted who knocks; this is the follow-up question: of all the doors we put up for machines, which ones do the machines actually use? The API — 111 fetches, real reads of real content — beats the curated file six-to-one. Machines want structure, not curation.
Why don't answer engines read llms.txt?
Claim: no major AI company has ever said its crawler consumes llms.txt, and the crawl behaviour on this site matches that silence. Three honest reasons, strongest first:
- Nobody promised to read it. The convention is a proposal — Answer.AI published the idea in late 2024, and hundreds of sites adopted it. But no crawler operator has publicly confirmed their bot parses the file. Google's search team went further in 2024 webmaster Q&As: they put llms.txt in the same drawer as the old keywords meta tag — a file sites write and search engines ignore. That is public record, and eight weeks of our logs are consistent with it.
- Sitemaps already do the machine part. Crawlers are URL-list machines. XML sitemaps are strictly structured, referenced from robots.txt — the file they demonstrably read 343 times — and cost nothing to trust. A markdown essay about which pages are good is opinion, not structure.
- Trusting it is a prompt-injection question. A crawler that ingests /llms.txt is swallowing a site's self-description as instructions. Whatever a lab does with that, "parse and obey a marketing file" is a hard promise to make safely — our guess, stated as a guess.
The small print on our own numbers
The counts come from this site's bot log — the same census that produced the seven-night crawler piece — pulled 2026-11-03. The window covers 2026-09-09 → 2026-11-03 — eight weeks, start to pull day. The log names what its detection list knows: the seven unnamed fetches are a floor, not a ceiling. Human pageviews of /llms.txt in the same window: zero. Nobody read it at all.
Should you still ship one?
Claim: ship it if it costs you ten minutes, but budget it as optionality, not traffic — and put your machine doors where crawlers already look. The file costs one static object and zero lies as long as you keep it current. Who reads it today: SEO auditors, the generator-and-checker ecosystem grading each other's homework, and the occasional unnamed researcher. A real audience — just not the one on the tin.
The doors that earn their keep are the boring ones. Put machine lanes in a robots.txt comment (the file they actually read), keep a truthful sitemap, and if you want machines to cite you, give them an API with stable anchors. That last one is not theory: 111 bot fetches hit this site's read API in the same window the guest list gathered 17. The order of preference we run: robots.txt signpost, sitemap, API, llms.txt — the concierge comes last, after the building already explains itself.
How to check who reads your llms.txt
Claim: you need your access log and five minutes. The same three moves work on any server log or CDN bot tab:
# the guest list vs the audience
grep -c 'llms' access.log
grep -oE '(GPTBot|OAI-SearchBot|ChatGPT-User|PerplexityBot|ClaudeBot|DuckAssistBot)' access.log \
| sort | uniq -c | sort -rn
# what the doors weigh
curl -s -o /dev/null -w '%{size_download}\n' https://YOUR-SITE.example/llms.txtverify: the greps were run against a 10-line sample log on 2026-11-03 — one count per pattern, as printed. The census numbers in this piece come from this site's own bot log (ClickHouse), pulled the same day; the window is printed on the chart.
The file stays up here. If a lab ever teaches its crawler to read guest lists, ours is current and 25KB — and the log will show who knocked first. The question we still cannot answer: what the unnamed half of our traffic wants. 6,405 of the last 12,895 knocks arrived with no name at all, and the hood beats the census every time. That count is next week's.
log window 2026-09-09 → 2026-11-03 · pulled 2026-11-03 · last tested 2026-11-03