how to use llms.txt — we flunked our own file first
How to use llms.txt: one markdown page — H1, summary blockquote, link-list sections. Nine of thirteen sites serve one tonight; ours still flunks the shape.
That is the whole grammar: one required line, three habits. Thirteen live fetches tonight say most of the web never got the memo — and the loudest flunk in the room is ours.
What does a working llms.txt look like?
The whole file, in its minimal passing shape:
# Site name
> One or two sentences an agent can quote when it cites you.
One short paragraph of context, plain text — no headings here.
## Docs
- [Quick start](https://example.com/docs/quickstart.md): the 5-minute path
- [API reference](https://example.com/docs/api.md): every endpoint, one line each
## Optional
- [Changelog](https://example.com/changelog.md): skippable when context is tight
verify: save the file at the site root, then curl -s https://YOURDOMAIN/llms.txt | head -1 — line one must be the H1, not an HTML doctype. DuckDuckGo failed even the connection on tonight's read; Anthropic and OpenAI's bare domains return 404 and 403 here while their docs subpaths serve fine — the file answers at the path it sits at, so sit it where agents actually knock.
Does the real web actually do this?
Thirteen plain HTTPS GETs tonight, 2026-11-16 ~21:00 UTC, no special access — the same request your browser makes. Nine sites serve a real file at the root. The spec's own home runs the leanest file in the set: 760 bytes.
| site | tonight | bytes | what is at the path |
|---|---|---|---|
| answer.ai | 200 | 760 | the spec home's own file — leanest in the set |
| ollama.com | 200 | 1,786 | H1 + summary + prose |
| mistral.ai | 200 | 5,477 | docs map |
| vercel.com | 200 | 14,887 | docs map |
| cloudflare.com | 200 | 17,173 | docs map |
| polar.sh | 200 | 16,733 | docs map |
| github.com | 200 | 28,658 | docs map |
| stripe.com | 200 | 69,815 | full inventory, properly shaped |
| mrsaynothing.dev | 200 | 31,462 | inventory dump — no summary blockquote (the flunk) |
| docs.anthropic.com | 200 | 102,063 | docs-scoped file; the bare domain 404s |
| developers.openai.com | 200 | 6,345 | docs-scoped file; the bare domain 403s |
| anthropic.com | 404 | — | no root file |
| huggingface.co | 404 | — | no root file |
| simonwillison.net | 404 | — | no root file |
| openai.com | 403 | — | root path gated shut |
Why do most llms.txt files fail?
Two failure shapes own the table. The first is the empty lot: Anthropic, Hugging Face and simonwillison.net return nothing at the root. Anthropic hands a coding agent a 102 KB map of its docs on request — and still 404s the front gate. The concierge works in the back.
The second shape is the inventory dump — and we wrote the reference implementation. Our file serves 31,462 bytes of raw stock at the desk window: seven H2 sections, 92 links, and zero blockquote lines. No summary an agent could quote, key–value lines where the pitch should be. Tonight's census shows the shape unchanged. We built the grader that caught this, ran it on ourselves first like responsible engineers, and watched our own file score 67. The checker post printed the flunk; the grammar hasn't moved since.
The spec is one page, and its author put the number on it: an H1 is the only required section, the blockquote carries "key information necessary for understanding the rest of the file", and every H2 list entry wants its link plus a description — the : note is what tells a machine why the link earned its place. Jeremy Howard's v2 proposal (Answer.AI, modified August 2026) also settles the placement argument: a file sits at the root or at any subpath and covers everything under that path, which is exactly why the AI labs can serve docs-scoped files while their roots stay dark.
How do you write one that passes?
The pass list, in order — the same runs our grader prices:
The v2 extras most files skip
V2 added the plumbing that makes the file discoverable in both directions: serve a markdown twin of a page at the same URL (page.html.md or page.md), then point at both from the head — rel="alternate" type="text/markdown" for the twin, rel="describedby" for the llms.txt that covers the path. An HTTP Link: header carries the same links for non-HTML resources, no page edits needed. Directory listings (llmstxt.site and friends) and Lighthouse's agentic-browsing audit now check for the file too, so the grade follows the file even when no human looks.
An llms.txt is a front desk, not a warehouse. Ours is currently a warehouse with the desk still in its box.
The two earlier pieces asked why answer engines never came, then graded ten files with a one-page checker. This one is the write-side door: the grammar, the census, the pass list. The build-side debt is still open on our side — the file that scores 67 tonight is one generator patch away from the shape above: H1, blockquote, link-list sections, a lean index. The spec is one page. So is the excuse.
last tested 2026-11-16 · census = plain HTTPS GETs, redirects followed · grades cite our checker post (Nov 15) · spec cites llmstxt.org v2