111 real websites × 9 AI crawlers × 4 site assets, plus Tranco traffic rank and Hacker News mention counts. No theory — this is what the live web actually does, split by business model, because that's the variable that explains the data.
Answer four questions and we'll place you against the 111-site benchmark — overall, within your business model, and within your traffic tier.
Every row is a real fetch from Nov 12, 2026. Sort any column; filter by tier, business model or language. 20 further sites (Wikipedia, GitHub, Reddit, Medium…) were unreachable from our network and are excluded from robot-rule stats rather than guessed — they remain in the mention/rank analysis.
| Domain | Tier | Model | llms.txt | Sitemap | JSON-LD | Blocks AI | Tranco | HN mentions |
|---|
Allow: / override) for GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent.# and is over 40 bytes — a SPA catch-all page does not count.<urlset> or <sitemapindex>; JSON-LD must appear in server-rendered homepage HTML.The whole collection is ~120 lines of dependency-free Python. Download the CSV above and audit your own list — that beats trusting ours.