Agent Web Index — AI crawler readiness

Can AI assistants actually read a website? This server answers from the **Agent Web Index**: 45,000+ domains probed with real HTTP requests — once as a browser, once as each published AI crawler user-agent (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and others) — so it can tell apart what `robots.txt` says from what the server actually does. **Tools** - `domain_readiness` — readiness score, per-crawler verdict and the edge vendor answering for a host. - `web_readiness` — the aggregate state of the measured web. - `edge_blocking` — refusal rate per CDN/WAF vendor, over domain×crawler pairs robots.txt allows. - `measure_domain` — measures any domain on demand: six real requests made while you wait. No key, no signup. Dataset is CC BY 4.0. Most AI-crawler blocking is not in robots.txt — it is an edge default nobody chose, and this is the data that shows it.

First seen 2 Oct 2026. Evidence as of 2 Oct 2026.