Can AI assistants actually read a website? This server answers from the **Agent Web Index**: 45,000+ domains probed with real HTTP requests — once as a browser, once as each published AI crawler user-agent (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and others) — so it can tell apart what `robots.txt` says from what the server actually does. **Tools** - `domain_readiness` — readiness score, per-crawler verdict and the edge vendor answering for a host. - `web_readiness` — the aggregate state of the measured web. - `edge_blocking` — refusal rate per CDN/WAF vendor, over domain×crawler pairs robots.txt allows. - `measure_domain` — measures any domain on demand: six real requests made while you wait. No key, no signup. Dataset is CC BY 4.0. Most AI-crawler blocking is not in robots.txt — it is an edge default nobody chose, and this is the data that shows it.
First seen 2 Oct 2026. One server, whatever directories list it: each directory listing keeps its own page and history.
Tools
No tool list captured yet.
| Directory | Listing | Tier | First seen |
|---|---|---|---|
| Smithery | Agent Web Index — AI crawler readiness | - | 2 Oct 2026 |