Check a site's robots.txt rules for AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more).
Listed on
First seen 21 Nov 2026. One server, whatever directories list it: each directory listing keeps its own page and history.
1
Directories
Collected by InvokeRank
2
Tools
From an anonymous probe
-
ToolBench grade
Not graded by Arcade
-
GitHub stars
No repository data
Tools
| Tool | Description | Behaviour |
|---|---|---|
| check_ai_crawlers | For a domain (and optional path), whether each known AI crawler/agent user-agent is allowed by robots.txt, with the deciding rule. | Read-only |
| check_user_agent | Whether a given crawler user-agent token may fetch a path on a domain according to robots.txt. | Read-only |
| Directory | Listing | Tier | First seen |
|---|---|---|---|
| Official MCP Registry | AI Crawler Permission Check | - | 21 Nov 2026 |