Free Technical SEO utility
robots.txt AI crawler checker
Enter a site and we fetch its public robots.txt, then report for each known AI crawler token whether the site's root path (/) is allowed or blocked, whether the bot is named directly or falls back to the * group, and which line decided it.
How the check works
- Groups are matched the way RFC 9309 and Google's robots.txt spec describe: user-agent names are case-insensitive, every group naming the bot is combined, and the
*group is used only when no group names it. - Within the applicable groups, the longest matching
Allow/Disallowpath wins; on a tie,Allowwins.*and$wildcards are supported. - A 4xx response (such as 404) means no restrictions; a 5xx or 429 means crawlers should assume everything is disallowed until robots.txt is reachable again.
- Only the root path is checked. A site can allow
/but block sections like/articles/, so read the full file before drawing conclusions. - robots.txt is a request. It doesn't enforce anything, and some vendors say their user-triggered fetchers may not follow it (marked in the table).
Which user-agents are listed
Only tokens we could confirm in each vendor's own documentation: OpenAI, Anthropic, Google, Perplexity, Common Crawl, Apple, Meta, Amazon, DuckDuckGo, Mistral AI. Older or unofficial names that appear in many blocklists (for example anthropic-ai, Claude-Web and Bytespider) are left out because we couldn't confirm them in current vendor docs as robots.txt tokens.