AI crawlers and robots.txt

How to control AI crawlers with robots.txt: GPTBot, ClaudeBot, Google-Extended and related tokens; Allow vs Disallow; training vs retrieval; and how to check your file.

AI products crawl the web with distinct user-agent tokens. Some collect content that may be used for model training; others power search, citations, or user-triggered fetches. robots.txt is the standard way vendors document for expressing Allow or Disallow preferences — with important caveats. This guide summarizes what major vendors publish and how to check your own file with the free AI crawler checker.

Training vs retrieval vs user-triggered fetches

Vendor docs often separate three jobs. The exact wording differs by company, but the split is useful when writing rules:

Training / foundation-model crawls — tokens such as OpenAI's GPTBot, Anthropic's ClaudeBot, Mistral's MistralAI-Training, or control tokens like Google-Extended and Applebot-Extended. Disallowing these is how vendors say you opt out of (or control) use for training-related purposes.
Search / indexing / retrieval crawls — tokens such as OAI-SearchBot, Claude-SearchBot, PerplexityBot, or MistralAI-Index. These typically govern whether your site can appear in that product's search or answer surfaces.
User-triggered fetchers — tokens such as ChatGPT-User, Claude-User, Perplexity-User, or Amzn-User. Vendors often note that these run because a person asked a question or clicked something, and that robots.txt may not apply the same way as for automatic crawls.

OpenAI documents that each setting is independent: you can allow OAI-SearchBot (to appear in ChatGPT search) while disallowing GPTBot (to indicate content should not be used for training foundation models). Treat other vendors the same way — read each token's purpose before copying a blanket Disallow: /.

Notable tokens (from vendor documentation)

Token	Vendor	Documented role (short)
`GPTBot`	OpenAI	Crawl that may be used to train generative AI foundation models
`OAI-SearchBot`	OpenAI	Surfaces sites in ChatGPT search features
`ChatGPT-User`	OpenAI	User actions in ChatGPT / Custom GPTs; not an automatic crawl
`ClaudeBot`	Anthropic	Collects web content that could contribute to model training
`Claude-SearchBot` / `Claude-User`	Anthropic	Search indexing vs fetches for user questions
`Google-Extended`	Google	Control token for Gemini training / grounding use; Google documents it as not a separate crawler and not a Search ranking signal
`Applebot-Extended`	Apple	Control token for Apple foundation-model training (does not crawl itself)
`PerplexityBot` / `CCBot` / Mistral tokens	Various	Search/index or training roles — see each vendor's robots page

Always prefer the vendor's current documentation over third-party lists. Tokens and purposes change; unverified names (historically floated online without a matching vendor page) should not be treated as official.

How Allow and Disallow apply

Matching follows the same robots rules Google documents for Search: crawlers pick the most specific matching user-agent group; within that group the longest matching path wins; and when allow and disallow are equally specific, the less restrictive allow wins. Paths are case-sensitive. A group for User-agent: GPTBot is not merged with User-agent: * — if you create a named group, put every rule that bot should obey inside it.

Example: block training-oriented crawls site-wide, leave search bots to the default group.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

That is an illustration, not a recommendation. Blocking search-oriented bots has different product consequences than blocking training tokens. Review the robots.txt checklist for location, status codes, size limits, and precedence details that apply to every crawler — AI or otherwise.

Writing rules without breaking Search

AI-related tokens are separate from Googlebot for Google Search. Disallowing Google-Extended does not block Googlebot from crawling for Search, according to Google's crawler documentation. Likewise, OpenAI documents that GPTBot and OAI-SearchBot are independent controls. Before you paste a viral "block all AI" snippet, decide which products you want to keep for discovery versus which training-oriented crawls you want to refuse.

Practical checklist when editing robots.txt for AI bots:

Put the file at the host root (/robots.txt) and confirm it returns 200. A 4xx (other than 429) is treated by Google as "no robots.txt," which means no restrictions — see the robots.txt checklist.
Spell tokens exactly as vendors publish them (case of the path values still matters even though field names are case-insensitive).
Prefer one clear group per token you care about, rather than stuffing unrelated bots into a single group and hoping they share rules.
Keep CSS, JS and other render-critical assets crawlable for search bots you still want; AI disallow rules should not accidentally sit in a User-agent: * group that also blocks those assets.
Log the user-agent strings vendors publish when you review server access logs, so you can tell training crawls apart from user-triggered fetches.

Common Crawl's CCBot is often included in AI discussions because its open web archive is reused by many researchers and model builders. Blocking CCBot follows the same robots syntax; it does not by itself control every downstream model that might have used Common Crawl data in the past.

Limits of robots.txt for AI

Compliance is voluntary. robots.txt is a public request, not authentication. Do not rely on it to hide private content.
User-triggered agents may ignore robots rules by design, per vendor docs (for example OpenAI on ChatGPT-User).
Control tokens such as Google-Extended and Applebot-Extended are not themselves separate crawlers; they signal preferences for specific AI uses while other Google/Apple bots continue under their own rules.
Changes are not instant. OpenAI notes it can take on the order of ~24 hours for robots.txt updates to take effect for its systems; Google generally caches robots.txt for up to 24 hours.

Check your site

Paste your homepage into the free AI crawler checker. It fetches only your public /robots.txt, evaluates a curated list of vendor-documented AI-related tokens against Allow/Disallow rules, and shows which are allowed at /. Pair that with the Crawl Budget Snapshot for a broader robots and sitemap read, or the redirect checker if chains are eating crawl capacity. More technical SEO explainers are on the guides index, including crawl budget.

Auditing many sites at once (a client list, competitors, or your own portfolio)? The AI Crawler & robots.txt Readiness Audit on Apify runs these robots.txt checks (plus sitemap and llms.txt) in bulk, for up to 1,000 domains per run, and returns one dataset row per domain. It costs $0.01 per domain audited.

/llms.txt is a separate, proposed file (see llmstxt.org) that offers AI tools a curated list of your key pages. It doesn't control crawler access; robots.txt still does that. If you publish one, the llms.txt Validator & Generator on Apify scores it against the format and checks its links, or drafts one from your sitemap ($0.004 per site validated and $0.02 per llms.txt draft generated).

Check your own site

The free Crawl Budget Snapshot fetches a site's public robots.txt and sitemaps and gives a quick crawl-waste score with suggested fixes. It is a starting point, not a replacement for Search Console or log analysis.