robots.txt checklist

A practical robots.txt checklist based on Google's robots.txt spec: location, syntax, rule precedence, status-code handling, and mistakes that block the wrong URLs.

robots.txt is a small file with outsized consequences: one wrong line can hide an entire site from crawlers, and a missing line can let crawlers waste time in crawl traps. Use this checklist when creating a file, reviewing a site, or before a launch. Every point below reflects how Google documents its own handling of robots.txt.

1. Location and scope

The file must be at the top-level directory of the host: https://www.example.com/robots.txt. Crawlers don't look for robots.txt in subdirectories.
Rules apply only to the exact protocol, host and port where the file is served. example.com, www.example.com, shop.example.com and http:// vs https:// each need their own file.
The robots.txt URL itself is case-sensitive: use lowercase robots.txt.

2. HTTP status and availability

How the file is served matters as much as what it says:

Response	How Google treats it
2xx	Processes the file as served.
3xx	Follows at least five redirect hops, then treats it as a 404.
4xx (except 429)	As if no robots.txt exists — no crawl restrictions at all.
5xx, 429, timeouts, DNS errors	Treated as a server error. For the first 12 hours Google stops crawling the site while retrying; then for up to 30 days it uses the last good cached copy. After that it depends on whether the rest of the site is available.

Checklist items: the file returns 200 quickly; it doesn't redirect through a long chain; it isn't accidentally returning 5xx behind a CDN or firewall rule; and a 401/403 isn't being used as a way to slow crawlers (Google says 4xx codes other than 429 have no effect on crawl rate).

Google generally caches robots.txt for up to 24 hours, so changes are not instant.

3. Format

UTF-8 plain text, served as text (not an HTML error page).
Under 500 KiB. Google ignores anything after that limit. If you're close, consolidate rules — for example by moving excluded content under one directory.
One field: value per line. Field names are case-insensitive; path values are case-sensitive.Disallow: /Admin/ does not block /admin/.
Paths must start with /. A rule with an empty path is ignored.
Comments start with #.

4. Only use fields Google supports

Google supports user-agent, allow, disallow and sitemap. Other fields are ignored, including:

crawl-delay — not supported by Google (some other search engines do use it).
noindex in robots.txt — not supported. Use a robots meta tag or X-Robots-Tag header on a crawlable page instead.

5. User-agent groups

A crawler obeys only one group: the one with the most specific matching user-agent. If you add a User-agent: Googlebot group, Googlebot ignores the User-agent: * group entirely — so repeat any shared rules inside it.
The order of groups in the file doesn't matter.
Multiple groups for the same user-agent are merged.
Consecutive user-agent lines share the rules that follow, and a sitemap line does not end a group. So user-agent: a, then a sitemap line, then user-agent: b and disallow: / blocks both a and b. Keep groups clearly separated.

6. Rule matching and precedence

* matches any sequence of characters; $ marks the end of the URL. A trailing * is redundant.
Rules match from the start of the path. Disallow: /fish blocks /fish.html and /fishheads but not /catfish.
The most specific rule (longest matching path) wins. If an allow and a disallow are equally specific, the less restrictive allow wins.

Example: block internal search and sort parameters, but keep everything else open.

User-agent: *
Disallow: /search
Disallow: /*?*sort=
Disallow: /*?*sessionid=

Sitemap: https://www.example.com/sitemap.xml

7. Don't block what Google needs

Keep CSS, JavaScript and images that pages need for rendering crawlable, otherwise Google may not understand the page as users see it.
Don't block pages you want de-indexed. A blocked URL can still be indexed without its content if other pages link to it, and Google can't see a noindex on a page it can't crawl.
Don't use robots.txt to hide private content. The file is public and compliance is voluntary; use authentication instead.
Check for leftover staging rules like Disallow: / after a launch or migration.

8. Sitemap lines

Use a full absolute URL including protocol and host.
You can list several sitemaps; the line isn't tied to any user-agent group.
The sitemap can live on a different host than the robots.txt file.

More on what goes inside those files in XML sitemap best practices.

9. Test before and after deploying

Use the robots.txt report in Search Console to see which robots.txt files Google found for your property, when they were last crawled, and any warnings or errors.
Use URL Inspection to confirm whether specific important URLs are blocked.
Run a quick check with the free Crawl Budget Snapshot to see your disallow rules and sitemap references side by side.
If you also set rules for AI crawlers (GPTBot, ClaudeBot, Google-Extended and others), confirm each one resolves the way you intended with the free AI crawler checker. For many domains at once, the AI Crawler & robots.txt Readiness Audit on Apify audits them in bulk ($0.01 per domain audited).
Afterwards, watch the Crawl Stats report host status for robots.txt fetch problems.

Check your own site

The free Crawl Budget Snapshot fetches a site's public robots.txt and sitemaps and gives a quick crawl-waste score with suggested fixes. It is a starting point, not a replacement for Search Console or log analysis.