Crawl budget explained

A plain-English explanation of crawl budget based on Google's documentation: crawl capacity, crawl demand, who needs to care, and what actually helps.

"Crawl budget" is one of the most talked-about and most misunderstood ideas in technical SEO. This guide sticks to what Google has actually documented: what the term means, which sites genuinely need to think about it, and which changes help versus which ones are myths.

What crawl budget means

Google describes crawl budget as the set of URLs that Googlebot can and wants to crawl on a site. It is not a single number you can look up. It is the result of two things working together:

Crawl capacity limit (also called hostload) — how much Google can crawl without overloading your servers. It limits the total time your server spends holding connections open for Google, taking into account both the number of parallel connections and how long they last.
Crawl demand — how much Google wants to crawl. This depends on how many URLs Google knows about, how popular they are, and how stale Google's copy is likely to be.

Put simply: capacity is the ceiling, demand decides how much of that ceiling is used. If demand is low, Google will not crawl up to the limit even if your server could handle it.

Two details are worth knowing. First, Google treats each unique hostname as a separate site, so www.example.com and shop.example.com have separate crawl budgets. Second, the capacity limit is shared across Google's crawlers, so heavy demand from one (for example AdsBot) can reduce what is available to others.

What moves the capacity limit

Every site starts with the same conservative default. According to Google, the limit goes up when the site responds consistently and response times stay stable or improve, and goes down when latency increases or the site returns server errors (5xx) or rate-limiting signals such as 429. Google's own crawling resources are also finite, so it has to make choices across the whole web.

What moves crawl demand

Perceived inventory. Without guidance from you, Googlebot tries to crawl all or most URLs it knows about. If many of those are duplicates, or URLs you do not want crawled, time is wasted. Google calls this the factor you can control most.
Popularity. URLs that are more popular on the internet tend to be crawled more often to keep them fresh.
Staleness. Google wants to recrawl documents often enough to pick up changes.

Site-wide events such as a site move can also trigger a temporary increase in crawl demand while Google reindexes content under new URLs.

Does your site need to care?

Google's guide is explicitly written for large or fast-changing sites. It lists these rough profiles (Google stresses the numbers are estimates, not exact thresholds):

Large sites (around 1 million or more unique pages) with content that changes moderately often, such as weekly.
Medium or larger sites (around 10,000 or more unique pages) with content that changes very rapidly, such as daily.
Sites with a large portion of their total URLs classified in Search Console as Discovered – currently not indexed.

Google also says that if your new pages tend to be crawled the same day they are published, you probably do not need to worry about crawl budget. For most small and medium sites, indexing problems are more often caused by content quality, duplication, or technical blocks than by a lack of crawling.

Even so, the hygiene that protects crawl budget — clean URL inventory, accurate sitemaps, fast responses — is good practice on any site.

What actually helps

These recommendations come directly from Google's crawl budget documentation:

Manage your URL inventory. Consolidate duplicate content so crawling focuses on unique pages. Use robots.txt to block URLs that are important to users but should not appear in Search — for example infinite scrolling pages that duplicate linked content, or differently sorted versions of the same list. See how to find crawl traps.
Return 404 or 410 for permanently removed pages. Google will not forget a URL it knows about, but a 404 is a strong signal not to crawl it again soon. Blocked URLs, by contrast, stay in the crawl queue longer.
Eliminate soft 404s. Pages that say "not found" but return 200 keep getting crawled.
Keep sitemaps up to date. Include the content you want crawled and use <lastmod> for updated content. See XML sitemap best practices.
Avoid long redirect chains. They have a negative effect on crawling.
Make pages efficient to load. If Google can load and render pages faster, it may be able to read more content from your site. Supporting 304 Not Modified responses for unchanged pages also saves server resources.
Monitor crawling. Watch for availability problems and check whether important pages are being crawled, using the Crawl Stats report and your server logs.

Common myths and mistakes

Using noindex to save crawl budget. Google still has to request a page to see a noindex rule, so it does not reduce crawling of that URL. Use it to control indexing, not crawling.
Shuffling robots.txt rules to "reallocate" budget. Google advises against temporarily blocking and unblocking pages in robots.txt to shift crawl to other pages. Block only what you never want crawled.
Expecting a crawl-rate increase on request. You cannot ask Google to crawl more. Google's documentation names two ways to get more crawl budget: add server resources if capacity on your end is the bottleneck, and improve the quality and usefulness of your content. (The old crawl-rate limiter setting in Search Console has been retired.)
Treating crawling as a ranking factor. Crawling more often does not by itself mean better rankings. Crawling is a prerequisite for being indexed, nothing more.

If Googlebot is overloading your server

In an emergency, Google's documentation says you can temporarily return 500, 503 or 429 status codes to Googlebot requests. Googlebot will slow down when it sees many of these, and the slowdown applies to the whole hostname. Do not keep this up for long (Google warns against doing it for more than 1–2 days), because if Googlebot sees these codes on the same URL for multiple days, the URL may be dropped from the index.

A quick way to start

Before digging into logs, check the basics: does your robots.txt block the obvious waste (internal search, sort and filter parameters) without blocking important pages or resources? Do your sitemaps list only canonical, indexable URLs with honest lastmod dates? The robots.txt checklist covers the first question, and the free Crawl Budget Snapshot tool gives a quick read on both.

Check your own site

The free Crawl Budget Snapshot fetches a site's public robots.txt and sitemaps and gives a quick crawl-waste score with suggested fixes. It is a starting point, not a replacement for Search Console or log analysis.