Reading the Google Search Console Crawl Stats report

A walkthrough of the Search Console Crawl Stats report: totals, host status, and breakdowns by response, file type, purpose and Googlebot type, with what to look for.

The Crawl Stats report is the closest thing Search Console offers to a view of your server logs from Google's side. It shows how many requests Google made, what your server answered, and whether Google ran into availability problems. This guide explains each part of the report and what patterns are worth investigating.

Before you start

Where to find it: in Search Console, open Settings (property settings) → Crawl stats.
Which properties: it's only available for root-level properties — a Domain property (example.com) or a URL-prefix property at the root (https://example.com/).
Who needs it: Google says it's aimed at advanced users, and that sites with fewer than about a thousand pages generally don't need to worry about this level of detail.
Time range: the report covers roughly the last 90 days.

How the data is counted

A few counting rules explain most of the surprises people run into:

URLs are the actual URLs Google requested, not assigned to canonicals as in some other reports.
Each step of a redirect chain is a separate request. A URL that goes 301 → 301 → 200 counts as three requests.
Page resources (images, CSS, JavaScript) hosted on your site are counted; resources on other domains are not.
Repeat requests for the same URL are counted individually.
For a Domain property, data includes all subdomains, and you can drill into individual hosts.
Google notes the report may not count every request, so small differences from your own logs are expected.

The three summary charts

Total crawl requests

All crawl requests for URLs on your site, successful or not. Look at the trend rather than the absolute number. A sudden, sustained jump can mean new content was published — or that a crawl trap opened up. A sudden drop can point to availability or robots.txt problems.

Total download size

Bytes downloaded during crawling. Rising download size with flat request counts suggests pages or resources are getting heavier.

Average response time

The average time to fetch resources from your site. Google's crawl budget documentation says consistent, stable or improving response times let the crawl capacity limit rise, and slowdowns lower it. Spikes here that line up with drops in crawl requests are worth investigating with your hosting provider.

Host status

Host status tells you whether Google hit significant availability problems in the last 90 days, in three categories:

robots.txt fetching — the failure rate for robots.txt requests. A 200 (even an empty file) or a 404 is fine. A 429 or 5xx is not: Google will slow or stop crawling until it gets an acceptable response. See the robots.txt checklist.
DNS resolution — whether your DNS server recognised the hostname and responded.
Server connectivity — whether your server was unresponsive or returned incomplete responses.

Each category chart shows a threshold line; days above it count as an issue. The status tells you how recent the last issue was — within the last week, or earlier in the 90-day window — so you can judge whether it was a one-off or is ongoing.

The four breakdowns

By response

Share of requests by response type (by request, not by unique URL). Most should be 200. Things to check:

301/302 share — normal during a migration; persistently high outside one suggests internal links pointing at redirecting URLs.
404s — often fine; returning 404 for removed pages is correct. Look at examples to spot broken internal links.
Server errors (5xx), timeouts, DNS or fetch errors — these should be fixed; they can reduce crawling.
304 Not modified — a good sign that conditional requests are supported.

By file type

HTML, image, JavaScript, CSS, JSON, and so on. Redirects appear under "Other file type". If you have slow responses, check whether a particular type (for example large images or an API returning JSON) is driving them, and click in to see response time by type.

By purpose

Discovery — URLs Google had never crawled before.
Refresh — recrawls of known URLs.

After publishing a lot of new content or submitting a sitemap, you'd expect discovery to rise. If discovery is high but you haven't added content, the examples often reveal parameter or faceted URLs being found for the first time. If important, frequently updated pages aren't being refreshed often enough, make sure they're in your sitemap with accurate lastmod values.

By Googlebot type

Smartphone, Desktop, Image, Video, Page resource load, AdsBot, StoreBot and Other agent type. "Page resource load" is the secondary fetch of resources used to render your pages. If you see a spike in crawling, this breakdown tells you which crawler caused it — for example, Google notes that AdsBot spikes are often linked to new Dynamic Search Ads targets.

Example URLs

Clicking any row shows example URLs with crawl time, response code and size. These are a representative sample, not a full list: a URL missing from the examples doesn't mean it wasn't crawled. For a complete picture, use your server logs.

A simple monthly review

Is host status clean? If not, which category and when?
Any unexplained change in total requests or response time?
Has the share of non-200 responses changed?
Do Discovery examples show URL patterns you didn't expect?
Did one Googlebot type drive a spike?

Pair this with a quick look at your robots.txt and sitemaps using the free Crawl Budget Snapshot. For the concepts behind the numbers, read crawl budget explained.

Check your own site

The free Crawl Budget Snapshot fetches a site's public robots.txt and sitemaps and gives a quick crawl-waste score with suggested fixes. It is a starting point, not a replacement for Search Console or log analysis.