How to find crawl traps

Practical steps to find crawl traps such as faceted navigation, infinite calendars, session IDs and parameter loops, and how to fix them without hurting indexing.

A crawl trap is any part of a site that can generate a practically unlimited number of URLs, most of which show duplicate, empty or near-useless content. Crawlers cannot know a URL is useless until they have fetched it, so traps soak up crawling that could have gone to your real pages. Google's documentation calls these "infinite URL spaces" and names two harms: overcrawling, and slower discovery of new useful URLs.

The usual suspects

Faceted navigation. Filters for colour, size, price, brand and so on, usually as query parameters. A handful of filters with a few values each can combine into an enormous number of URLs.
Sort and view parameters. ?sort=price_asc, ?view=grid, ?per_page=96 — the same items in a different order or layout.
Calendars and date archives. "Next month" links that go on forever, into years with no events. Google lists calendars with many date URLs among the common causes of sharp crawl increases.
Internal search results. Every query becomes a URL, and if search pages link to other searches the space never ends.
Session IDs and tracking parameters in internal links. Each visit or campaign creates a new URL for the same page.
Relative-link loops. A broken relative link (for example on a template that also serves 200 for unknown paths) can create ever-deeper URLs such as /shop/shop/shop/….
Soft 404s for nonsense URLs. If any made-up path or filter combination returns a normal page with status 200, crawlers have no signal that the URL should be abandoned.

How to find them

1. Look at what Google is actually fetching

Open the Crawl Stats report in Search Console and look at the example URLs under the breakdowns, especially By purpose → Discovery and By response → OK (200). Parameter-heavy URLs, repeated path segments, or dates far in the future or past are strong hints. The examples are a sample, not a complete list, so treat them as leads.

2. Analyse server logs

Logs are the most complete record of crawling. Filter to verified Google crawler requests — user-agent strings can be spoofed, so verify using reverse DNS or Google's published IP ranges as described in Google's documentation. Then group requested URLs by path pattern and by parameter name. Useful questions:

Which directories or parameters account for the most requests?
How many distinct URLs exist per template (for example per category page)?
Are there URLs with many parameters, repeated parameters, or parameters in different orders?
What share of crawled URLs are not in your sitemaps and are not canonical?

3. Crawl the site yourself

Run a desktop crawler with a sensible URL limit and watch for the crawl count exploding in one section. Sort the results by URL length, by number of query parameters, and by duplicate titles. A crawl that never "finishes" is itself a finding.

4. Compare against what you intend to be crawled

Your XML sitemaps should list the canonical URLs you care about. If the number of URLs being crawled or discovered is far larger than your sitemap inventory, find out where the difference comes from. The free Crawl Budget Snapshot gives a quick view of your sitemap URL count and robots.txt rules to compare against.

5. Test nonsense URLs

Request a URL that should not exist, such as a filter combination with no products or a page number far beyond the last page. If it returns 200 with a template page, you have a soft-404 pattern that can feed a trap.

How to fix them

Google's faceted navigation guide gives a clear decision: if you don't need the filtered URLs indexed, prevent crawling; if you do, make them as crawl-friendly as possible.

If the URLs don't need to be in Search

Disallow them in robots.txt. Keep individual item pages and an unfiltered listing page crawlable. Google's example pattern looks like this:
```
user-agent: Googlebot
disallow: /*?*products=
disallow: /*?*color=
disallow: /*?*size=
allow: /*?products=all$
```
Test patterns carefully — see the robots.txt checklist.
Use URL fragments for filters. Google Search generally doesn't use fragments (#color=green) for crawling and indexing, so filter state kept in the fragment has no crawl impact.

If some filtered URLs should be indexable

Use the standard & parameter separator, not commas, semicolons or brackets.
If filters are in the path, keep them in a fixed order and prevent duplicate filters.
Return a real 404 for filter combinations with no results, duplicate or nonsensical filters, and non-existent pagination pages — at that URL, not via a redirect to a generic error page.

Weaker signals

rel="canonical" pointing filtered pages at the unfiltered version may reduce crawling of the variants over time, and rel="nofollow" on filter links can help only if every link to those URLs carries it. Google describes both as generally less effective in the long term than robots.txt or fragments. A noindex tag does not stop crawling at all, because Google must fetch the page to see it.

Other traps

Calendars: stop generating "next"/"previous" links beyond the range that has content, and 404 empty dates.
Internal search: disallow the search results path in robots.txt.
Session and tracking parameters: remove them from internal links; keep state in cookies or analytics tooling instead.
Relative-link loops: fix the broken links and make unknown paths return 404.

After the fix

Changes take time to show. Watch the Crawl Stats report and your logs over the following weeks: requests to the trapped patterns should fall, and the share of crawling spent on real pages should rise. For background on why this matters, see crawl budget explained.

Check your own site

The free Crawl Budget Snapshot fetches a site's public robots.txt and sitemaps and gives a quick crawl-waste score with suggested fixes. It is a starting point, not a replacement for Search Console or log analysis.