Crawl budget is the number of URLs Google can and wants to crawl on your site. It only becomes a real problem on very large or fast-changing sites.

Crawl budget is the number of URLs Google can and wants to crawl on your site in a given period. For most sites it isn’t a problem at all. It starts to matter when a site has hundreds of thousands of URLs, changes a lot of pages every day, or wastes Googlebot’s time on duplicates and junk.
That last point is where most businesses go wrong. They either worry about crawl budget on a 300 page site that Google crawls easily, or ignore it on an ecommerce or listings site where filters have quietly created millions of URLs.
This guide covers how Google decides how much to crawl, how to tell whether you have a problem, and what to fix first.
How Google decides how much to crawl
Google sets crawl budget from two things: how much crawling your server can handle, and how much Google wants to crawl. Its own crawl budget documentation calls these the crawl capacity limit and crawl demand.
- Crawl capacity limit. Googlebot tries not to overload your server. If pages respond quickly, it crawls more. If responses slow down or return server errors (5xx) or “too many requests” (429), it backs off.
- Crawl demand. Google crawls more of what it thinks is worth crawling: popular pages, pages that change often, and URLs it believes exist on your site. Old, unpopular or duplicate URLs get less attention.
Put together, crawl budget is the set of URLs Google is able to crawl and wants to crawl. You can influence both sides.
When crawl budget matters
Crawl budget matters mainly for very large or fast-changing sites. Google’s guidance gives rough markers: sites with over a million unique pages that change about weekly, sites with over 10,000 pages that change daily, or sites where a large share of URLs sit in the “Discovered, currently not indexed” status in Search Console.
If your new pages are usually crawled the day you publish them, you don’t have a crawl budget problem. Spend your time on content and links instead.
Smaller sites can still waste crawl, though. The common culprits are:
- Faceted navigation and filters that create a new URL for every combination of size, colour and price.
- Tracking parameters, session IDs and sort orders added to internal links.
- Duplicate versions of pages (with and without trailing slashes, HTTP and HTTPS, upper and lower case).
- Long redirect chains and soft 404s, where a missing page returns a 200 status.
- Thin tag, archive and search results pages. Our guide to index bloat covers how these pile up.
How to check your crawl budget
You check crawl budget by comparing what Google crawls with what you want crawled. Three places tell you most of what you need.
- Crawl Stats in Search Console. Under Settings, the Crawl Stats report shows total requests, average response time and a breakdown by response code and file type. A rise in response time or server errors usually means Google will crawl less.
- The Page Indexing report. Large numbers of URLs in “Discovered, currently not indexed” suggest Google knows about pages but hasn’t got round to crawling them. We explain the status in our post on Discovered, currently not indexed.
- Server log files. Logs show every request Googlebot actually makes. They’re the only way to see whether it’s spending its time on your money pages or on parameter URLs. Our guide to log file analysis walks through it.
How to stop wasting crawl budget
You stop wasting crawl budget by giving Google fewer pointless URLs to find and making the useful ones fast to fetch. Google’s own recommendations line up with what we see on client sites:
- Consolidate duplicates. Pick one version of each page, redirect or canonicalise the rest, and link internally to the version you want.
- Block URLs that should never be crawled. Use robots.txt for internal search results, endless filter combinations and similar spaces. Don’t use it as a temporary switch to move budget around.
- Return proper status codes. Removed pages should return 404 or 410, not a 200 with “page not found” text.
- Keep sitemaps clean. List only canonical, indexable URLs and keep the lastmod date accurate.
- Shorten redirect chains. Point links straight at the final URL.
- Make pages fast to serve. Quicker server responses raise the crawl capacity limit.
One thing not to do: don’t rely on noindex to save crawl budget. Google still has to request the page to see the tag, so the crawl is spent either way.
Crawl budget and AI crawlers
Googlebot isn’t the only crawler on your server any more. AI crawlers from OpenAI, Anthropic, Perplexity and others now request pages too, and on some sites they generate a lot of traffic. They don’t share Google’s crawl budget, but they do share your server. If heavy bot traffic slows responses, Googlebot may back off.
The answer isn’t to block everything. Check your logs, make sure your server can cope, and decide crawler by crawler. We cover the trade-offs in should you block GPTBot and other AI crawlers.
Where to start
If you run a large ecommerce, marketplace, publishing or B2B catalogue site, start with Crawl Stats and a week of log files. If most of Googlebot’s requests go to parameters, filters or redirects, that’s your first fix.
Crawl problems are usually a symptom of wider technical issues, so they’re best fixed as part of a proper audit. Our technical SEO agency team runs crawl and log analysis as standard, and our B2B SEO work often starts here for firms with large product or parts catalogues. Book a free strategy session and we’ll tell you whether crawl budget is worth your time.
Related reading: our guides to creating an XML sitemap and canonical tags.
Common questions
Does my small website need to worry about crawl budget?
Usually not. If Google crawls your new pages within a day or so of publishing, crawl budget isn’t holding you back. It matters most for sites with hundreds of thousands of URLs or many pages that change daily.
Does noindex save crawl budget?
No. Google still has to crawl a page to see a noindex tag, so the request is used either way. Use robots.txt for areas that should never be crawled, and noindex for pages that can be crawled but shouldn’t appear in results.
Where can I see how often Google crawls my site?
The Crawl Stats report in Google Search Console, found under Settings, shows crawl requests, response times and status codes. Server log files give a fuller picture of every request Googlebot makes.
Can I ask Google to crawl my site more?
Not directly. Google crawls more when your server responds quickly and reliably and when your pages are popular and kept up to date. Removing duplicate and low-value URLs helps Google spend its time on the pages that matter.




