What is Googlebot?

Googlebot is the name of Google’s web crawler, the automated program that fetches pages from the web so Google can index them and show them in search results. If Googlebot can’t reach and read a page, that page can’t rank on Google.

Googlebot doesn’t decide rankings. Its job is to collect. It requests URLs, downloads what the server sends back and passes it on to Google’s indexing systems, which render the page, work out what it’s about and decide whether to store it.

That makes Googlebot the first gate every page has to pass. Most technical SEO work comes down to making that visit easy: fast responses, clean status codes, links it can follow and content it can render.

How Googlebot works

Googlebot starts with URLs it already knows about and finds new ones by following links and reading sitemaps. Each URL goes into a queue, and Google’s systems decide what to fetch and how often.

1. Discovery. Googlebot finds URLs through links on pages it has already crawled, XML sitemaps and URLs submitted in Search Console.
2. Permission check. Before fetching, it reads the site’s robots.txt file to see which paths it’s allowed to request.
3. Fetching. It requests the page and records the status code, headers and HTML.
4. Rendering. Pages that rely on JavaScript are queued for rendering in an up-to-date version of Chromium, so Google can see content added by scripts.
5. Handing over. The fetched and rendered content goes to indexing, where Google decides whether the page is worth keeping.

Googlebot sets its own pace. Google says it aims to crawl as many pages as it can on each visit without overwhelming the server. If your server slows down or starts returning errors, Googlebot backs off. If it responds quickly, Googlebot may crawl more.

Googlebot Smartphone and Googlebot Desktop

There are two main versions of Googlebot. Googlebot Smartphone pretends to be a mobile device. Googlebot Desktop pretends to be a desktop computer. Both identify themselves with a user agent string that includes the word “Googlebot”.

Because Google uses mobile-first indexing, Googlebot Smartphone does almost all of the crawling for almost all sites. Google indexes and ranks the mobile version of your pages, so that’s the version that has to carry your full content, structured data and internal links. Googlebot Desktop still visits now and then, but you shouldn’t rely on it to see anything the mobile version hides.

Our guide to mobile-first indexing covers what that means for site builds and why trimmed-down mobile pages cause problems.

Other Google crawlers

Googlebot is the main crawler for Search, but Google runs others with their own user agents. Googlebot Image and Googlebot Video fetch media. Google-InspectionTool is used when you test a URL in Search Console. Google-Extended isn’t a separate crawler at all: it’s a robots.txt token that lets you control whether content Google crawls can be used to train and ground its Gemini models, and blocking it doesn’t affect your Search rankings.

Most sites never need to treat these differently. If you do write rules for them, check Google’s list of crawlers for the exact user agent tokens, because a typo means the rule is ignored.

How to verify that a visitor is really Googlebot

Anyone can put “Googlebot” in a user agent string, and plenty of scrapers do. If you see heavy traffic claiming to be Googlebot, check it before you block it or give it special treatment.

Google gives two ways to verify a request:

1. Reverse DNS lookup. Run a reverse DNS lookup on the IP address. A genuine Googlebot hostname ends in googlebot.com, google.com or googleusercontent.com. Then run a forward DNS lookup on that hostname and confirm it points back to the same IP.
2. IP range check. Google publishes the IP ranges its crawlers use as JSON files. Match the visitor’s IP against the Googlebot list.

Many CDNs and firewalls do this check for you and label verified bots. If yours does, make sure it isn’t blocking real Googlebot traffic by mistake. A firewall rule that blocks Googlebot can drop pages from the index within days.

How to see what Googlebot does on your site

The Crawl stats report in Search Console shows how many requests Googlebot made, how fast your server responded and which status codes it got back. The URL Inspection tool shows when a single page was last crawled and what Google saw when it rendered it.

For the full picture, look at your server logs. Log file analysis shows every request Googlebot made, so you can see which sections it visits often, which it ignores and where it hits errors. That’s how you spot crawl time going to filter pages or parameter URLs instead of the pages that earn money.

Making Googlebot’s job easier

• Keep server response times quick, especially for important templates.
• Return the right status codes, and fix redirect chains and soft 404s.
• Link to important pages from other pages with plain HTML links.
• Keep an up-to-date XML sitemap and point to it from robots.txt.
• Make sure key content and links appear in the rendered HTML, not only after a click or scroll. Our guide to JavaScript SEO explains the common traps.
• Don’t block CSS or JavaScript files Google needs to render the page.

If you think Googlebot isn’t seeing your site the way you need it to, our technical SEO agency can find out why. Book a free strategy session and we’ll go through it with you.

Related reading: our guides to robots.txt and indexability.

Common questions