Log file analysis is the only way to see exactly what search engine bots requested on your site, rather than what a tool estimates they did.
Log file analysis means reading your server’s access logs to see exactly which URLs search engine bots requested, when, how often and what response they got. It is the only direct record of crawler behaviour on your site. Everything else, including Search Console, is a sample or a summary.
What a log line contains
Every request to your server writes a line. That line holds the IP address, timestamp, requested URL, HTTP status code, response size, referrer and user agent. Filter for Googlebot user agents, verify the IPs are genuinely Google’s, and you have a complete picture of how Google spent its time on your site.
The raw files live on your hosting, usually in a folder called logs or access_logs, and you may need to ask your host or developer to enable retention. Two to four weeks of data is normally enough to see patterns on a mid sized site.
What it tells you that nothing else does
Crawl budget becomes visible. On a large site Google allocates a finite number of requests, and logs show you where they go. We regularly find sites spending 40% or more of their crawl on faceted URLs, old paginated archives or parameter strings that should never have been crawlable, while new product pages wait weeks to be picked up.
Logs also expose orphaned pages Google still requests, redirect chains it keeps re crawling, 5xx errors that only appear under load, and pages that have not been touched by a bot in months. A page Google has not crawled since March is not going to reflect the changes you made in August.
Running an analysis
Get the files, verify the bot, then join the data against a crawl of your site so you can see which URLs are crawled, which are indexed and which are neither.
- Verify Googlebot properly. Run a reverse DNS lookup on the IP. Plenty of scrapers set the Googlebot user agent, and counting them ruins the numbers.
- Segment by page type. Products, categories, blog, filters. Crawl share by template is where the story usually is.
- Look at status codes by section. A section returning a high share of 301s or 404s to bots is burning requests for nothing.
- Compare crawl frequency against value. Your highest margin pages should be crawled most. Very often they are not.
This sits at the deeper end of technical SEO and tends to pay off most on sites above roughly 10,000 URLs.
Related reading: our guides to indexability, technical SEO, Googlebot, discovered currently not indexed, the URL Inspection API and Python for SEO.
Common questions
Do small sites need log file analysis?
Usually not. Under a few thousand URLs, Google will generally crawl everything it can reach and Search Console’s Crawl Stats report covers what you need. The exercise earns its keep on large ecommerce sites, publishers and anywhere crawling is clearly not keeping up with publishing.
What tools do I need?
Screaming Frog Log File Analyser handles most cases, and Splunk or BigQuery make sense at real scale. For a one off look, a spreadsheet will cope with a few hundred thousand lines. The tool matters far less than getting clean, verified data in the first place.
How much log data should I pull?
Start with 30 days. That is long enough to see weekly patterns and the effect of any recent releases, and short enough to stay manageable. If you are investigating a specific incident, pull the window around it instead and compare against a normal period.
Can I use Cloudflare or CDN logs instead?
Yes, and often they are better. A CDN sees requests before they reach your origin server, so it captures traffic your server logs miss entirely. Check that bot requests are being logged rather than served and discarded at the edge.
See our glossary entry for log file analysis.
