Google has indexed a URL your robots.txt blocks it from crawling. Unblock it if you want it indexed, or unblock it and add noindex.
“Indexed, though blocked by robots.txt” means Google has added a URL to its index even though your robots.txt file tells it not to crawl that page. To fix it, decide whether you want the page in search results: if you do, remove the robots.txt rule blocking it; if you don’t, remove the block and add a noindex tag instead.
This status appears in the Page Indexing report in Google Search Console. It’s a warning rather than an error, and it often points to a rule that’s doing something different from what whoever wrote it intended.
The reason it happens catches a lot of people out. Robots.txt controls crawling, not indexing. If other pages link to a blocked URL, Google can still index the address without ever reading the page, which is why these results often show a bare URL with “No information is available for this page” underneath.
Why Google indexes pages it can’t crawl
Google indexes a blocked URL when it has enough signals from elsewhere, usually links from your own site or other sites, to think the page might be useful. It knows the page exists, but it can’t see what’s on it.
That has two side effects. The result in Google looks poor, with no proper title or description. And because Google can’t crawl the page, it can’t see a noindex tag on it either, so adding noindex to a blocked page does nothing. That’s the most common mistake we see with this status. Our guide to indexability explains the difference between crawling and indexing in more detail.
How to fix “Indexed, though blocked by robots.txt”
Work through the affected URLs in this order:
- Export the list. In Search Console, open Indexing, then Pages, and click the “Indexed, though blocked by robots.txt” row to see example URLs. Export them.
- Group them by pattern. Most affected URLs share a folder or a parameter, such as internal search results, filter pages or a /tag/ folder. You’ll usually be fixing a handful of rules, not hundreds of pages.
- Find the rule that blocks them. Use the URL Inspection tool on a few examples. It confirms that crawling is blocked, and the robots.txt report under Settings shows the file Google last fetched.
- Decide whether each group should be in Google. This is the step that matters. Be honest about whether these pages would help a searcher.
- Apply the right fix. See the table below.
- Validate the fix. Back in the Page Indexing report, click Validate fix so Google rechecks the URLs. It can take a few weeks to clear.
| You want the page | What to do |
|---|---|
| In search results | Remove or narrow the robots.txt rule so Google can crawl it, then request indexing in URL Inspection |
| Out of search results | Remove the robots.txt block, add a noindex meta tag or X-Robots-Tag header, and let Google recrawl it |
| Gone completely | Return a 404 or 410 status, or redirect it if there’s a relevant replacement |
| Private | Put it behind a login or password, because robots.txt doesn’t protect anything |
If you want a page out of the index and also want to stop crawlers visiting it, do it in two stages. Unblock it, let Google see the noindex and drop it, then add the robots.txt rule back. Doing both at once leaves you where you started.
Common causes of this warning
A handful of patterns cause most of these warnings on the sites we audit:
- Internal search results. Many sites block search URLs in robots.txt but link to them from search forms, tag clouds or old campaigns.
- Filter and sort parameters. Ecommerce sites often block parameter URLs to save crawl budget, while the filters themselves link to them on every category page.
- Old staging rules. A rule written for a development site gets copied to the live site and blocks a real folder.
- Over-broad wildcards. A rule meant for one path also matches pages you care about, because robots.txt matches the start of the URL path.
- Admin and system files. Some CMS paths are blocked on purpose and still linked publicly. These are usually safe to leave alone.
When you can ignore it
You can leave this warning alone if the blocked URLs are ones you’d never expect anyone to search for and they don’t appear for real queries. Check Search Console’s Performance report for impressions on those URLs. If there are none, the warning is mostly noise.
It’s worth acting when the URLs are pages you actually want to rank, when they’re showing for branded searches with an ugly blank snippet, or when the count is climbing. A rising number often means a template is generating new blocked links every day.
Our guide to robots.txt covers how to write rules that don’t overreach. Robots.txt and indexing checks are part of every SEO audit we run, and our technical SEO team can untangle the rules on your site if you book a free strategy session.
Related reading: our guides to running a site crawl and log file analysis.
Common questions
Is "Indexed, though blocked by robots.txt" an error?
No. It’s a warning. The page is in Google’s index, but Google couldn’t crawl it, so it may appear with no title or description.
Why doesn't my noindex tag work on a blocked page?
Google can’t crawl a page blocked by robots.txt, so it never sees the noindex tag. Remove the block first so Google can read the tag.
How long does the fix take to show in Search Console?
Usually a few weeks. Google has to recrawl each URL, and validation in the Page Indexing report runs through them gradually.
Should I just delete the robots.txt rule?
Only if you want the pages crawled. If the rule protects crawl budget from thousands of low-value URLs, it may be better to stop linking to those URLs instead.
