What are meta robots tags?

Meta robots tags are snippets of HTML in a page’s head that tell search engines how to handle that page, such as whether to index it, whether to follow its links and what to show in the search snippet. The most common one is <meta name="robots" content="noindex">, which keeps a page out of Google’s results.

They work page by page. That makes them different from robots.txt, which controls which URLs crawlers may request across a whole site. A meta robots tag only works if the crawler is allowed to fetch the page and read the tag.

Used well, they keep thin, private or duplicate pages out of search. Used carelessly, a single stray noindex can remove a key page, or a whole site, from Google within days.

What a meta robots tag looks like

The tag sits inside the <head> of the page:

<meta name="robots" content="noindex, nofollow">

The name says which crawlers the rule is for. “robots” means all search engines. You can target one crawler by name instead, for example name="googlebot". The content holds one or more rules separated by commas.

If a page has no meta robots tag, search engines treat it as index, follow. You never need to add those two values: they’re the default.

The main meta robots rules

Rule	What it does
noindex	Keeps the page out of search results
nofollow	Tells Google not to follow the links on the page
none	Same as noindex, nofollow
nosnippet	Stops Google showing a text or video snippet, and stops the content being used in AI Overviews
max-snippet:[number]	Limits the snippet to a set number of characters
max-image-preview:[setting]	Sets the largest image preview (none, standard or large)
noimageindex	Stops images on the page being indexed
notranslate	Stops Google offering a translation of the page
unavailable_after:[date]	Removes the page from results after a set date

Google used to support noarchive for its cached copies, but it retired the cache in 2024 and no longer uses that rule. Leaving it in place does no harm.

When to use noindex

Noindex is the right tool when a page needs to exist for users but has no place in search results. Typical examples:

• Thank-you and order confirmation pages.
• Internal search results pages.
• Thin tag or author archives.
• Login, account and checkout pages.
• Campaign landing pages that duplicate other content.

Don’t use noindex on pages you want to rank, and don’t combine it with a canonical tag pointing to another URL. The two send opposite signals. If a page is a duplicate, the canonical alone is usually the better choice.

What nofollow does on a page

A page-level nofollow tells Google not to follow any link on the page. Google treats it as a hint rather than a strict rule, and it’s rarely useful. It stops internal links passing signals through the page, which usually hurts more than it helps. If you want to mark individual links, use rel="nofollow", rel="sponsored" or rel="ugc" on those links instead.

The X-Robots-Tag HTTP header

The X-Robots-Tag does the same job as the meta robots tag, but it’s sent as an HTTP header from the server rather than written into the HTML:

X-Robots-Tag: noindex

It supports the same rules. Its advantage is that it works on files that don’t have an HTML head, such as PDFs, images and Word documents. It’s also useful for applying a rule to a whole folder or file type through server configuration, rather than editing pages one by one.

The X-Robots-Tag is easy to miss because you can’t see it in the page source. If a page won’t index and the HTML looks fine, check the response headers in your browser’s developer tools or the URL Inspection tool in Search Console.

Meta robots vs robots.txt

This is where most mistakes happen. Robots.txt stops Google crawling a URL. A meta robots noindex stops Google indexing it. If you block a page in robots.txt, Google can’t fetch it, so it never sees the noindex tag. The page can then stay indexed (often with no description) if other sites link to it.

So the rule is simple: to remove a page from search, allow crawling and use noindex. Only block it in robots.txt once it has dropped out of the index, and only if you also want to save crawl time. Our guide to indexability covers how these signals fit together.

How to check your meta robots tags

1. Use the URL Inspection tool in Search Console to see whether Google found a noindex on a page.
2. Check the Page indexing report for the reason “Excluded by ‘noindex’ tag” and review every URL listed.
3. Crawl the site and filter by indexability to find pages with noindex or X-Robots-Tag rules.
4. After every site launch or migration, check that the staging noindex has been removed. Leaving it in place is one of the most common causes of a site vanishing from Google.

Our technical SEO agency checks robots directives on every page as part of our audits. Book a free strategy session if you’d like us to look at yours.

Related reading: our guides to SEO audits and how to do a site crawl.

Common questions