Robots.txt is a plain text file at the root of a domain that tells crawlers which parts of the site they may request. It controls crawling, not indexing.
Robots.txt is a plain text file at the root of a domain that tells search engine crawlers which parts of the site they are allowed to request. It lives at yourdomain.com/robots.txt and nowhere else. Well behaved crawlers read it before they fetch anything.
The important thing to understand first: it controls crawling, not indexing. Those are different jobs and confusing them is the most common way sites break themselves with this file.
What the file looks like
A typical file is short:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /*?sort= Sitemap: https://example.com/sitemap_index.xml
User-agent names the crawler the rules below it apply to, and an asterisk means all of them. Disallow blocks a path prefix. Allow carves an exception out of a Disallow. Sitemap points at your XML sitemap and applies regardless of user agent.
Google supports two wildcards: an asterisk matches any run of characters, and a dollar sign anchors a rule to the end of a URL, so Disallow: /*.pdf$ blocks PDFs. Where two rules conflict, the most specific one wins, and if they are the same length, Allow beats Disallow. Google ignores crawl-delay entirely, and has ignored noindex directives in robots.txt since September 2019.
Blocking a page does not remove it from Google
A disallowed URL can still show up in search results. If other pages link to it, Google knows the URL exists and can list it with a note saying no information is available for this page. It has not crawled the page, so it cannot see a noindex tag even if you put one there.
That is the trap. To keep a page out of the index you have to let Google crawl it and serve a noindex meta robots tag or X-Robots-Tag header. Blocking in robots.txt and adding noindex at the same time cancels itself out: the directive is on a page Google is not allowed to read.
What is worth blocking
- Infinite URL spaces. Faceted navigation, sort parameters, calendar archives and internal search results pages. These can generate millions of near identical URLs and eat crawl capacity that should go to product and category pages.
- Admin and checkout paths. Cart, account and login URLs have nothing to offer a searcher.
- Staging and test hosts, though a password is safer than a Disallow rule.
Leave your CSS, JavaScript and image files alone. Google needs them to render the page, and blocking them makes your site look broken to the crawler. That was a widespread problem a decade ago and it still turns up in old WordPress files.
Mistakes that cost traffic
Disallow: / shipped from a staging environment is the classic, and it can take weeks to recover from. After that: paths are case sensitive, so /Products/ and /products/ are separate rules. The file only applies to its exact host, protocol and port, so a file on www.example.com does nothing for shop.example.com. And it must return a 200 status. If robots.txt returns a server error, Google stops crawling the site for a while rather than assuming everything is allowed.
Check the file after every migration and every platform change. It is one line of text and it can switch a site off. If you want someone to check yours, that is part of what a technical SEO agency does in the first week of any engagement.
Related reading: our guides to indexability, log file analysis, meta robots tags, getting Google to index your pages, fixing Indexed, though blocked by robots.txt, URL parameters and canonical tags.
Common questions
Do I need a robots.txt file?
Not strictly. If the file is missing, crawlers treat the whole site as crawlable, which is usually fine for a small site. It is still worth having one so you can point at your sitemap and block obvious junk paths later without a developer involved.
Can I block AI crawlers with it?
Yes, for the ones that honour it. GPTBot, ClaudeBot, PerplexityBot and Google-Extended all respect robots.txt and can be blocked by name. Think carefully before you do: blocking them removes you from answers those tools generate.
How do I test my rules?
Google Search Console has a robots.txt report showing the version Google last fetched and any parse errors. For individual URLs, the URL Inspection tool tells you whether the live URL is blocked and by which rule.
Is there a size limit?
Google reads the first 500KiB and ignores anything past it. If your file is anywhere near that you have a URL structure problem, not a robots.txt problem.
If you want the one line version to hand, there is an entry for robots.txt in our SEO glossary.
