For most businesses that want customers to find them through ChatGPT, the answer is no: you should not block the AI crawlers that power search answers, and...

For most businesses that want customers to find them through ChatGPT, the answer is no: you should not block the AI crawlers that power search answers, and you can usually block GPTBot (the training crawler) without losing visibility. The key is knowing which bot does which job, because blocking the wrong one quietly removes you from AI answers.
Since 2023, the big AI companies have published their own crawlers. Some collect pages to train future models. Others fetch pages so an assistant can cite them in a live answer. A single robots.txt line can treat both the same, and plenty of sites have blocked their own AI visibility by copying a “block all AI bots” snippet from a forum.
This guide explains what each crawler does, what blocking it costs you, and a sensible default for UK businesses that want to be recommended in AI search.
What GPTBot actually does
GPTBot is OpenAI’s training crawler. According to OpenAI’s crawler documentation, it collects content that may be used to train its generative AI foundation models, and disallowing it tells OpenAI your content should not be used for that training.
Blocking GPTBot does not remove you from ChatGPT search. OpenAI treats each of its robots.txt settings as independent, so you can block training while still allowing the bot that surfaces sites in search answers.
The OpenAI crawlers and what each one controls
OpenAI runs three user agents that matter to site owners. Each has a different job, and only one of them decides whether you appear in ChatGPT’s search answers.
| User agent | What it does | What blocking it means |
|---|---|---|
| GPTBot | Crawls content that may be used to train OpenAI’s models | Your content is excluded from future training |
| OAI-SearchBot | Surfaces websites in ChatGPT’s search features | Your pages stop appearing in ChatGPT search answers, apart from navigational links |
| ChatGPT-User | Visits a page when a ChatGPT user asks for it | OpenAI says robots.txt rules may not apply, because a person started the request |
OpenAI also notes that robots.txt changes can take around 24 hours to take effect for search. So if you change your rules, don’t judge the result the same afternoon.
Other AI crawlers worth knowing
The same split between training and answering exists across the industry. The names change, but the question you ask is the same each time: does this bot train a model, or does it fetch pages for answers people see?
- Google-Extended is a control token rather than a separate crawler. It lets you opt out of your content being used for Google’s Gemini models. Google has said it does not affect inclusion or ranking in Google Search, and AI Overviews are part of Search, so they follow normal Googlebot rules.
- ClaudeBot and PerplexityBot are the crawlers Anthropic and Perplexity use. Check each company’s current documentation before writing rules, because the names and roles get updated.
- Applebot-Extended lets you opt out of Apple using your content for its AI models while normal Applebot still crawls for Siri and Spotlight.
Our guide to how web crawlers work covers the basics if these names are new to you.
When blocking AI crawlers makes sense
Blocking training crawlers makes sense when your content is the product. Publishers, paid research firms and course creators have a reasonable case for keeping their work out of model training, because the model could answer questions that would otherwise bring paying readers to them.
There are also practical reasons to restrict some bots:
- Server load. Aggressive crawling can slow a small site. Your log files will show which bots hit you hardest.
- Private or gated areas. Staging sites, client portals and member areas should be blocked from every crawler, AI or otherwise.
- Thin or duplicate sections. If a folder holds pages you would not want quoted, keep all crawlers out of it.
When blocking hurts you
For most service businesses, being cited in AI answers is now a route to new enquiries. Blocking the search crawlers removes that route. If a buyer asks ChatGPT for “a B2B accountant in Leeds” or “a technical SEO agency in the UK” and your site is blocked, the answer will name someone else.
Blocking training crawlers carries a softer cost. Models learn which brands are associated with which topics partly from what they were trained on. If you are trying to build that association, keeping your expertise out of training data works against you. It is a trade off, not a rule, and the right call depends on whether your content is what you sell or how you sell.
A sensible default for UK businesses
For a business that wants leads from AI search, this is the setup we use most often. It keeps you visible in answers while giving you a deliberate choice about training.
- Allow OAI-SearchBot, PerplexityBot and other answer crawlers across your public pages.
- Decide on GPTBot, Google-Extended and other training controls as a business decision, not a technical default. If your content is your product, block them. If your content is your marketing, allow them.
- Block every crawler from staging, admin, basket and account areas.
- Check your CDN or security plugin. Some firewalls block AI bots by default, which overrides whatever your robots.txt says.
- Review your logs a week later to confirm the bots you allowed are actually reaching your pages.
Here is a simple example that blocks training by OpenAI while keeping ChatGPT search access:
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: /
Our explainer on what robots.txt is covers the syntax in more detail, including how to test a rule before it goes live.
How to check what your site currently allows
Open yourdomain.com/robots.txt in a browser and look for any of the user agents above. Then check two other places that often get missed: your CDN’s bot settings and any security plugin on your CMS. A robots.txt that allows OAI-SearchBot does nothing if a firewall rule returns an error to it first.
If you want a fuller picture, our technical SEO team can audit your crawl rules, logs and firewall settings together, and our AI SEO service tracks whether those changes lead to more citations. You can also take the free AI visibility check to see where you stand today, or book a free strategy session and we’ll talk it through.
Related reading: our guides to meta robots tags and how to rank in AI Overviews.
Common questions
Does blocking GPTBot remove my site from ChatGPT search?
No. GPTBot is the training crawler. ChatGPT search uses OAI-SearchBot, and OpenAI treats each robots.txt setting separately, so you can block one and allow the other.
Does blocking Google-Extended affect my Google rankings?
No. Google has said Google-Extended does not affect inclusion or ranking in Google Search. It only controls whether your content is used for its Gemini models.
How long do robots.txt changes take to work for ChatGPT search?
OpenAI says it can take around 24 hours from a robots.txt update for its search systems to adjust.
Can a firewall block AI crawlers even if robots.txt allows them?
Yes. CDN and security plugin settings can block bots before they ever read robots.txt, so check those settings as well as the file itself.




