November 20, 2026 AI Search

Should You Block AI Crawlers? The robots.txt Call B2B SaaS Teams Get Wrong

Blocking Google-Extended won't remove you from AI Overviews, and blocking GPTBot won't stop ChatGPT citations. The AI crawler rules B2B SaaS teams get wrong.

Dark report cover titled Should You Block AI Crawlers, with an AI Search label on howardcreativeco.com

Blocking AI crawlers is a reasonable choice, but only if you block the right token for the outcome you want. Blocking Google-Extended does not remove you from Google's AI Overviews, and blocking GPTBot does not stop ChatGPT from citing your pages. Those are two different robots.txt tokens doing two different jobs, and most teams block the wrong one.

The short version

  • Google-Extended controls whether your content trains Gemini and grounds its answers. Google states plainly that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
  • Google's AI Overviews and AI Mode run on the regular Search index. The only way to keep pages out of them is to block Googlebot, which also removes you from every other part of Google Search.
  • OpenAI runs three separate crawlers: GPTBot for model training, OAI-SearchBot for surfacing and citing you in ChatGPT search, and ChatGPT-User for browsing a person triggers by hand.
  • Blocking a training crawler and blocking a citation crawler are opposite decisions. One keeps your content out of models; the other makes you invisible in the answer engines your buyers now use.

Why this is landing on Q4 planning docs right now

The "block the AI bots" line item is moving out of Slack threads and into actual robots.txt files, and it is usually written by someone who has not checked which token does what.

Every few weeks another analysis circulates counting how many sites have added artificial intelligence (AI) crawler blocks, and the direction only goes one way. The trouble is that teams treat robots.txt as a single on-off switch for "AI." It is not. Robots.txt is a list of named user agents, standardized as the Robots Exclusion Protocol (RFC 9309), and each AI vendor ships more than one agent with more than one job. Edit the file without reading the names and you will ship an outcome you did not intend.

When I audited a B2B SaaS client's robots.txt this fall, they had a Disallow for Google-Extended and felt covered. They were not. They were still fully trainable and citable by every answer engine they cared about, and the one block they had added protected nothing they were actually worried about. We rewrote it in about twenty minutes once we mapped each token to a decision.

Google-Extended: the token that does less than you think

Google-Extended is a training-and-grounding opt-out, not a Search control.

Per Google's crawler documentation, Google-Extended governs whether crawled content is used to train future Gemini models and to ground answers in Gemini Apps and Grounding with Google Search on Vertex AI. Google's wording is direct: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." Its AI features guidance adds that "Google-Extended is not a method for managing how your content appears in Google Search."

That last sentence is the one teams miss. AI Overviews and AI Mode are part of Google Search, not separate products. They are assembled from the same index Googlebot builds. So disallowing Google-Extended does nothing to your presence in an AI Overview. If you genuinely want out of AI Overviews, your only levers are to block Googlebot or use snippet controls like nosnippet, and blocking Googlebot deletes you from the ten blue links too. For almost every B2B SaaS company, that trade is a non-starter.

OpenAI: three crawlers, three decisions

OpenAI does not have "a bot." It has three, and they do not overlap.

  • GPTBot is the training crawler. Disallowing it keeps your content out of OpenAI's foundation-model training. It does not affect whether ChatGPT can cite you when it searches the live web.
  • OAI-SearchBot is the search crawler. It is how your pages get surfaced and linked inside ChatGPT's search results, and OpenAI says it is not used to train its models. Block it and you opt out of being a cited source in one of the referral channels your buyers increasingly use.
  • ChatGPT-User covers actions a person starts inside ChatGPT, such as asking it to open a specific URL. OpenAI's own documentation notes that because these are user-initiated, robots.txt rules may not apply, so a disallow will not reliably stop them.

The practical trap: a team that wants to protect its content from training adds a Disallow for GPTBot and assumes it is now invisible to ChatGPT. It is not. OAI-SearchBot still crawls, ChatGPT still cites, and a user who pastes your URL still gets the page fetched. The opposite mistake is just as common. A team that wants to appear in ChatGPT answers blocks everything with "OpenAI" in the name and quietly removes itself from the citation channel it was trying to win.

A decision framework that fits in one file

The whole thing resolves once you separate two questions and answer them per vendor.

Question one: do you want your content used to train models? Question two: do you want to be cited in AI answers and AI search? Those are independent, so map them directly:

  • Want citations, fine with training: allow everything. This is the default for most B2B SaaS sites that run on inbound.
  • Want citations, not fine with training: allow the search crawlers (Googlebot, OAI-SearchBot) and disallow the training crawlers (Google-Extended, GPTBot). You stay visible and opt out of the training corpus.
  • Want neither: understand that you are choosing to disappear from AI search. Be sure that is a real decision and not a reflex, because the competitors who stay in will own the citations you gave up.

A note on the other names you will see. Anthropic's crawler is ClaudeBot, Common Crawl's is CCBot (its archive feeds many downstream AI datasets), and Perplexity ships PerplexityBot alongside a user-triggered fetcher. Treat each one the same way: find the token, decide training versus citation, and write the rule. Do not copy a blocklist from a listicle, because the tokens change and a stale list blocks crawlers that no longer exist while missing the ones that matter.

What this does not cover

This is about crawl control through robots.txt, not content licensing or litigation. Robots.txt is a public, voluntary standard. A well-behaved crawler honors it, but it is not an access-control mechanism and not every fetcher respects it, which is exactly why user-triggered bots slip through. If your goal is legal protection for proprietary content, that is a contracts-and-counsel conversation, not a robots.txt edit. This also does not cover server-level blocking, meaning firewall or web application firewall (WAF) rules by user agent or internet protocol (IP) address, which is the heavier tool for a crawler that ignores the file.

And it does not cover llms.txt, which guides AI systems toward your best content rather than blocking them. I wrote about that separately.

Frequently asked questions

Does blocking Google-Extended remove my site from AI Overviews?

No. Google states that Google-Extended does not affect inclusion or ranking in Google Search, and AI Overviews and AI Mode are part of Search. To keep pages out of AI Overviews you would have to block Googlebot, which removes you from all of Google Search.

Will blocking GPTBot stop ChatGPT from citing my pages?

No. GPTBot is OpenAI's training crawler. Citations in ChatGPT's search come through OAI-SearchBot, a separate token. Blocking GPTBot opts you out of training while leaving you citable in ChatGPT search.

Can robots.txt stop every AI bot from reading my site?

No. Robots.txt is a voluntary standard (RFC 9309) that well-behaved crawlers honor. User-triggered fetchers such as ChatGPT-User may ignore it, and a crawler that does not respect robots.txt needs a server-level block instead.

Should a B2B SaaS company block AI crawlers at all?

Usually not the search and citation crawlers. If your pipeline depends on being found, opting out of AI search hands those citations to competitors. Blocking training crawlers while allowing citation crawlers is the middle path most inbound-driven teams want.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot crawls content that may train OpenAI's foundation models. OAI-SearchBot crawls to surface and link your site inside ChatGPT's search results and is not used for training. They are controlled by separate robots.txt tokens, so you can allow one and block the other.

One next step

If you want me to audit your robots.txt and map every AI crawler token to a yes-or-no decision for your site, book a 20-minute read-out and I will send you the rewrite.