AI crawlers: which to allow on your store

AI companies now send crawlers across the web, and a lot of advice says to block them all. For a merchant, that can be a costly mistake. Some AI crawlers train models. Others fetch your pages so ChatGPT, Claude, Perplexity or Meta AI can answer a shopper’s question and link to you. This guide lists the main AI crawlers in their makers’ own words, explains which ones decide whether you can appear in AI answers, and shows how robots.txt applies to each, whichever platform your store runs on.

What are AI crawlers?

AI crawlers are bots run by AI companies that read web pages. They do three different jobs: collecting content that may train AI models, indexing pages so an assistant can find and cite them in answers, and fetching a page when one user asks an assistant to.

The split matters because the three jobs have different value to a store:

1. Training crawlers collect content that may be used to build future models. Blocking them affects what a model learns, not whether today’s assistant can find your store.
2. Search and answer crawlers index pages so an assistant can retrieve them when a shopper asks a question. Blocking them can take you out of those answers.
3. User-triggered fetchers visit a page because one person asked an assistant to read it. Several makers say these may not follow robots.txt, because a person asked for the visit.

Most large AI companies now run a separate bot for each job, each with its own name, so you can allow one and block another. A few still use one crawler for more than one purpose, which is why reading each maker’s own description matters.

What are examples of AI bots?

The main ones for a store are OpenAI’s GPTBot, OAI-SearchBot and ChatGPT-User, Google’s Googlebot and Google-Extended, Anthropic’s ClaudeBot and Claude-SearchBot, PerplexityBot, Applebot, Amazon’s Amazonbot and Amzn-SearchBot, Meta’s crawlers and Common Crawl’s CCBot. Each maker documents what its bot does.

Here is what each maker says its crawlers do, as of October 15, 2026:

Maker	Search and answers	Training	Acts for one user
OpenAI	OAI-SearchBot	GPTBot	ChatGPT-User
Google	Googlebot (Search, AI Overviews, AI Mode)	Google-Extended (a control token for Gemini training and Gemini Apps grounding)	Google-Agent
Anthropic	Claude-SearchBot	ClaudeBot	Claude-User
Perplexity	PerplexityBot	Not used for model training, Perplexity says	Perplexity-User
Apple	Applebot (Spotlight, Siri, Safari)	Applebot-Extended (a control token)
Amazon	Amzn-SearchBot (search experiences such as Alexa)	Amazonbot (may train Amazon AI models)	Amzn-User
Meta	Meta-WebIndexer (Meta AI search)	Meta-ExternalAgent	Meta-ExternalFetcher
Common Crawl		CCBot (an open web archive, which Common Crawl does not describe as training)

Sources: OpenAI, Google, Anthropic, Perplexity, Apple, Amazon, Meta and Common Crawl.

Two rows need a note. Google uses Googlebot for AI Overviews and AI Mode, and Google-Extended is a token, not a crawler, that controls whether Google may use your content to train Gemini models and to ground answers in Gemini Apps. Apple’s Applebot powers search in Spotlight, Siri and Safari, and Applebot-Extended only controls whether Apple may use that content to train its models. Common Crawl describes CCBot as building an open archive of web data. It does not say what others use the archive for.

Microsoft is missing from the table because we could not open Bing’s crawler page for this guide. Bing says it respects robots.txt preferences, and its Webmaster Tools now report how content appears in Microsoft Copilot.

Which AI crawlers decide whether you appear in AI answers?

The search and answer crawlers. For ChatGPT that is OAI-SearchBot, for Claude it is Claude-SearchBot, for Perplexity it is PerplexityBot, and for Google AI Overviews and AI Mode it is plain Googlebot. Blocking a training crawler such as GPTBot or ClaudeBot does not remove you from those answers.

The makers say this plainly:

• OpenAI. Sites that opt out of OAI-SearchBot “will not be shown in ChatGPT search answers,” though they can still appear as navigational links. OpenAI recommends allowing it if you want to appear.
• Google. AI Overviews and AI Mode are built into Search, so robots.txt rules for Googlebot are the control, its AI features page says. Google-Extended “does not impact a site’s inclusion in Google Search.”
• Anthropic. Disabling Claude-SearchBot prevents Anthropic from indexing your content for search.
• Perplexity. PerplexityBot surfaces and links websites in Perplexity’s results, and is not used to train foundation models.
• Apple. Pages that disallow Applebot-Extended can still be included in search results.

The crawling data shows why the difference matters. Cloudflare measured what AI crawlers on its network were doing, by purpose, a year apart:

What AI crawling on Cloudflare's network was for, by purpose

• July 2024
• July 2025

Source: Cloudflare, The crawl-to-click gap (August 29, 2025)

Show data

What AI crawling on Cloudflare's network was for, by purpose

Purpose of the crawl	July 2024 (%)	July 2025 (%)
Training	72	79
Search	26	17
User actions	2	3.2

Most AI crawling is for training, which returns no visits to your store. A smaller share is the crawling that can put your products in front of a shopper. A blanket block shuts out both. A targeted rule can tell them apart.

Does robots.txt block AI crawlers?

It asks them not to crawl, and the major AI companies say their automated crawlers follow it. It is not a lock. The robots.txt standard says its rules are not access authorization, and several makers say their user-triggered fetchers may not apply robots.txt, because a person asked for the page.

The standard, RFC 9309, defines rules “that crawlers are requested to honor.” It adds: “These rules are not a form of access authorization.”

For automated crawling, the makers commit to robots.txt. Google says its common crawlers always obey robots.txt when crawling automatically. Anthropic says its bots honor industry standard directives. Amazon says its listed crawlers respect the Robots Exclusion Protocol.

User-triggered fetchers are the exception. OpenAI says robots.txt rules may not apply to ChatGPT-User. Google says its user-triggered fetchers generally ignore them. Perplexity, Amazon and Meta say the same of Perplexity-User, Amzn-User and Meta-ExternalFetcher.

Changes are not instant either. OpenAI says its search systems can take about 24 hours to adjust after you update robots.txt. Perplexity and Meta give similar figures, and Amazon says it may use a cached copy of robots.txt for up to 30 days.

Should I block AI crawlers?

Block training crawlers if you do not want your content used to train models. Allow the search and answer crawlers if you want your products recommended in AI answers. For most stores that means allowing OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot, and deciding separately on GPTBot, ClaudeBot and Google-Extended.

A store sells better when shoppers can find it, so the default for a merchant differs from the default for a news publisher. Weigh three questions:

1. Do you want to appear in AI answers? If yes, the search crawlers must be allowed on your product, collection and content pages.
2. Do you mind your copy being used for training? That is a business and rights decision. By the makers’ own descriptions, blocking GPTBot or ClaudeBot does not cost you answer visibility. Google-Extended is different: Google says it also controls whether Gemini Apps may use your pages to ground their answers.
3. Is any crawler costing you server load? Cloudflare found large gaps between how often AI companies crawl and how often they send a visitor back. In July 2025 it counted about 1,091 OpenAI crawls per referral and about 195 for Perplexity, against about 5 for Google. If load is the problem, a rate limit at your host or CDN may serve you better than a block.

If you sell on Shopify, one more fact helps: Shopify’s robots.txt help says that if you sell through agentic storefronts such as ChatGPT, your product data reaches them through Shopify Catalog, independently of robots.txt. Blocking crawlers there affects open web discovery only. Our ChatGPT recommendations guide covers the catalog and feed side.

Being reachable is only the first step. Whether an assistant then recommends your products is something you measure. We built SourceVision for that: we ask ChatGPT, Gemini and Perplexity the questions your shoppers ask, many times over, and report how often your products appear, with the range across runs. Then we draft what each product page is missing from facts you confirm, and nothing is written without your approval. See how it works.

SourceVision is in an invite-only beta.

Request an invite

How do you block or allow AI crawlers on your store?

Check what your robots.txt allows, then add a group with Allow or Disallow rules for each crawler you want to handle differently. Where you edit it depends on your platform: a theme template on Shopify, a filter or SEO plugin on WordPress, or the file at your site’s root elsewhere.

Start by checking what your store allows now. Our free checker reads your robots.txt and shows, crawler by crawler, whether your home, product, collection and content pages are allowed, and which rule decides it:

AI crawler access checker

Check whether 15 AI crawlers, from GPTBot to PerplexityBot, may read your site's pages.

Your website's address

For example yoursite.com.

Check my site

A quick check confirms you are a person, not a bot.

Descriptions checked 2026-10-13. We do not save the address or the result.

Open the full tool

A rule set that blocks training crawlers and keeps the main answer crawlers in looks like this. Drop the Google-Extended group if you want Gemini Apps to keep using your pages to ground their answers:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /
Disallow: /checkout
Disallow: /cart

User-agent: PerplexityBot
Allow: /
Disallow: /checkout
Disallow: /cart

A crawler follows the group that names it, so keep your existing rules for checkout, cart and account pages in each group you add. Where to make the change:

• Shopify. Every store has a default robots.txt. To change it, add a robots.txt.liquid template to your theme. Shopify calls this an unsupported customization and warns that incorrect use can cost you all of your traffic, so edit carefully.
• WordPress and WooCommerce. WordPress generates a virtual robots.txt that a site can change through the robots_txt filter, which is how some SEO plugins edit it. A physical robots.txt file at the site root replaces the virtual one, and some plugins edit that file instead.
• Other platforms. Most let you edit robots.txt in the admin or upload the file to the site root. Check your platform’s help pages before you change it.

After any change, run the checker again, and allow a day or more before you expect AI companies to act on it. If you want to know what AI visibility is and how to measure it once the door is open, start with our guide to what AI visibility is.