How to write robots.txt for AI crawlers
Short answer
Robots.txt for AI crawlers is a set of groups in the robots.txt file at your site's root, each naming a crawler and listing the paths it may not crawl. A crawler follows only the group that names it, or the * group if none does. Name a crawler only to handle it differently, repeat your private paths in its group, then test the live file.
By Ronen Amit, Founder, SourceVision. Published .
On this page
Writing robots.txt for AI crawlers takes a few lines of text, and one wrong line can let a crawler into your checkout or keep an assistant away from your products. This guide shows how the file works, how to write groups for search and answer crawlers and for training crawlers, where to edit it on Shopify, WooCommerce, Wix and other platforms, and how to test the result.
If you first want to know which AI crawlers exist and which ones affect your visibility in AI answers, read our guide to AI crawlers. This one is about writing the file.
What is robots.txt?
Robots.txt is a plain text file at the root of your site that tells crawlers which paths they may not crawl. It holds groups: each group names one or more crawlers on User-agent lines, then lists Allow and Disallow rules. Anything no rule blocks is allowed.
The standard is RFC 9309, the Robots Exclusion Protocol. It defines a rule as one line, such as Disallow: /checkout, and a group as one or more User-agent lines followed by rules. A group ends at the next User-agent line or at the end of the file.
A few format rules decide whether a crawler can read your file at all:
- One file, at the root. The file must be named robots.txt and sit at the top of the host, such as yourstore.com/robots.txt. Google’s creation guide says a site can have only one, and crawlers ignore copies in subfolders.
- One host at a time. Rules apply only to the host, protocol and port where the file is served. A store on shop.example.com needs its own file, separate from example.com.
- Plain text in UTF-8. Write it in a text editor, not a word processor, which can add characters crawlers do not expect.
- Paths are case sensitive.
Disallow: /Checkoutdoes not block /checkout. Field names such as User-agent are not case sensitive.
Google supports four fields: user-agent, allow, disallow and sitemap. Its specification page says other fields, such as crawl-delay, are not supported, and it ignores anything after 500 kibibytes.
How does a crawler choose its group in robots.txt?
A crawler looks for the group whose User-agent line matches its own name, ignoring case. If several groups name it, their rules are combined. If none names it, it follows the User-agent: * group. A group that names a crawler replaces the * rules for that crawler completely.
That last point is where most AI crawler edits go wrong. RFC 9309 says a crawler obeys the * group only when no group matches its name. Google’s specification says the same in plain words: only one group is valid for a crawler, and specific groups and the global group are not combined.
So this file lets OAI-SearchBot crawl your checkout:
User-agent: *
Disallow: /checkout
Disallow: /cart
User-agent: OAI-SearchBot
Allow: /
OAI-SearchBot finds its own group and never reads the * rules. If you name a crawler, copy every path you keep private into its group too. If you only want a crawler to follow your normal rules, do not name it at all.
Inside a group, the most specific rule wins: the one whose path matches the most characters of the URL. If an Allow and a Disallow rule match equally, the Allow rule is used. Two special characters help: * matches any run of characters and $ marks the end of a URL. Shopify’s default file uses Disallow: /collections/*sort_by* to keep sorted collection pages out.
Step through the whole process as a crawler runs it:
How a crawler reads your robots.txt, step by step
-
1 1. Fetch the file
Before crawling a host, the crawler requests /robots.txt at its root. It should follow at least five redirects. The file applies only to that host, protocol and port.
-
2 2. Read the response
A successful response means it follows the rules it can parse. A 4xx error such as 404 means it may crawl anything. A 5xx error or network failure means it must assume the whole site is disallowed.
-
3 3. Find its group
It looks for User-agent lines that match its name, ignoring case, and merges every group that names it. If none does, it uses the User-agent: * group. If there is no * group either, no rules apply.
-
4 4. Match the URL
It compares the URL path with each Allow and Disallow rule in its group. The rule matching the most characters wins. If an Allow and a Disallow rule tie, the Allow rule is used. No match means the URL is allowed.
-
5 5. Cache the result
It may keep the file for up to 24 hours before fetching it again, or longer if the file cannot be reached. Google says it generally caches robots.txt for up to 24 hours.
Step 1 of 5
Each step comes from RFC 9309, except the host, protocol and port rule in step 1, which is from Google’s creation guide. Step 5 also draws on Google’s specification page. Google adds detail for errors: after a 5xx it stops crawling for 12 hours, then uses its last good copy for up to 30 days.
How do you block AI crawlers in robots.txt?
Add a group that names each crawler or token you want to block and gives it Disallow: /. You can put several User-agent lines above one set of rules. Use the exact token each company publishes, and leave unnamed every crawler that should follow your normal rules.
Each AI company publishes its own tokens and says what each one is for. Here is what their pages say about the ones a store meets most:
- OpenAI. OAI-SearchBot surfaces websites in ChatGPT’s search features. GPTBot crawls content that may be used to train OpenAI’s foundation models. OpenAI says each setting is independent of the other.
- Anthropic. Claude-SearchBot indexes content to improve search results for Claude users. ClaudeBot collects content that could contribute to model training. Anthropic’s help article asks you to add the rules for every subdomain you want to opt out.
- Perplexity. PerplexityBot surfaces and links websites in Perplexity’s results. Perplexity says it is not used to crawl content for AI foundation models.
- Google. Google-Extended is a token, not a separate crawler. Its crawler list says it controls whether content may be used to train future Gemini models and to ground answers in Gemini Apps and on Vertex AI. It does not affect inclusion in Google Search.
- Apple. Applebot-Extended is also a token that does not crawl. Apple says disallowing it opts your content out of training Apple’s foundation models, and pages that disallow it can still appear in search results.
- Amazon. Amazonbot may be used to train Amazon AI models. Amzn-SearchBot makes content eligible for search experiences such as Alexa and is not used for model training.
Whether to block any of them is your decision. The makers describe what each token controls, and the choice depends on how you weigh model training against reach.
| Company | Search or answers | Model training |
|---|---|---|
| OpenAI | OAI-SearchBot surfaces websites in ChatGPT search | GPTBot crawls content that may train foundation models |
| Anthropic | Claude-SearchBot indexes content for Claude search results | ClaudeBot collects content that could contribute to training |
| Perplexity | PerplexityBot surfaces and links websites in results | PerplexityBot is not used for foundation model training |
| Google-Extended does not affect inclusion in Google Search | Google-Extended, a token for Gemini training and grounding | |
| Apple | Pages that disallow Applebot-Extended can still appear in search | Applebot-Extended, a token that does not crawl |
| Amazon | Amzn-SearchBot, for search experiences such as Alexa | Amazonbot may be used to train Amazon AI models |
Each company's own pages say what its tokens control, and whether to block any of them is your decision.
A group that blocks the four training tokens above, while every other crawler follows your normal rules, looks like this:
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
Sitemap: https://yourstore.com/sitemap.xml
A group that keeps one answer crawler out of a section, without letting it into your private paths, repeats those paths:
User-agent: PerplexityBot
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /wholesale/
Two details are easy to miss. Amazon says that if your file does not mention Amzn-SearchBot but allows other search bots, it follows the rules you gave those bots. And blocking by IP address is not a substitute: Anthropic says it may not give a reliable opt-out, because it keeps its bots from reading your robots.txt.
When you know which crawlers to name, our free generator writes the file for you. It keeps every other rule as written, replaces any lines that already name the AI crawlers it manages, adds a group only for each crawler you block, and checks the result rule by rule before you copy it:
Can robots.txt stop AI crawlers from reading your store?
Only crawlers that choose to obey it. RFC 9309 says its rules are not a form of access authorization, and the major AI companies tell site owners to use it to manage their crawlers. It does not hide pages from search results, protect private content or stop fetchers acting for one user.
Four limits matter for a store:
- It asks, it does not enforce. Google’s introduction says respectable crawlers obey robots.txt but others might not. Shopify’s robots.txt help calls the rules directional and advisory.
- It controls crawling, not indexing. A page blocked in robots.txt can still appear in Google results, without a description, if other pages link to it. To keep a page out of results, Google says to use noindex or password protection.
- It is public. RFC 9309 warns that listing a path in robots.txt makes it discoverable. Never list a private URL you want nobody to find, such as a hidden staff discount page.
- User-triggered fetchers may skip it. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person asked for the visit. Perplexity says Perplexity-User generally ignores them. Our crawlers guide covers these agents.
On Shopify there is one more layer. Shopify says product data reaches the agentic storefronts you have turned on, such as ChatGPT, through Shopify Catalog, independently of robots.txt. Blocking a crawler there affects open web discovery only. Shopify also handles bot management at the network layer and does not recommend a proxy in front of your store.
Robots.txt decides who may read your pages. It does not tell you whether an assistant then recommends your products. That is what SourceVision measures: we report how often your products appear in assistants’ answers, with the range across runs, and propose fixes from facts you confirm, applied only with your approval. See how it works.
SourceVision is in an invite-only beta.
How do you edit robots.txt in Shopify?
Add a robots.txt.liquid template to your published theme: in your Shopify admin, open Online Store, choose Edit code on the theme, add a new template and select robots. Keep the Liquid that prints Shopify’s default rules, and add your AI crawler groups below it.
Every Shopify store has a default robots.txt, and Shopify says it works for most stores. Adding the template replaces that generated file, so start by saving a copy of your current yourstore.com/robots.txt to compare against.
Shopify’s customization guide recommends keeping the Liquid objects that output the default groups and adding custom rules, such as blocking a crawler, outside that loop. The shape looks like this, but copy the default block straight from Shopify’s template page so you paste exactly what Shopify publishes:
{% for group in robots.default_groups %}
{{- group.user_agent }}
{%- for rule in group.rules -%}
{{ rule }}
{%- endfor -%}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{% endfor %}
User-agent: GPTBot
Disallow: /
Shopify’s notes on the template and in its help center are worth reading before you save:
- It is an unsupported customization. Shopify Support cannot help with edits, and Shopify warns that incorrect use can cost you all traffic.
- The template must be robots.txt.liquid, not a JSON template, and the rendered output may not match the generated file exactly.
- You can replace everything with plain text rules, but Shopify strongly recommends keeping its Liquid objects, because plain rules can go out of date.
- Changes are instant, though crawlers do not react at once. Uploading a theme through the admin does not import robots.txt.liquid.
- To go back to the current default, delete the template.
WooCommerce and WordPress
WooCommerce stores run on WordPress, which builds its robots.txt output itself. A developer can change that output through the robots_txt filter. If you are not sure how your site serves the file, ask your developer or host before replacing it.
Wix
Wix has a Robots.txt Editor. Its help page puts it under SEO & GEO, then Tools and settings, then Robots.txt Editor and View File. You can reset the file to its default there, and Wix Customer Care cannot help with your changes.
Other platforms
Google says hosted platforms may not let you edit robots.txt directly and may offer a search settings page instead. Where you can upload files, save the file as robots.txt in UTF-8 at the root of your site. If you cannot reach the root, ask your developer or host.
How do you test robots.txt?
Open yourstore.com/robots.txt in a private browser window to confirm the live file is the one you wrote. Then check what each crawler may read, look for parsing errors in Search Console’s robots.txt report, and allow about a day for crawlers to pick up the change.
Work through these checks after every edit:
- See the live file. Google suggests opening it in a private window. You can also download it with
curl https://yourstore.com/robots.txt -o robots.txt. - Check each crawler. Our AI crawler checker reads your live robots.txt and shows, crawler by crawler, whether your home, product, collection and content pages are allowed, and which rule decides it.
- Read Search Console. The robots.txt report shows the files Google found, when it fetched them, and any errors or warnings. Errors stop a rule from being used. It works for domain-level properties only.
- Refresh Google’s copy if it is urgent. Google’s update guide says it refreshes its cached file every 24 hours. Request a recrawl in the report when you have unblocked important pages or fixed an error.
- Give the AI crawlers time. OpenAI says its search systems can take about 24 hours to adjust after you update robots.txt. Perplexity says up to 24 hours. Amazon says about 24 hours, and its crawlers may use a cached copy from the last 30 days.
Make the robots.txt check part of your routine: our 8-step guide shows where crawler access fits with structured data, product questions and feeds.
Frequently asked questions
Is robots.txt mandatory?
No. Without a robots.txt file, crawlers assume there are no restrictions: Google says every file is implicitly allowed unless your file says otherwise. Shopify, Wix and WordPress each serve a default file, so most stores already have one even if nobody wrote it.
How to fix blocked by robots.txt error?
First decide whether the page should be crawled. If it should, remove or narrow the rule that blocks it, then use Request a recrawl in Search Console's robots.txt report. Shopify says the 'Indexed, though blocked by robots.txt' message is normal for pages its default rules block on purpose, such as cart and checkout pages.
Why are there so many bots on my Shopify store?
Search engines, AI companies and many other services crawl public stores. Shopify says it handles bot management at the network layer for stores on its platform, with no action needed from you. Robots.txt can ask named crawlers to stay out of parts of your store, but it is advisory.
Where is robots.txt located?
At the root of each host, for example yourstore.com/robots.txt. Its rules apply only to that host, protocol and port, so a subdomain such as shop.example.com needs its own file. Crawlers do not look for robots.txt files in subfolders.
Which pages should robots.txt rules cover?
Pages that are unique to each shopper or that duplicate other pages: cart, checkout, account and order pages, plus filtered and sorted collection URLs. That is what Shopify's default file disallows. Product, collection, content and policy pages stay open to the crawlers you want reading them.
Related guides
GTIN, UPC and barcodes for online sellers
What GTIN, UPC and EAN codes are, where to get them, which ones Shopify, Google, ChatGPT and Amazon ask for, and what to do when a product has none.
Product descriptions for AI shopping
How to write product descriptions that shoppers, search engines and AI assistants read correctly: structure, facts with units, feeds and what to leave out.
Product titles for AI shopping
Write product titles for AI shopping, Google Shopping and marketplaces: what to include, in what order, each channel's length limit and what to leave out.
Sources
- RFC 9309: Robots Exclusion Protocol, IETF, accessed .
- How Google interprets the robots.txt specification, Google for Developers, accessed .
- How to write and submit a robots.txt file, Google for Developers, accessed .
- Update your robots.txt file, Google for Developers, accessed .
- Introduction to robots.txt, Google Search Central, accessed .
- Google's common crawlers, Google for Developers, accessed .
- robots.txt report, Search Console Help, accessed .
- Overview of OpenAI crawlers, OpenAI, accessed .
- Does Anthropic crawl data from the web, and how can site owners block the crawler?, Claude Help Center, accessed .
- Perplexity crawlers, Perplexity, accessed .
- About Applebot, Apple Support, accessed .
- Amazonbot, Amazon Developer, accessed .
- robots.txt.liquid, Shopify Dev Docs, accessed .
- Customize robots.txt, Shopify Dev Docs, accessed .
- Editing robots.txt.liquid, Shopify Help Center, accessed .
- robots_txt hook, WordPress Developer Resources, accessed .
- Installing and uninstalling WooCommerce, WooCommerce, accessed .
- Editing your site's robots.txt file, Wix Help Center, accessed .