Robots.txt generator for AI crawlers
For any online store: choose which AI search, shopping and training crawlers may read your site, and get a robots.txt you can paste, checked before you copy it.
Check what your current robots.txt allows with the AI crawler checker.
Choose the crawlers
Every crawler is allowed until you change it. The file is written and checked in your browser, and nothing you choose or paste is sent to us.
This tool never changes Googlebot and Bingbot: they crawl for Google and Bing search, and Google says Googlebot's rules are also the control for its AI features in Search.
Your robots.txt
Only the AI crawler rules, for a Shopify template
Checked before you copy
We read the file above the way the AI crawler checker reads a live one, rule by rule under RFC 9309, for your home page and an example product, collection and content page.
| Crawler | You chose | This file | Notes |
|---|
Where the file goes
- Shopify: Your store builds robots.txt from the theme template robots.txt.liquid. Save a copy of your current yourstore.com/robots.txt first. In the theme code editor, add the template to the Templates folder if your theme does not have one, keep the Liquid that outputs Shopify's default rules, and paste only the AI crawler rules below it. Shopify notes that the rendered file may not match its own exactly. (Shopify: Customize robots.txt)
- WordPress and WooCommerce: WordPress serves a robots.txt it builds itself, and a developer can change it with the robots_txt filter. If you are not sure how your site serves robots.txt, ask your developer or host before you replace it. (WordPress: robots_txt filter)
- Any other site: Save the file as robots.txt, in UTF-8, at the root of your site so it opens at yoursite.com/robots.txt. A site has one robots.txt. If you cannot upload to the root, ask your developer or host. (Google: Create a robots.txt file)
How the file is written
- A crawler you block gets a group of its own: its name on a
User-agentline, thenDisallow: /. -
A crawler you allow gets no group of its own, so it follows your
User-agent: *rules like every other crawler. Under RFC 9309 a group that names a crawler replaces the*rules for it, so an allow group would also let it into paths your file keeps every crawler out of, such as the cart or checkout. - If you paste your current file, every line that is not about these crawlers stays as you wrote it, in the same order: comments, blank lines, Sitemap lines and the rules for every other crawler. A group that names one of these crawlers next to another one is split, and the other crawler keeps its rules.
- A comment line names this generator and the date, so you can find the AI crawler rules later.
How SourceVision helps
Choosing who may read your pages is one part. SourceVision measures how ChatGPT, Gemini and Perplexity describe your products and proposes changes to your pages that you approve before anything changes on your store.
Request an inviteQuestions
What does robots.txt do, and what does it not do?
It asks crawlers which paths of your site they may read. Crawlers that follow the standard, RFC 9309, do as it says. It is a request, not a lock: the standard says its rules are not a form of access authorization, and an agent that does not follow it can still read a public page. Google adds that a page it may not crawl can still appear in its results without a description. To keep a page private, put it behind a password or a login.
Should I block AI training crawlers?
That is your decision, and this tool does not recommend either choice. Blocking a training crawler or token asks its owner not to use your pages to train its models. Allowing it does not. Each switch above shows the owner's own description of what the crawler or token covers, with a link to its page, so you can decide crawler by crawler.
Does blocking a training crawler keep me out of AI search?
Each owner answers this for its own crawlers. OpenAI says GPTBot is for training, OAI-SearchBot is for search in ChatGPT, and each setting is independent of the others. Anthropic lists ClaudeBot for training and Claude-SearchBot for search, and says blocking Claude-SearchBot may reduce your visibility in user search results. Google says Google-Extended does not affect inclusion in Google Search. Apple says pages that disallow Applebot-Extended can still be included in search results. Perplexity says PerplexityBot is for search and is not used to train AI foundation models. Amazon describes one crawler, Amazonbot, used to improve its products and services, whose content may be used to train Amazon's AI models.
Where does the file go on my platform?
At the root of your site, so it opens at yoursite.com/robots.txt. On Shopify it comes from the theme template robots.txt.liquid: keep the Liquid that outputs Shopify's default rules and add the AI crawler rules below it. WordPress builds its own robots.txt, and a developer can change it with the robots_txt filter. On any other site, upload the file as robots.txt to the root folder, or ask your developer or host.
How do I check it works?
Open yoursite.com/robots.txt and make sure it shows your new rules. Then run the AI crawler checker: it reads the live file the same way this tool checked it before you copied it, and shows which pages each crawler may read.
More step-by-step help is in our guides.
Guides on this topic
AI crawlers: which to allow on your store
Which AI crawlers exist, which ones decide whether your products can appear in AI answers, which train models, and how robots.txt applies to each.
Optimize your store for AI search: 8 steps
How to optimize an online store for AI search on any platform, in 8 steps: crawler access, llms.txt, product data, policies, answers and measuring.
What is llms.txt, and does it work?
What llms.txt is, what Google and the AI assistants have said, what your Shopify store already serves, and whether to edit it.