Skip to content

Free tools

Robots.txt generator for AI crawlers

For any online store: choose which AI search, shopping and training crawlers may read your site, and get a robots.txt you can paste, checked before you copy it.

Check what your current robots.txt allows with the AI crawler checker.

Choose the crawlers

Every crawler is allowed until you change it. The file is written and checked in your browser, and nothing you choose or paste is sent to us.

This tool never changes Googlebot and Bingbot: they crawl for Google and Bing search, and Google says Googlebot's rules are also the control for its AI features in Search.

AI search and answers

Crawlers that find pages for an assistant's search answers.

Visits a person asks for

Agents that open a page when someone using an assistant asks a question that needs it.

  • ChatGPT-User (OpenAI)

    Visits a page when a ChatGPT user's request needs it. It does not crawl on its own. OpenAI's page about ChatGPT-User

    OpenAI says robots.txt rules "may not apply" for this agent.

  • Perplexity-User (Perplexity)

    Fetches a page when a Perplexity user asks for it. It is not used for crawling or training. Perplexity's page about Perplexity-User

    Perplexity says robots.txt rules "generally ignores robots.txt rules" for this agent.

  • Claude-User (Anthropic)

    Visits a page when a Claude user asks a question that needs it. Anthropic's page about Claude-User

Model training and datasets

Crawlers and tokens about using pages to train AI models or to build a public dataset.

Start from your current robots.txt (optional)

Open yoursite.com/robots.txt, copy all of it and paste it here. We keep every line that is not about the crawlers above, in the same order, and replace or add only their groups. Up to 500 KiB.

For example https://yourstore.com/sitemap.xml. A Sitemap line already in your file is kept as it is.

How the file is written

  • A crawler you block gets a group of its own: its name on a User-agent line, then Disallow: /.
  • A crawler you allow gets no group of its own, so it follows your User-agent: * rules like every other crawler. Under RFC 9309 a group that names a crawler replaces the * rules for it, so an allow group would also let it into paths your file keeps every crawler out of, such as the cart or checkout.
  • If you paste your current file, every line that is not about these crawlers stays as you wrote it, in the same order: comments, blank lines, Sitemap lines and the rules for every other crawler. A group that names one of these crawlers next to another one is split, and the other crawler keeps its rules.
  • A comment line names this generator and the date, so you can find the AI crawler rules later.

How SourceVision helps

Choosing who may read your pages is one part. SourceVision measures how ChatGPT, Gemini and Perplexity describe your products and proposes changes to your pages that you approve before anything changes on your store.

Request an invite

Questions

What does robots.txt do, and what does it not do?

It asks crawlers which paths of your site they may read. Crawlers that follow the standard, RFC 9309, do as it says. It is a request, not a lock: the standard says its rules are not a form of access authorization, and an agent that does not follow it can still read a public page. Google adds that a page it may not crawl can still appear in its results without a description. To keep a page private, put it behind a password or a login.

Should I block AI training crawlers?

That is your decision, and this tool does not recommend either choice. Blocking a training crawler or token asks its owner not to use your pages to train its models. Allowing it does not. Each switch above shows the owner's own description of what the crawler or token covers, with a link to its page, so you can decide crawler by crawler.

Does blocking a training crawler keep me out of AI search?

Each owner answers this for its own crawlers. OpenAI says GPTBot is for training, OAI-SearchBot is for search in ChatGPT, and each setting is independent of the others. Anthropic lists ClaudeBot for training and Claude-SearchBot for search, and says blocking Claude-SearchBot may reduce your visibility in user search results. Google says Google-Extended does not affect inclusion in Google Search. Apple says pages that disallow Applebot-Extended can still be included in search results. Perplexity says PerplexityBot is for search and is not used to train AI foundation models. Amazon describes one crawler, Amazonbot, used to improve its products and services, whose content may be used to train Amazon's AI models.

Where does the file go on my platform?

At the root of your site, so it opens at yoursite.com/robots.txt. On Shopify it comes from the theme template robots.txt.liquid: keep the Liquid that outputs Shopify's default rules and add the AI crawler rules below it. WordPress builds its own robots.txt, and a developer can change it with the robots_txt filter. On any other site, upload the file as robots.txt to the root folder, or ask your developer or host.

How do I check it works?

Open yoursite.com/robots.txt and make sure it shows your new rules. Then run the AI crawler checker: it reads the live file the same way this tool checked it before you copied it, and shows which pages each crawler may read.

More step-by-step help is in our guides.

Guides on this topic

The beta is invite only for now.

Free during the founding beta. Founding partners get 50% off list price after the beta.