Can AI assistants read your website?
robots.txt is a plain text file at the root of your website that tells crawlers which pages they may read. AI assistants send crawlers too, so its rules decide which of them may read your product pages. This free checker reads your site's robots.txt and shows, for 15 AI crawlers, whether your home, product, collection and content pages are allowed.
Descriptions checked 2026-10-13. We do not save the address or the result.
Results
| Crawler | What it does | Home | Products | Collections | Pages | Deciding rule |
|---|
Notes
What to fix
A change goes in your site's robots.txt, the plain text file in its root folder. If you do not manage that folder yourself, ask your developer or host.
On Shopify, the store serves your robots.txt from the theme template robots.txt.liquid, so a change goes there. In
the theme code editor, add the template if your theme does not have one yet, then remove the rule that blocks the
crawler or add a group that allows it, like this:
User-agent: OAI-SearchBot
Allow: /
A group that names a crawler replaces the general User-agent: * group for that crawler, so copy any
rule you still want it to follow, such as the cart and checkout paths, into the new group.
Shopify's own guide covers the details: Editing robots.txt.liquid. If robots.txt allows a crawler and it still cannot read your pages, check your CDN, firewall or bot-protection app.
How SourceVision helps
Letting AI crawlers in is one part. SourceVision measures how ChatGPT, Gemini and Perplexity describe your products and proposes fixes that you approve before anything changes on your store.
Request an inviteQuestions
What is robots.txt?
A plain text file at the root of a website, such as yoursite.com/robots.txt, that tells crawlers which paths they may read. It follows a public standard, RFC 9309. It is a request, not a lock: crawlers that follow the standard respect it, and a server, CDN or bot-protection app can still block a crawler whatever the file says.
Does blocking GPTBot remove my site from ChatGPT?
No. OpenAI says GPTBot collects content that may be used to train OpenAI's models, while OAI-SearchBot finds pages to show in ChatGPT's search answers. They are separate tokens, so a rule for GPTBot does not apply to OAI-SearchBot, and ChatGPT-User visits a page when a ChatGPT user's request needs it.
Does Google-Extended affect Google Search?
No. Google says Google-Extended is a robots.txt token, not a crawler: it controls whether content may be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI, and it does not affect Google Search. Googlebot crawls for Google Search, and Google says there is no separate control for AI Overviews and AI Mode.
Do all AI agents follow robots.txt?
Not all of them. OpenAI says robots.txt rules "may not apply" to ChatGPT-User, which visits a page when a ChatGPT user's request needs it, and Perplexity says Perplexity-User "generally ignores robots.txt rules". The operators of the other 13 crawlers on this page say they follow robots.txt.
Is this tool free?
Yes. There is no sign-up. The checker reads your site's robots.txt once and shows you the result. It does not save the address or the result; it counts only which platform the site runs on, when robots.txt shows it.
Guides on this topic
AI crawlers: which to allow on your store
Which AI crawlers exist, which ones decide whether your products can appear in AI answers, which train models, and how robots.txt applies to each.
How to check your AI visibility for free
A free, hands-on way to check your AI visibility: ask ChatGPT, Gemini and Perplexity your shoppers' questions, log who is named, and repeat the runs.
What is AI visibility? An ecommerce guide
AI visibility is how often and how accurately AI assistants recommend your products. What it means, how GEO and AEO fit, and how to measure it honestly.