AI Crawler Access Test
See which AI and search crawlers your robots.txt allows for a page, and whether your server treats their user-agents differently from an ordinary browser.
The AI Crawler Access Test is a free tool that checks whether crawlers such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot are allowed by your robots.txt for a given page. It also sends each crawler's user-agent to your server and compares the answer with a normal browser, reporting honestly that real crawlers may still be treated differently.
Last updated:
What does the AI crawler access test check?
It answers two questions for the exact page you enter: what does your robots.txt tell each AI crawler to do, and does your server answer that crawler's user-agent differently from an ordinary browser?
- robots.txt decision: for each crawler token we apply the rules as RFC 9309 describes (most specific group, longest matching rule, Allow wins a tie) to the page's path.
- Server response: we request the page once per crawler user-agent and once as an ordinary browser, without following redirects, and compare status codes and challenge pages.
- Page directives: noindex, nosnippet and noai in the meta robots tag or the X-Robots-Tag header.
The result is a verdict per crawler: allowed, blocked by robots.txt, server answers differently, or inconclusive.
Which AI crawlers are there and what are they for?
Most companies run separate crawlers for training, search and user-requested fetches, so you can allow one purpose and block another. The table follows the vendors' own documentation.
| Crawler | Company | Purpose according to the vendor |
|---|---|---|
| GPTBot | OpenAI | Collects content that may be used to train generative AI models |
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search results |
| ChatGPT-User | OpenAI | Opens pages when a person asks ChatGPT; robots.txt rules may not apply |
| ClaudeBot | Anthropic | Collects content that may contribute to model training |
| Claude-SearchBot | Anthropic | Improves the quality of Claude's search results |
| Claude-User | Anthropic | Opens pages when a Claude user asks a question |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search; not used for foundation-model training |
| Perplexity-User | Perplexity | User-initiated visits; generally ignores robots.txt |
| Google-Extended | A robots.txt token with no separate crawler: controls use for Gemini training and grounding, and does not affect Google Search | |
| Applebot-Extended | Apple | A robots.txt token that does not crawl: controls use of Applebot data for training Apple's generative models |
The test also covers Applebot, Bingbot, Googlebot, Amazonbot, DuckAssistBot, MistralAI-User, meta-externalagent, meta-externalfetcher, CCBot and Bytespider.
How do I read the verdicts?
- Allowed: robots.txt permits the page and the server answered the crawler's user-agent the same way as a browser.
- Allowed by robots.txt, server not tested: no request was sent, either because the token does not crawl or because the crawler is verified by IP address (Googlebot, Bingbot).
- Blocked by robots.txt: a Disallow rule applies. Check which group it comes from.
- Server answers differently: a 403, 429, challenge page or no answer where the browser got the page. Look at your firewall, CDN bot protection or hosting security rules.
- Inconclusive: the comparison request itself failed, so nothing can be concluded about the server.
If search crawlers and user-requested fetchers are blocked, AI assistants cannot find or open the page through those channels.
How do I allow or block an AI crawler?
Add a group to robots.txt that names the crawler. A group that starts with User-agent: GPTBot and contains Disallow: / blocks that crawler from the whole site, while Allow: / lets it in. A named group replaces the User-agent: * rules for that crawler instead of adding to them.
- Allow search and user-request crawlers if you want to be found in AI answers.
- Decide on training crawlers separately. Blocking them is a content-use choice, not a visibility one.
- Do not rely on robots.txt for secrets. It is a request, and some user-triggered fetchers may ignore it.
The robots.txt Generator writes these groups for you, and the llms.txt Generator creates an optional summary file.
Why can a real crawler be treated differently from this test?
This test sends each crawler's user-agent string from our server's address. Real crawlers come from the vendors' own address ranges, and many firewalls and CDNs check those ranges instead of trusting the user-agent. The vendors publish IP lists for this reason.
- A firewall that checks IP addresses may block the real crawler while letting this test through, or the reverse.
- A rule that blocks by user-agent will usually show up here.
- For a firm answer, find the crawler's requests in your server logs and compare their IP addresses with the vendor's published list.
Sites behind a CDN or web application firewall, such as a bilingual .ae company site, should pay special attention to this point. See also our hosting and security service.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI documents separate crawlers: GPTBot for content that may be used to train its models and OAI-SearchBot for ChatGPT search, so you can disallow one and allow the other. ChatGPT-User, used when a person asks ChatGPT to open a page, is a third crawler, and OpenAI says robots.txt rules may not apply to it.
Does blocking Google-Extended affect Google Search?
No. Google describes Google-Extended as a robots.txt token without a separate user agent. It controls whether content Google has crawled may be used for Gemini training and grounding, and Google states that it does not affect a site's inclusion in Google Search. Search itself is crawled by Googlebot, which this test only checks in robots.txt.
Why does the test say my server answers differently?
For that crawler's user-agent your server returned a different status, such as 403 or 429, a challenge page or no answer, while an ordinary browser request got the page. It usually points to a firewall, a CDN bot rule or a hosting security setting. Real crawlers are often verified by IP address, so confirm in your server logs before changing rules.
Is robots.txt enough to keep AI systems away?
No. It is a request that well-behaved crawlers follow, not access control. The vendors document that some user-triggered fetchers, such as ChatGPT-User and Perplexity-User, may not follow it. Content that must stay private needs a login or other authentication, not a robots.txt rule.
What does the test send to my website?
One request for the page you entered with an ordinary browser user-agent, one for robots.txt, and one request per crawler user-agent: about twenty requests in all, sent from our server within a few seconds without following redirects. Nothing is submitted and no cookies are kept. Each connection can run a limited number of tests every few minutes.
Related pages
Sources
See a sample design for your own business
Browse fictional sample designs from hundreds of industries. Cannot see your type of business? Tell us what you do on WhatsApp and we will prepare a free sample design for you, with no obligation.
