AI VisibilityFixer

Free robots.txt checker

AI Crawler Robots.txt Checker

Check what your robots.txt says for the AI crawlers and control tokens from OpenAI, Anthropic, Perplexity, Google, Apple, Amazon, Meta and others — grouped by what each one actually does: AI search, model training, data-use control, or user-triggered fetches.

Check My robots.txt

We fetch /robots.txt from the domain you enter and read the rules for each AI crawler and control token. Public pages only.

Not all AI bots do the same thing

“AI bot” covers at least five different behaviours. A robots.txt rule means something different for each, so this checker groups them — and only flags a blocked AI search crawler as notable.

AI search / discovery

Automated crawlers that discover and index pages so they can be surfaced, summarised, and linked inside an AI search or answer product.

If blocked: The site is asking not to be discovered or linked by that product. It does not affect Google Search or model training.

AI training

Crawlers that collect public content which may be used to train or improve a foundation model.

If blocked: A training-data / content-use policy choice. It does not add or remove the site from any AI search product.

Data-use control tokens

robots.txt tokens that do not fetch anything themselves. They only tell an operator whether content it has already crawled may be used for AI training or grounding.

If blocked: An opt-out of that operator's AI data use. The operators state it does not affect their search inclusion or ranking.

User-triggered fetchers

Fetch a single page because a person asked the assistant about it — not on a crawl schedule. Several operators say robots.txt may not apply to these requests.

If blocked: May or may not take effect, depending on the operator. These fetches do not determine what an AI search crawler discovers.

General / dataset crawlers

General-purpose web crawlers that are not tied to one AI product. Their datasets are used for research and, among other things, AI training.

If blocked: Removes the site from that dataset. It is not linked to any specific AI search product.

OpenAI: OAI-SearchBot vs GPTBot vs ChatGPT-User

These three OpenAI user-agents are frequently treated as interchangeable. They are not — and this is the single most common robots.txt misunderstanding for AI crawlers.

OAI-SearchBot

What it does
Discovers, fetches, and indexes pages so they can be surfaced, summarized, and linked in ChatGPT search.
Affects ChatGPT search?
Yes — this is the crawler tied to ChatGPT search discovery. If you want to be eligible to appear there, this user-agent must not be blocked.
Blocking it means
You are asking not to be discovered or linked by ChatGPT search.

GPTBot

What it does
Crawls public content that may be used to train and improve OpenAI's foundation models.
Affects ChatGPT search?
No. GPTBot is a training / model-improvement crawler. Allowing it is not a requirement for ChatGPT search, and blocking it does not remove a site from ChatGPT search.
Blocking it means
You are opting your content out of model training — a separate decision from search visibility.

ChatGPT-User

What it does
Fetches a specific page when a person (or a GPT Action) explicitly asks ChatGPT to open or use that URL.
Affects ChatGPT search?
No. ChatGPT-User is not a bulk crawler and is not used to decide what appears in ChatGPT search. OpenAI states robots.txt rules may not apply to these user-initiated requests.
Blocking it means
A Disallow may have no effect, per OpenAI. It does not change what ChatGPT search indexes.

The common misconception is that “to appear in ChatGPT search you just allow GPTBot.” That is not correct. GPTBot is about training data. ChatGPT search discovery is governed by OAI-SearchBot.

AI crawlers & tokens this checker covers

Every entry is based on the operator’s own documentation. The exact robots.txt token is shown in monospace.

AI search / discovery

  • OpenAI — ChatGPT searchOAI-SearchBot

    Surfaces websites in ChatGPT's search features. OpenAI: sites opted out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links.

  • Anthropic — Claude searchClaude-SearchBot

    Navigates the web to improve the quality and relevance of search results shown to Claude users. Anthropic states its crawlers honour robots.txt.

  • Perplexity — search indexPerplexityBot

    Surfaces and links websites in Perplexity's search results. Perplexity states it is not used to crawl content for AI foundation models.

  • Apple — ApplebotApplebot

    Apple's web crawler for Spotlight, Siri, and Safari suggestions. Apple states Applebot respects robots.txt directives targeted at Applebot.

  • Amazon — Alexa searchAmzn-SearchBot

    Improves search experiences in Amazon products such as Alexa. Amazon states it does not crawl content for generative AI model training and respects the Robots Exclusion Protocol.

  • You.com — search indexYouBot

    Crawls publicly accessible pages to provide real-time search results for You.com. You.com states YouBot fully respects robots.txt; it is not used for model training.

  • DuckDuckGo — AI-assisted answersDuckAssistBot

    Fetches pages in real time for DuckDuckGo's AI-assisted answers, which cite their sources. DuckDuckGo states the data is not used to train AI models and that opting out via robots.txt takes effect within 72 hours.

AI training

  • OpenAI — model trainingGPTBot

    Crawls content that may be used to train OpenAI's generative AI foundation models. OpenAI: disallowing GPTBot indicates a site's content should not be used for that.

  • Anthropic — model trainingClaudeBot

    Collects web content that could contribute to training and improving Anthropic's models. Anthropic states its crawlers honour robots.txt.

  • Meta — model trainingMeta-ExternalAgent

    Crawls the web for use cases such as training foundation AI models or improving products by indexing content directly. Meta documents robots.txt as the control file for it.

  • Amazon — general crawlerAmazonbot

    Amazon's general crawler used to improve products and services; Amazon states the content may also be used to train Amazon AI models. It respects the Robots Exclusion Protocol.

Data-use control tokens

  • Google — Gemini data-use controlGoogle-Extended

    A standalone robots.txt token. Google: it “doesn't have a separate HTTP request user agent string”; it only manages whether content Google already crawls may be used to train Gemini models and ground Gemini answers.

  • Apple — foundation-model data-use controlApplebot-Extended

    A robots.txt token that does not crawl webpages. Apple: it lets publishers opt out of content collected by Applebot being used to train Apple's foundation models.

User-triggered fetchers

  • OpenAI — ChatGPT user fetchChatGPT-User

    Fetches a page when a user (or a GPT Action) asks ChatGPT to open a specific URL. OpenAI: “not used for crawling the web in an automatic fashion” and “robots.txt rules may not apply” because the request is user-initiated.

  • Anthropic — Claude user fetchClaude-User

    Accesses a website when an individual asks Claude a question that needs it. Anthropic states its crawlers, including Claude-User, honour robots.txt directives.

  • Perplexity — user fetchPerplexity-User

    Visits a page to help answer a specific user question in Perplexity. Perplexity: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.”

  • Meta — user link fetchMeta-ExternalFetcher

    Fetches individual links at a user's request in Meta AI. Meta states this crawler may bypass robots.txt rules because the fetches are user-requested.

  • Amazon — Alexa user fetchAmzn-User

    Supports user actions such as responding to Alexa queries that need up-to-date information. Amazon: it may not follow all robots.txt directives, and does not crawl for generative AI training.

General / dataset crawlers

  • Common Crawl — dataset crawlerCCBot

    A general-purpose web crawler run by the non-profit Common Crawl. The resulting open dataset is widely used for research and, among other things, AI model training. CCBot honours robots.txt.

Investigated but not included

  • Bytespider (ByteDance) — No first-party publisher documentation. Independent server-log analyses report it crawling paths that robots.txt disallows, so a robots.txt rule for it cannot be relied on.
  • Timpibot (Timpi) — No first-party publisher documentation, and Timpi's decentralised node operators do not reliably honour robots.txt.
  • cohere-training-data-crawler / cohere-ai (Cohere) — Cohere publishes no first-party crawler / robots documentation page; the token spelling is attested only via third-party trackers.
  • OAI-AdsBot (OpenAI) — Only visits pages explicitly submitted as ads and does not use robots.txt for crawl control, so a robots.txt rule for it is not meaningful for normal publishers.
  • Google-CloudVertexBot (Google) — Fetches on demand for Vertex AI Agents built by Google Cloud customers — an enterprise integration, not a general publisher-facing AI search or training crawler.

Can I block AI training but allow AI search?

As a robots.txt policy, yes. The search and training crawlers are different user-agents, so you can allow one and disallow the other. A site that wants AI search discovery but not model training might express that like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

This is an example, not a recommendation. Whether to allow training crawlers, search crawlers, or both is a policy decision for each site owner. Blocking Google-Extended does not affect Google Search.

Need to build the rules? Use the AI Crawler Robots.txt Generator to choose Allow or Block per crawler — grouped by role — and copy a snippet to merge into your robots.txt, then re-run this checker on the deployed file.

Common robots.txt mistakes

1Treating every AI bot as the same kind of thing

A robots.txt rule for an AI search crawler (OAI-SearchBot, PerplexityBot) is a discoverability decision. A rule for a training crawler (GPTBot, ClaudeBot) is a content-use decision. Google-Extended and Applebot-Extended are not crawlers at all. Blocking one says nothing about the others.

2A blanket User-agent: * Disallow: / that blocks everything by accident

A catch-all rule left over from a staging site can block every crawler that has no group of its own — including the AI search crawlers. If a crawler has no specific group, the `User-agent: *` group is what applies to it.

3Assuming a Disallow stops a user-triggered fetch

ChatGPT-User, Perplexity-User, Meta-ExternalFetcher and Amzn-User fetch a page because a user asked. Their operators say robots.txt may not apply to those requests. Blocking them in robots.txt is often ineffective — and it never changes what an AI search crawler discovers.

4Using robots.txt as access control for private content

robots.txt is a request to well-behaved crawlers, not a security boundary. It does not authenticate anyone, does not stop non-compliant bots, and listing a path in Disallow can advertise it. Protect private content with authentication.

5Mistaking a CDN or firewall block for a robots.txt problem

A Cloudflare challenge, WAF rule, rate limit, or bot-management product can block a crawler even when robots.txt fully allows it. If robots.txt says NOT BLOCKED but a crawler still can't fetch pages, look at your CDN and server logs.

6Assuming a robots.txt allow guarantees crawling, indexing, or citation

Allowing a crawler only removes one obstacle. It does not guarantee that the operator fetches the site, includes it in an index, or cites it in an answer. Those outcomes depend on many factors outside robots.txt.

How to fix AI crawler access

  1. 1Decide your policy per role, on purpose

    AI search discovery, model training, data-use control, and user-triggered fetches are independent choices. Decide what you want for each before editing robots.txt.

  2. 2If you want to be eligible for AI search, don't block the search crawlers

    Make sure no group — neither a specific group nor the User-agent: * catch-all — disallows the paths you want discovered by OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, and the rest.

  3. 3Use explicit groups so intent is unambiguous

    Give each crawler you care about its own User-agent group with clear Allow / Disallow rules, rather than relying on the catch-all.

  4. 4Set the data-use control tokens deliberately

    Google-Extended and Applebot-Extended only govern whether already-crawled content trains Gemini / Apple models. The operators state they do not affect Google or Apple Search. Set them to match your content-use policy.

  5. 5Keep robots.txt at the origin root and served as plain text

    It must be reachable at https://yourdomain/robots.txt, return HTTP 200, and be text/plain — not an HTML page, not a redirect to another host.

  6. 6Re-check after the change is deployed

    robots.txt changes can take time to propagate (OpenAI notes roughly 24 hours). Re-run this checker on the live file and review server logs for the crawler user-agents you expect.

Disclaimer: This tool provides an automated readiness audit based on publicly visible technical, structured-data, and content signals. It is not a guarantee that any website will be ranked, recommended, cited, or mentioned by ChatGPT, Claude, Gemini, Perplexity, or any other AI system. AI models decide what to surface using many factors outside any website’s control, including factors that change over time and are not publicly documented.

Frequently asked questions

Which AI crawlers does this checker read?

It reads your robots.txt for every AI crawler and control token in our registry: OpenAI (OAI-SearchBot, GPTBot, ChatGPT-User), Anthropic (Claude-SearchBot, ClaudeBot, Claude-User), Perplexity (PerplexityBot, Perplexity-User), Google (Google-Extended), Apple (Applebot, Applebot-Extended), Amazon (Amzn-SearchBot, Amazonbot, Amzn-User), Meta (Meta-ExternalAgent, Meta-ExternalFetcher), You.com (YouBot), DuckDuckGo (DuckAssistBot), and Common Crawl (CCBot). Each is grouped by what it actually does.

What is the difference between an AI search crawler and a training crawler?

An AI search / discovery crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, YouBot, DuckAssistBot, Amzn-SearchBot) indexes pages so they can be surfaced and linked inside an AI answer product. A training crawler (GPTBot, ClaudeBot, Meta-ExternalAgent, Amazonbot) collects content that may train a foundation model. Blocking a training crawler is a content-use choice; it does not remove a site from any AI search product.

Is Google-Extended a crawler?

No. Google states Google-Extended "doesn't have a separate HTTP request user agent string" — it is a robots.txt control token only. It governs whether content Google already crawls may be used to train Gemini models and ground Gemini answers. Google states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

Is Applebot-Extended a crawler?

No. Apple states "Applebot-Extended does not crawl webpages." It is a control token that lets publishers opt content collected by Applebot out of training Apple's foundation models. Applebot itself — the crawler for Spotlight, Siri, and Safari — is separate.

Will blocking GPTBot remove my site from ChatGPT search?

No. ChatGPT search discovery is governed by OAI-SearchBot. GPTBot is a training crawler. You can disallow GPTBot while allowing OAI-SearchBot to express "do not train on my content, but do let ChatGPT search discover and link it." Whether your site then appears in ChatGPT search still depends on factors outside robots.txt.

Why is a blocked crawler sometimes shown as INFO instead of a problem?

Because blocking is not always a mistake. A blocked training crawler or data-use control token is usually a deliberate content-use policy. A blocked user-triggered fetcher often has no effect anyway. This checker only flags a blocked AI search crawler as notable, since that is the case where a site is unintentionally opting out of discovery.

Do user-triggered fetchers obey robots.txt?

It varies by operator. OpenAI (ChatGPT-User), Perplexity (Perplexity-User), Meta (Meta-ExternalFetcher) and Amazon (Amzn-User) state robots.txt may not apply to user-initiated fetches. Anthropic states Claude-User honours robots.txt. Either way, these fetches are per-request and do not determine what an AI search crawler indexes.

Does robots.txt protect private content?

No. robots.txt is a request to compliant crawlers, not an access-control or security mechanism. It does not authenticate requests, does not stop non-compliant bots, and naming a path under Disallow can even reveal it. Use authentication, not robots.txt, to protect private content.

Why does my robots.txt say not blocked but the crawler still cannot access my site?

robots.txt is only one layer. A CDN challenge, WAF rule, rate limit, IP block, geo-restriction, or bot-management product can stop a crawler at the edge even when robots.txt fully allows it. A soft 404, an offsite redirect on /robots.txt, or server errors can also make the real rules unreadable. Check your server and CDN logs for the specific crawler user-agent.