Free robots.txt tool

AI Crawler Robots.txt Generator

Choose Allow or Block for each documented AI crawler and control token — from OpenAI, Anthropic, Perplexity, Google, Apple, Amazon, Meta, You.com, DuckDuckGo and Common Crawl — grouped by role (AI search, model training, data-use control), and copy a standard robots.txt snippet to merge into your file. Runs entirely in your browser.

Build robots.txt Rules

What this generator does

It writes standard robots.txt groups — one User-agent line and one Allow: / or Disallow: / line per crawler you set. Crawlers you leave at “Not set” are left out entirely, so the snippet only contains decisions you made on purpose.

The user-agent tokens are the exact strings from each provider's own documentation.
Output order is fixed, so the same choices always produce the same text.
It generates a snippet to merge into your existing robots.txt — it does not produce a full replacement file, and it never edits your site.
It does not fetch anything. Everything is computed locally from your selections.

AI crawler reference

Every entry is based on the operator’s own current documentation, grouped by what it does. “If blocked” is what a Disallow actually changes — not a promise about visibility.

AI search / discovery

Automated crawlers that discover and index pages so they can be surfaced, summarised, and linked inside an AI search or answer product.

OpenAIOAI-SearchBot
Surfaces websites in ChatGPT's search features. OpenAI: sites opted out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links.
If blocked: You are asking to be left out of ChatGPT search discovery. It does not change training (GPTBot) or Google Search.
robots.txt: operator says it is honored
AnthropicClaude-SearchBot
Navigates the web to improve the quality and relevance of search results shown to Claude users. Anthropic states its crawlers honour robots.txt.
If blocked: Claude's search feature is asked not to use the page. Separate from ClaudeBot (training).
robots.txt: operator says it is honored
PerplexityPerplexityBot
Surfaces and links websites in Perplexity's search results. Perplexity states it is not used to crawl content for AI foundation models.
If blocked: You are asking not to be indexed or linked by Perplexity's search.
robots.txt: operator says it is honored
AppleApplebot
Apple's web crawler for Spotlight, Siri, and Safari suggestions. Apple states Applebot respects robots.txt directives targeted at Applebot.
If blocked: You are asking to be left out of Apple's search features. Apple foundation-model training is a separate token (Applebot-Extended).
robots.txt: operator says it is honored
AmazonAmzn-SearchBot
Improves search experiences in Amazon products such as Alexa. Amazon states it does not crawl content for generative AI model training and respects the Robots Exclusion Protocol.
If blocked: Your content is asked not to appear in Amazon search experiences such as Alexa.
robots.txt: operator says it is honored
You.comYouBot
Crawls publicly accessible pages to provide real-time search results for You.com. You.com states YouBot fully respects robots.txt; it is not used for model training.
If blocked: You are asking not to be indexed by You.com's AI search.
robots.txt: operator says it is honored
DuckDuckGoDuckAssistBot
Fetches pages in real time for DuckDuckGo's AI-assisted answers, which cite their sources. DuckDuckGo states the data is not used to train AI models and that opting out via robots.txt takes effect within 72 hours.
If blocked: Your pages are asked not to be used as a source for DuckDuckGo's AI-assisted answers. DuckDuckGo states this does not affect organic search rankings.
robots.txt: operator says it is honored

AI training

Crawlers that collect public content which may be used to train or improve a foundation model.

OpenAIGPTBot
Crawls content that may be used to train OpenAI's generative AI foundation models. OpenAI: disallowing GPTBot indicates a site's content should not be used for that.
If blocked: You are opting your content out of OpenAI foundation-model training. It is NOT required for ChatGPT search and does not remove a site from it.
robots.txt: operator says it is honored
AnthropicClaudeBot
Collects web content that could contribute to training and improving Anthropic's models. Anthropic states its crawlers honour robots.txt.
If blocked: You are opting your content out of Anthropic model training. Separate from Claude-SearchBot (Claude's search feature).
robots.txt: operator says it is honored
MetaMeta-ExternalAgent
Crawls the web for use cases such as training foundation AI models or improving products by indexing content directly. Meta documents robots.txt as the control file for it.
If blocked: You are opting your content out of Meta's AI training / indexing crawl.
robots.txt: operator says it is honored
AmazonAmazonbot
Amazon's general crawler used to improve products and services; Amazon states the content may also be used to train Amazon AI models. It respects the Robots Exclusion Protocol.
If blocked: You are asking Amazon not to crawl the site for product improvement or AI training. (Amazon search for Alexa is a separate token, Amzn-SearchBot.)
robots.txt: operator says it is honored

Data-use control tokens

robots.txt tokens that do not fetch anything themselves. They only tell an operator whether content it has already crawled may be used for AI training or grounding.

GoogleGoogle-Extended
A standalone robots.txt token. Google: it “doesn't have a separate HTTP request user agent string”; it only manages whether content Google already crawls may be used to train Gemini models and ground Gemini answers.
If blocked: Content Google already crawls is opted out of Gemini training and grounding. Google states it “does not impact a site's inclusion in Google Search nor is it used as a ranking signal.”
robots.txt: control token only — does not fetch pages
AppleApplebot-Extended
A robots.txt token that does not crawl webpages. Apple: it lets publishers opt out of content collected by Applebot being used to train Apple's foundation models.
If blocked: Content Applebot has collected is opted out of Apple foundation-model training. It does not remove the site from Apple's search features.
robots.txt: control token only — does not fetch pages

User-triggered fetchers

Fetch a single page because a person asked the assistant about it — not on a crawl schedule. Several operators say robots.txt may not apply to these requests.

OpenAIChatGPT-User
Fetches a page when a user (or a GPT Action) asks ChatGPT to open a specific URL. OpenAI: “not used for crawling the web in an automatic fashion” and “robots.txt rules may not apply” because the request is user-initiated.
If blocked: A Disallow may have no effect, per OpenAI. It is not used to decide what appears in ChatGPT search.
robots.txt: operator says it may not apply (user-triggered)
AnthropicClaude-User
Accesses a website when an individual asks Claude a question that needs it. Anthropic states its crawlers, including Claude-User, honour robots.txt directives.
If blocked: A Disallow for Claude-User is honoured by Anthropic, but this is a per-request user fetch, not the Claude search crawler (Claude-SearchBot).
robots.txt: operator says it is honored
PerplexityPerplexity-User
Visits a page to help answer a specific user question in Perplexity. Perplexity: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.”
If blocked: A Disallow generally has no effect, per Perplexity. It is separate from PerplexityBot, the search index crawler.
robots.txt: operator says it may not apply (user-triggered)
MetaMeta-ExternalFetcher
Fetches individual links at a user's request in Meta AI. Meta states this crawler may bypass robots.txt rules because the fetches are user-requested.
If blocked: A Disallow may be bypassed, per Meta. Separate from Meta-ExternalAgent (training / indexing).
robots.txt: operator says it may not apply (user-triggered)
AmazonAmzn-User
Supports user actions such as responding to Alexa queries that need up-to-date information. Amazon: it may not follow all robots.txt directives, and does not crawl for generative AI training.
If blocked: A Disallow may not be followed, per Amazon. Separate from Amzn-SearchBot (Alexa search index).
robots.txt: operator says it may not apply (user-triggered)

General / dataset crawlers

General-purpose web crawlers that are not tied to one AI product. Their datasets are used for research and, among other things, AI training.

Common CrawlCCBot
A general-purpose web crawler run by the non-profit Common Crawl. The resulting open dataset is widely used for research and, among other things, AI model training. CCBot honours robots.txt.
If blocked: Your pages are excluded from future Common Crawl dataset snapshots. This is not tied to any one AI product.
robots.txt: operator says it is honored

The generator produces rules for the AI search, training, data-use control, and general dataset groups. It does not generate rules for user-triggered fetchers (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User) because several operators say robots.txt does not apply to a fetch a person initiated. Claude-User is honored by Anthropic — add it by hand if you need it.

OAI-SearchBot vs GPTBot

This is the pair people most often confuse. They are different OpenAI crawlers:

OAI-SearchBot

ChatGPT search discovery. OpenAI: opted-out sites “will not be shown in ChatGPT search answers, though can still appear as navigational links.” Allow it to be eligible for ChatGPT search.

GPTBot

Training crawler. OpenAI: “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.” Separate from ChatGPT search eligibility.

Allowing GPTBot does not put a site in ChatGPT search, and blocking GPTBot does not remove it. If ChatGPT search eligibility is the goal, the rule that matters is on OAI-SearchBot.

ClaudeBot and PerplexityBot

ClaudeBot & Claude-SearchBot

Anthropic operates ClaudeBot to collect web content that may contribute to training its models, and Claude-SearchBot to improve the quality of search results shown to Claude users. Anthropic states its crawlers honor robots.txt directives, and documents blocking them with a Disallow: / group per user-agent. A third token, Claude-User, covers user-requested fetches.

PerplexityBot

Perplexity describes PerplexityBot as the crawler that “surface[s] and link[s] websites in search results on Perplexity” and says it “is not used to crawl content for AI foundation models.” Perplexity recommends allowing it in robots.txt. User-requested fetches use a separate Perplexity-User agent that, per Perplexity, generally ignores robots.txt.

Google-Extended, explained

Google-Extended is not the crawler that powers Google Search, and it is not a general-purpose fetcher. Google describes it as a “standalone product token” that publishers use to control whether content Google already crawls may be used to train Gemini models and to ground Gemini answers.

Google’s documentation: “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” Blocking it is purely a decision about Gemini training and grounding.

Because Googlebot and Google-Extended are separate tokens, you can keep normal Google Search crawling while opting out of Gemini training — or the reverse.

How to add these rules to an existing robots.txt

1Back up the current file
Save a copy of your live robots.txt before changing anything. It may contain Googlebot rules, Disallow paths for admin or cart URLs, and one or more Sitemap: lines that must stay.
2Keep the existing rules
Add the generated groups alongside what is already there. Do not delete other user-agent groups, path rules, or Sitemap lines.
3Merge, don't duplicate
If a user-agent from the snippet already has a group in your file (for example a GPTBot group you added earlier), edit that group instead of pasting a second one. Two groups for the same user-agent are ambiguous.
4Check for conflicts with User-agent: *
A catch-all Disallow: / applies to any crawler without its own group. Once a crawler has its own group, only that group applies to it — the catch-all no longer does.
5Deploy at the site root as plain text
robots.txt must be reachable at https://yourdomain/robots.txt, return HTTP 200, and be served as text/plain — not an HTML page and not a redirect to another host.
6Re-check after deploying
Providers note it can take time (OpenAI mentions roughly 24 hours) for robots.txt changes to be picked up. Verify the live file and watch your server logs for the crawler user-agents.

Example: allow AI search, block AI training

A site that wants to be findable in AI search and answer products but does not want its content used for model training can express that with the “Allow search, block training” preset. It produces:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Disallow: /

This is an example, not a recommendation. Whether to allow training crawlers is a policy decision for each site owner. Blocking Google-Extended here does not affect Google Search. Do not copy this block without deciding it is what you want.

Why robots.txt alone does not guarantee AI visibility

Allowing a crawler removes one obstacle. It does not, by itself, cause any of the following:

That the crawler actually visits the site, or visits it often.
That the content is added to any index the AI product uses.
That a page is chosen, ranked, cited, or recommended in an answer.
That an allow overrides an edge block. A WAF, CDN challenge, rate limit, IP block, or bot-management product can stop a crawler even when robots.txt permits it.
That an allow overrides a page-level noindex, an authentication wall, or thin or duplicate content that the product chooses not to use.

In OpenAI’s own words, opting out of OAI-SearchBot keeps a site out of ChatGPT search answers — but allowing it only makes the site eligible. Treat this generator as a way to state a clear, correct policy, not as a growth lever.

Disclaimer: This tool provides an automated readiness audit based on publicly visible technical, structured-data, and content signals. It is not a guarantee that any website will be ranked, recommended, cited, or mentioned by ChatGPT, Claude, Gemini, Perplexity, or any other AI system. AI models decide what to surface using many factors outside any website’s control, including factors that change over time and are not publicly documented.

Check your deployed robots.txt

After you merge these rules and deploy, verify the live file. The AI Crawler Robots.txt Checker reads your published robots.txt and reports what it says for every AI crawler and control token — grouped by role, and including when a User-agent: * catch-all is the rule that applies.

For the wider picture — not just robots.txt — the ChatGPT Website Visibility Checker covers the other technical and content signals that affect whether AI systems can crawl and understand a site.

Frequently asked questions

What is an AI crawler?

An AI crawler is an automated bot operated by an AI company to fetch web pages. Different crawlers have different jobs: some discover and link pages for an AI search or answer product, some collect content that may be used to train models, and some fetch a single page because a user asked the assistant to open it. Each crawler identifies itself with a user-agent token you can name in robots.txt.

What is OAI-SearchBot?

OAI-SearchBot is OpenAI's crawler for ChatGPT's search features. OpenAI's documentation says it is "used to surface websites in search results in ChatGPT", and that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links". It obeys robots.txt.

What is the difference between OAI-SearchBot and GPTBot?

They are separate OpenAI crawlers with separate purposes. OAI-SearchBot is for ChatGPT search discovery. GPTBot is for crawling content that may be used to train OpenAI's foundation models. Allowing GPTBot is not required to appear in ChatGPT search, and blocking GPTBot does not remove a site from ChatGPT search. Decide each one on its own.

Can I allow ChatGPT search but block AI training?

As a robots.txt policy, yes. Because the search and training crawlers are different user-agents, you can Allow OAI-SearchBot (and Claude-SearchBot, PerplexityBot) while adding Disallow: / for GPTBot and ClaudeBot. The "Allow search, block training" preset in the generator does exactly this. Whether a site then actually appears in any product still depends on factors outside robots.txt.

Does ClaudeBot respect robots.txt?

Anthropic's support documentation states that its crawlers "respect 'do not crawl' signals by honoring industry standard directives", and gives a robots.txt example with User-agent: ClaudeBot and Disallow: /. Note that IP-based blocking is described as unreliable; robots.txt is the documented method.

Does PerplexityBot respect robots.txt?

Perplexity's documentation describes PerplexityBot as the crawler that surfaces and links websites in Perplexity search, says it is "not used to crawl content for AI foundation models", and recommends allowing PerplexityBot in robots.txt. A separate user-agent, Perplexity-User, handles user-requested fetches and, per Perplexity, "generally ignores robots.txt rules".

What does Google-Extended control?

Google-Extended is a standalone control token, not a crawler that fetches pages on its own. Google describes it as a way for publishers "to manage whether content Google crawls from their sites may be used for training future generations of Gemini models" and to ground Gemini answers. You add it to robots.txt as its own User-agent group.

Does blocking Google-Extended hurt Google Search rankings?

No. Google's documentation states that "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." Blocking it is a decision about Gemini training and grounding only. It is a different token from the crawlers that power Google Search.

Will allowing AI crawlers make my site appear in ChatGPT?

No. Allowing a crawler only removes a robots.txt obstacle, which makes a site eligible to be fetched. It does not guarantee that the crawler visits, that the content is indexed, or that any AI system cites or recommends the page. Those outcomes depend on many factors that no robots.txt rule and no external tool can control.

Should I replace my existing robots.txt with this generated file?

No. The generated text is a snippet to merge into your existing robots.txt, not a replacement. Your current file may contain rules for Googlebot, sitemaps, CMS paths, and private areas that must stay. Back up the current file, keep its rules, add these groups, and check for any user-agent that now has two groups.