Robots.txt Generator
Build a correct robots.txt with one-click policies for AI crawlers like GPTBot, ClaudeBot, and PerplexityBot.
1. AI crawlers
Choose which AI bots may crawl your site. Blocked bots get their own User-agent group with Disallow: / in the file below.
GPTBot
OpenAI's crawler that collects training data for future GPT models.
OAI-SearchBot
OpenAI's search crawler that powers ChatGPT search results and citations.
ChatGPT-User
Fetches a page live when a ChatGPT user asks about it in a conversation.
ClaudeBot
Anthropic's crawler that gathers public web data to train Claude models.
Claude-User
Fetches a page on demand when a Claude user requests it during a chat.
PerplexityBot
Indexes pages so Perplexity can cite them in its AI answer engine.
Google-Extended
Controls whether Google trains Gemini on your content. It does not affect Search rankings.
CCBot
Common Crawl's bot. Its open dataset feeds many AI training pipelines.
Bytespider
ByteDance's crawler, widely reported to gather data for TikTok's parent company AI.
Amazonbot
Amazon's crawler, used for Alexa answers and Amazon AI services.
Applebot-Extended
Controls whether Apple trains Apple Intelligence models on your pages.
Meta-ExternalAgent
Meta's crawler for AI training and content indexing across its products.
2. General crawl policy
These rules apply to every other crawler through the User-agent: * group, search engines included.
Your robots.txt
User-agent: * Disallow:
Save this as robots.txt at the root of your domain, for example https://example.com/robots.txt. Blocking a bot here stops polite crawlers only; it is a request, not a lock. Everything runs in your browser; nothing you type is stored or sent anywhere.
Go deeper
More AI & Agentic Web tools
- llms.txt Generator · Generate an llms.txt file that tells AI assistants what your site is, what matters, and what to cite.
- AI Tell Remover · Strip the giveaways from any draft: em dashes, curly quotes, invisible characters, and overused AI phrases, flagged and cleaned.