Create custom robots.txt files to control AI crawlers access to your website.
Block or Allow AI Bots with Ease – Free robots.txt Generator for AI Crawlers like Perplexity, Anthropic, and More
Want to control whether AI companies crawl your website? Use our AI Crawler Robots.txt Generator to instantly create a robots.txt file that blocks or allows access to popular AI crawlers like Anthropic’s Claude, Perplexity AI, ChatGPT (OpenAI), and others.
An AI crawler (or AI user-agent) is a bot used by AI companies to scrape websites in order to train language models. These crawlers act like search engine bots but often operate under different user-agents, such as:
If you don’t want your content to be used for training AI models, you must explicitly block these bots in your robots.txt file.
While AI tools can be helpful, many website owners prefer not to share their content for these reasons:
Our generator supports blocking or allowing major AI user-agents. Some of them are mentioned below.
| AI Company | User-Agent | Description |
|---|---|---|
| Anthropic AI | anthropic-ai | Claude AI crawler |
| Perplexity AI | perplexitybot | Perplexity’s AI search bot |
| OpenAI/ChatGPT | chatgpt-user, OpenAI-User-Agent | ChatGPT data collection agents |
| Google AI | Google-Extended | Google’s AI training bot |
| Common Crawl | CCBot | Used by many LLM companies |
Our tool is frequently updated with new AI bots as they appear.
User-agent: anthropic-ai
Disallow: /
User-agent: perplexitybot
Disallow: /
User-agent: chatgpt-user
Disallow: /
User-agent: OpenAI-User-Agent
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
Place the file in the root directory of your website. Example:
https://yourwebsite.com/robots.txt
This ensures crawlers find and respect your instructions.
Not all AI companies follow robots.txt protocols. Reputable companies like OpenAI, Google, and Anthropic generally do — but others may ignore it.
For stronger protection: