- Home
- SEO & Marketing
- Robots.txt Generator
Robots.txt Generator
Generate a robots.txt with path rules, sitemaps, AI crawler blocks and content signals.
AI crawlers and content signals
The same split Cloudflare uses: training, AI search and assistants. Search engines such as Googlebot, bingbot and Applebot are not in these lists.
Paths and other crawlers
How rule groups work
Each group targets a crawler by its user-agent; * means every crawler. Choose what is allowed by default, then list the paths that are the exception. Put the file at the root of your site, for example https://example.com/robots.txt.
What the AI controls do
Block AI training writes Disallow rules for the crawlers in the Cloudflare managed robots.txt: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot and meta-externalagent, plus Google-CloudVertexBot and FacebookBot from their AI crawler list. AI search and assistants are separate, so ordinary search bots stay allowed.
Content signals say how a crawler may use content it fetched: search for an index, ai-input for live answers, ai-train for model training, and use for what it may keep. The file only states a preference. A crawler can ignore it; actually blocking the request has to happen on the server or at a proxy.