- Home
- SEO & Marketing
- Robots.txt Generator
Robots.txt Generator
Generate a robots.txt with path rules, sitemaps, AI crawler blocks and content signals.
What it is for?AI crawlers and content signals
The same split Cloudflare uses: training, AI search and assistants. Search engines such as Googlebot, bingbot and Applebot are not in these lists.
Paths and other crawlers
What it is for?
How rule groups work
Each group targets a crawler by its user-agent; * means every crawler. Choose what is allowed by default, then list the paths that are the exception. Put the file at the root of your site, for example https://example.com/robots.txt.
What the AI controls do
Block AI training writes Disallow rules for the crawlers in the Cloudflare managed robots.txt: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Amazonbot and meta-externalagent, plus Google-CloudVertexBot and FacebookBot from their AI crawler list. AI search and assistants are separate, so ordinary search bots stay allowed.
Content signals say how a crawler may use content it fetched: search for an index, ai-input for live answers, ai-train for model training, and use for what it may keep. The file only states a preference. A crawler can ignore it; actually blocking the request has to happen on the server or at a proxy.