What this tool does
This is a free, browser-only generator for the two files search engines and crawlers read first: robots.txt (which URLs a crawler may request) and sitemap.xml (which URLs you want discovered and indexed). Instead of hand-writing directive syntax, you build robots.txt from User-agent groups with Allow/Disallow/Crawl-delay rules, load a one-click preset — including blocking the specific bots AI companies use to scrape training data — and get instant validation for the mistakes that most often break a robots.txt file silently. The sitemap generator turns a plain list of URLs into a spec-valid sitemap.xml with optional lastmod, changefreq and priority fields.
How to use it
For robots.txt: click a preset to start from a known-good configuration, or click "+ Add User-agent group" to build from scratch. Inside each group, set the User-agent (use * for all crawlers, or a specific bot name like GPTBot), add Allow/Disallow rules with a path, and optionally a Crawl-delay in seconds. Add one or more sitemap URLs below the groups. The output panel updates live, and the panel beside it explains what every generated line actually does. For sitemap.xml: paste your URLs (one per line, optionally with lastmod/changefreq/priority), click Generate sitemap.xml, and copy or download the result. Use "Use as robots.txt Sitemap:" to push the sitemap's URL back into the robots.txt builder above.
Example 1 — robots.txt: block AI bots, keep search engines, ship a sitemap
Input — rules built in the editor
Group 1 → User-agent: GPTBot → Disallow: /
Group 2 → User-agent: CCBot → Disallow: /
Group 3 → User-agent: Google-Extended → Disallow: /
Group 4 → User-agent: * → Disallow: /admin/
Allow: /admin/help/
Sitemap field → https://example.com/sitemap.xml
Output — robots.txt
User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: * Disallow: /admin/ Allow: /admin/help/ Sitemap: https://example.com/sitemap.xml
What it means
GPTBot, CCBot and Google-Extended each get their own group with Disallow: /, which blocks them from every URL on the site — this does not affect Googlebot's normal search indexing, since Google-Extended is a separate, AI-training-only user agent. The general * group blocks /admin/ for everyone else but carves out /admin/help/ with an explicit Allow, and the Sitemap line at the end points every well-behaved crawler at the full URL list.
Example 2 — sitemap.xml from a pasted URL list
Input — pasted into the URL list box
https://example.com/, 2026-07-01, daily, 1.0 https://example.com/pricing, 2026-06-15, weekly, 0.8 https://example.com/blog/robots-txt-guide
Output — sitemap.xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/</loc>
<lastmod>2026-07-01</lastmod>
<changefreq>daily</changefreq>
<priority>1.0</priority>
</url>
<url>
<loc>https://example.com/pricing</loc>
<lastmod>2026-06-15</lastmod>
<changefreq>weekly</changefreq>
<priority>0.8</priority>
</url>
<url>
<loc>https://example.com/blog/robots-txt-guide</loc>
</url>
</urlset>
The third URL has no lastmod, changefreq or priority — all three are optional per the sitemap protocol, so the generator simply omits those child elements for that <url> entry rather than inventing values.
Common robots.txt mistakes this tool catches
- Missing leading slash.
Disallow: adminis a relative, effectively meaningless value to most parsers — it needs to beDisallow: /admin(or/admin/to scope it to the folder) to actually block anything. - Assuming first-match order. Google and Bing apply the longest matching path in a group, not the first rule written, so a broad
Disallow: /can be legally overridden later by a more specificAllow: /public/. Older or simpler bots may instead use first-match-wins, so relying on ordering for anything important is fragile — this tool flags rule placements that would behave differently under the two models. - Duplicate User-agent groups. Repeating
User-agent: *in two separate blocks is technically legal but many parsers only honor the first or merge unpredictably; keeping one group per agent avoids the ambiguity. - Expecting Crawl-delay to throttle Google. Googlebot ignores it outright; it only affects Bing, Yandex and some smaller crawlers.
- Confusing robots.txt with noindex. Disallowing a URL stops crawling, but a URL that is already indexed and merely linked-to elsewhere can still appear in search results without a snippet; to truly remove a page from the index you need a
noindexmeta tag or header, which requires the page to be crawlable in the first place.
Blocking AI training crawlers in 2026
Several AI companies publish a distinct, named user agent for the crawler that gathers training data, separate from any crawler they run for product features like search or citations. The "Block AI training bots" preset here pre-fills a dedicated group for each of: GPTBot and ChatGPT-User (OpenAI), CCBot (Common Crawl, whose dataset feeds many models), Google-Extended (Google's opt-out for Gemini/AI-training use of your content, independent from Googlebot search indexing), PerplexityBot, ClaudeBot (Anthropic), Bytespider (ByteDance), Amazonbot, and Applebot-Extended. Robots.txt is an advisory standard, not an access-control mechanism: a crawler has to choose to fetch and honor it, and the bots above generally do, but nothing stops a scraper that ignores robots.txt or rotates its user-agent string. For true enforcement you need server-side blocking (user-agent or IP filtering, rate limiting, or a bot-management layer) in addition to a clear robots.txt signal.
Sitemap.xml basics
A sitemap is capped by the protocol at 50,000 URLs and 50MB uncompressed per file; larger sites split into multiple sitemap files referenced from a sitemap index file. Every <loc> must be an absolute, correctly escaped URL (ampersands become &, and so on) — this tool escapes automatically. lastmod should be an ISO-8601 date; changefreq is one of always, hourly, daily, weekly, monthly, yearly, never; priority is a relative hint from 0.0 to 1.0. Google has said it largely ignores changefreq and priority for crawl decisions, relying instead on observed change frequency and link signals, but Bing and other consumers of your sitemap still read them, so accurate values are still worth setting.
FAQ
Why does my Disallow rule not work?
The most common cause is a path missing its leading slash — Disallow: admin is not the same as Disallow: /admin, and most crawlers will simply ignore or misinterpret the relative form. The second most common cause is testing against Googlebot's actual behavior: Google (and Bing) apply the longest matching path, not the first one written, so a broad Disallow: / earlier in a group can still be overridden by a more specific Allow: /public/ later in the same group. This tool flags both cases automatically.
Does order matter in a robots.txt file?
It depends on the crawler. Google, Bing and most modern search engines evaluate all rules in a User-agent group and let the longest (most specific) matching path win, regardless of the order the lines appear in. Some older or simpler bots instead apply the first rule that matches, top to bottom. Because you cannot know which behavior every crawler on the internet uses, the safe practice is to write your most specific rules deliberately rather than relying on ordering — this tool warns you when a rule's placement could be ambiguous under a first-match parser.
Does Crawl-delay actually slow down Googlebot?
No. Google has stated publicly that Googlebot ignores the Crawl-delay directive entirely; crawl rate for Google is instead controlled through Google Search Console's crawl-rate settings (where available) or by returning HTTP 429/503 responses under load. Crawl-delay is still honored by Bing, Yandex, and a number of smaller or older crawlers, so it remains useful for those, just not as a universal throttle.
How do I block AI crawlers like GPTBot or CCBot from training on my content?
Add a dedicated User-agent group for each bot you want to block with Disallow: /, separate from your general User-agent: * group that governs search engines. The "Block AI training bots" preset in this tool pre-fills groups for OpenAI's GPTBot and ChatGPT-User, Common Crawl's CCBot, Google-Extended, PerplexityBot, Anthropic's ClaudeBot, ByteDance's Bytespider, Amazonbot and Applebot-Extended. Robots.txt is advisory, not enforced — well-behaved crawlers from major AI labs do respect it, but it cannot stop a scraper that ignores it outright.
What is the maximum size or number of URLs for a sitemap.xml file?
The sitemap protocol caps a single sitemap file at 50,000 URLs and 50MB uncompressed. If your site has more URLs than that, split them across multiple sitemap files and reference all of them from a sitemap index file, then point robots.txt at the index. This tool warns you when your pasted URL list exceeds 50,000 entries.
Do changefreq and priority in sitemap.xml still matter for SEO?
Google has said for years that it largely ignores both changefreq and priority when deciding what or how often to crawl, relying instead on observed change patterns and internal link signals. They remain part of the official sitemap protocol and are still read by Bing and some other crawlers and tools, so including accurate values does no harm and can help non-Google consumers of your sitemap, but you should not expect them to change your Google crawl rate on their own.