Robots.txt & meta robots generator
Build a valid robots.txt or a meta robots tag without second-guessing the syntax. Add crawl rules, block AI crawlers, drop in your sitemaps, or generate noindex / nofollow tags and X-Robots-Tag headers. Everything runs in your browser.
Or as an HTTP header, for non-HTML files like PDFs:
Always test changes in Google Search Console’s robots.txt report before relying on them. Blocking the wrong path can quietly remove pages from search.
Crawling is not indexing
This is the trap that catches people out. Robots.txt tells crawlers which URLs they may fetch. It does not decide what appears in search. A page you block in robots.txt can still be indexed if something links to it — Google just shows it without a description because it was never allowed to read the page.
If your goal is to keep a page out of the results, do the opposite of what feels natural: leave it crawlable and add a noindex meta tag or X-Robots-Tag header. Never combine Disallow with noindex on the same URL, because a blocked page can never be crawled, so Google never sees the noindex and the page can linger in the index.
A few things worth knowing
The file must sit at the domain root and is served per host and protocol. Paths are case-sensitive. Google ignores Crawl-delay and manages its own crawl rate, though Bing and Yandex honour it. Blocking AI crawlers such as GPTBot or Google-Extended works only because those bots choose to obey the standard — it is not enforcement. And robots.txt is public, so don’t use it as a list of secret URLs you’d rather nobody found.
Questions
Does robots.txt stop a page from being indexed?
No. Robots.txt controls crawling, not indexing. A URL you disallow can still appear in Google's results (usually without a description) if other pages link to it. To keep a page out of the index, allow crawling and use a noindex meta tag or X-Robots-Tag header instead.
Should I use both Disallow and noindex on the same page?
No, and this is a common mistake. If you disallow a page in robots.txt, Google can't crawl it, which means it can't see the noindex tag, so the page may stay indexed. Pick one: block crawling with robots.txt, or allow crawling and apply noindex.
Where does the robots.txt file go?
At the root of the domain, reachable at /robots.txt (for example https://example.com/robots.txt). It's specific to that host and protocol, and paths are case-sensitive, so /Folder/ and /folder/ are different.
Can I block ChatGPT and other AI crawlers?
Yes. Add user-agent groups for bots such as GPTBot, Google-Extended, CCBot and ClaudeBot with Disallow: / (the 'Block AI crawlers' preset does this). Reputable crawlers honour it, but robots.txt is a voluntary standard, so it isn't hard enforcement.
Does Google obey Crawl-delay?
No. Google ignores the Crawl-delay directive and manages crawl rate automatically. Bing and Yandex do respect it. It's included here for the crawlers that use it, but it won't affect Googlebot.
Technical SEO that doesn’t cost you rankings.
One wrong line in a robots file can bury a site. We handle the fiddly, high-stakes technical work so you don’t have to guess.