Robots.txt Generator
Generate a valid robots.txt with presets (WordPress, Shopify, block-all), per-bot rules, crawl-delay and a sitemap line — free, built in your browser and ready to download.
100% private — runs entirely in your browser. Nothing you enter is uploaded or logged.
noindex meta tag (and leave it crawlable).Place the finished file at https://yourdomain.com/robots.txt (site root). Built entirely in your browser — nothing is uploaded.
How to use the Robots.txt Generator
- Pick a preset (WordPress, Shopify, allow/block all).
- Add Disallow/Allow paths and your sitemap URL.
- Copy or download the robots.txt.
- Upload it to your site root.
What you can use it for
- Control which pages search engines crawl
- Point crawlers to your XML sitemap
- Block admin or cart URLs from crawling
- Start from a CMS-specific preset
More free tools
Optimizing your pages?
Generate meta tags, preview snippets and perfect your social cards with our free SEO tools.
Explore all free SEO tools →Frequently asked questions
Where do I put the robots.txt file?
At the root of your domain — https://yourdomain.com/robots.txt. It must be reachable there for crawlers to read it.
How do I block a page or folder from Google?
Add a Disallow line for the path, e.g. Disallow: /private/. Remember this blocks crawling, not indexing.
Does robots.txt stop a page from being indexed?
No — a disallowed URL can still appear in search if it's linked elsewhere. To keep a page out of results use a noindex meta tag instead.
What's the difference between Disallow and noindex?
Disallow tells crawlers not to fetch a URL; noindex (a meta tag) tells them not to show it in results. Don't disallow a page you want noindexed, or Google can't see the noindex.
How do I allow all bots to crawl everything?
Use User-agent: * with an empty Disallow line (Disallow:). This tool's 'Allow all' preset does exactly that.