How to use

Generate the policy first, then publish it at /robots.txt

The robots.txt Generator helps you create allow and disallow rules, user-agent groups, and sitemap references in the browser. Use it to prepare a clean file before uploading it to your site root.

Open the tool

Step 1

Pick a preset

Start with allow-all, block private paths, or block-all. Presets give you a safe baseline before you customize path rules.

Step 2

Tune user-agent rules

Add Allow and Disallow paths for * or specific bots. Keep private admin, cart, or API routes out of crawl paths when needed.

Step 3

Add sitemap URLs

List absolute sitemap URLs so crawlers can discover indexable pages faster after they read robots.txt.

Step 4

Publish at the site root

Copy or download the file and host it as https://your-domain.com/robots.txt. That exact path is what crawlers request.

Practical tips

robots.txt guides crawling; it does not guarantee privacy or noindex behavior.
Use Disallow: / only for staging or private sites that should not be crawled.
Prefer absolute https:// sitemap URLs.
Keep the file short and readable so rule mistakes are easy to spot.
After publishing, verify the live file and re-check with an SEO audit tool.