My Online Tools logo
robots.txt Generator

Build a clean robots.txt file with allow and disallow rules, user-agent groups, and sitemap URLs before you publish it at the site root.

My Online Tools logo
MTKits
Direct tools, fewer steps
Blog
Content shortcuts
Canonical Tag CheckerLorem Ipsum GeneratorOpen Graph PreviewReadability Checker
Sponsored

Working area

Create crawler rules that are easy to review

Start from a preset, tune path rules for each user-agent, add sitemap URLs, then copy or download a ready-to-publish robots.txt file.

CleanBlock private pathsBrowser-based

Presets

Start from a common policy

Rules

User-agent groups

Group 1

Discovery

Sitemap and host

Output

Generated robots.txt

Clean
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Disallow: /api/
Disallow: /cart/
Disallow: /checkout/

Sitemap: https://example.com/sitemap.xml
How to use

Checks

Publish readiness

User-agent

pass

1 user-agent group configured.

Crawl policy

pass

The default policy does not block the entire site for all crawlers.

Sitemap

pass

1 absolute sitemap URL included.

Output

info

9 lines ready to publish at /robots.txt.

GuideHow it works3 steps
01

Choose a starting preset

Begin with allow-all, block private paths, or block-all, then adjust the rules for your site.

02

Add allow, disallow, and sitemap lines

Enter path rules per user-agent and reference one or more sitemap URLs for crawler discovery.

03

Copy or download robots.txt

Review the readiness checks, then publish the generated file at your site root as /robots.txt.

More toolsRelated tools
SEO Checker

Audit a public URL for robots.txt presence, sitemap discovery, and on-page SEO signals.

Canonical Tag Checker

Check pasted HTML for canonical tags, duplicates, and expected URL mismatches.

robots.txt Generator Guide

Learn what robots.txt rules mean and how to publish a clean crawler policy.

FAQFrequently asked questions

Where should robots.txt be published?

Publish it at the site root, for example https://example.com/robots.txt. Crawlers look for that exact location.

Does robots.txt hide private pages from search results?

No. It guides crawling. Pages can still appear if they are linked elsewhere. Use authentication or noindex for stronger control.

Should I block my whole site with Disallow: /?

Only if the site should not be crawled. A site-wide block prevents search engines from discovering useful pages.

Can I list more than one sitemap?

Yes. Add one sitemap URL per line. Large sites often keep separate sitemaps for posts, products, or languages.

ShareShare this tool
Back to MTKits