robots.txt Generator

Generate a robots.txt file quickly.

User-agent

Allow paths (one per line)

Disallow paths (one per line)

Crawl-delay (optional)

Sitemap URL (optional)

robots.txt

User-agent: *
Disallow: /admin
Disallow: /private

Tip: rules are evaluated from most specific to least specific by many crawlers.

Was this page helpful?

Related tools