robots.txt Tester

Paste robots.txt content, enter a URL path and user-agent to test if it is allowed or disallowed.

robots.txt Content

Parsed Rules (10)

L1: User-agent: *

L2: Allow: /

L3: Disallow: /admin/

L4: Disallow: /private/*

L5: Disallow: /api/

L7: User-agent: Googlebot

L8: Allow: /api/public

L9: Disallow: /api/

L11: Crawl-delay: 10

L13: Sitemap: https://example.com/sitemap.xml

Test Configuration

User-Agent

URL Path

ALLOWED

/api/public for Googlebot

Matching rule (line 8): Allow: /api/public

Crawl-delay: 10s

Quick Test Paths

Was this page helpful?

Related tools