1) Load or paste robots.txt
Sitemaps: 0 found
Groups: 0 detected
Crawl‑delay: –
2) Test a URL against a user‑agent
How it works
- Groups are selected by the most specific User-agent that matches your chosen bot; fallback to *.
- Within the selected group(s), the longest matching rule wins. If two rules tie, Allow beats Disallow.
- Pattern syntax supports * wildcard and end anchor $ (per RFC 9309 / Google).
- Paths are matched on the URL path + query string; scheme/host are ignored during rule comparison.