Crawler Rules — Specialized Audit Tool
Robots.txt Directive Auditor
Audit page crawling guidelines. Inspects Robots.txt setups for user-agent rules, Disallow directives, and sitemap reference links.
Enter any live HTTPS website to inspect module rules and generated Schema.org JSON-LD code.