Free Robots.txt Tester
Enter any page URL to see whether a crawler is allowed to access it—and the exact robots.txt rule that decides the answer.
What this test checks
The tester fetches the site's live /robots.txt, finds the rule group for your selected crawler, and applies the most specific matching rule to the page path. The result includes the winning directive and source line so you can verify the answer yourself.
One important limit
Robots.txt controls crawling; it does not hide private content and does not guarantee that a URL stays out of search results. Protect private pages with authentication, and use an index-control directive when indexing is the issue.
Two common rules
Block every crawler from a folder
User-agent: * Disallow: /private/
Allow every crawler everywhere
User-agent: * Disallow: