Robots.txt Analyzer

Check and analyze robots.txt files to understand crawling rules and sitemap locations