GitHub - nickmoline/robots-checker: Library to check a url for all possible robots exclusion protocols (robots.txt, http X-Robots-Tag header, meta robots tag, or http status codes)
Blocked or not? Wildcards, typos and other robots.txt ambiguities | TechnicalSEO.com