mr-roboto
v0.1.0-alpha.0
Published
Utility to handle and parse robots.txt
Downloads
4
Readme
Mr-Roboto
Mr-Roboto is aimed at handling and parsing a website's robots.txt
file, providing a easy interface to validate if a specific path is allowed or not.
Mr-Roboto is NOT production ready, and as such should not be used in any mission critical codebase.