norobots
v0.1.0
Published
Robots.txt rules
Downloads
2
Maintainers
Readme
norobots
An attempt to implement the 1997 Internet Draft specification A Method for Web Robots Control.
Usage
const norobots = require('norobots')
let robots = norobots('User-Agent: *\nDisallow: /cgi-bin/\n')
if (robots.allow('fig-tree', '/example.html')) {
// Go fetch the url
}