@mixmaxhq/express-robots
v1.0.0
Published
Express middleware for generating your robots.txt
Downloads
931
Readme
express-robots
Express middleware for generating a robots.txt or responding with an existing file.
Using a file
app.use(robots(__dirname + '/robots.txt'));
Using an object
Basic object
app.use(robots({UserAgent: '*', Disallow: '/'}))
Will produce:
UserAgent: *
Disallow: /
Crawl Delay
You can optionally pass a CrawlDelay in just like passing in Disallow
app.use(robots({UserAgent: '*', Disallow: '/', CrawlDelay: '5'}))
Will produce:
UserAgent: *
Disallow: /
Crawl-delay: 5
Or an array of objects
app.use(robots([
{
UserAgent: 'Googlebot',
Disallow: '/no-google'
},
{
UserAgent: 'Bingbot',
Disallow: '/no-bing'
}
]));
Will produce:
UserAgent: Googlebot
Disallow: /no-google
UserAgent: Bingbot
Disallow: /no-bing
Or either property (UserAgent | Disallow) as an array
app.use(robots([
{
UserAgent: [ 'Googlebot', 'Slurp' ],
Disallow: [ '/no-google', '/no-yahoo' ]
},
{
UserAgent: '*',
Disallow: [ '/no-bots', '/still-no-bots' ]
}
]));
Will produce:
UserAgent: Googlebot
UserAgent: Slurp
Disallow: /no-google
Disallow: /no-yahoo
UserAgent: *
Disallow: /no-bots
Disallow: /still-no-bots