express-seo-robots

Simple robots.txt file generation for express

Install

npm install --save express-seo-robots

Usage

Add middleware to your express application before all other routes.

var express = require('express');
var app = express();
var robotsTxt = require('express-seo-robots');

// add middleware with robots.txt config
app.use(robotsTxt({ userAgent: '*', allow: '/', sitemap: 'https://yourdomain.com/sitemap.xml' }));

// standard express route
app.get('/', function(req, res) {
    res.send('Hello world!');
});

Based on the setup above, a requests to /robots.txt will now return:

User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml

You can provide multiple entries by passing an array of values, the following:

// add robots.txt middleware with custom config
app.use(robotsTxt([
    { userAgent: 'Googlebot-news', allow: '/news', CrawlDelay: '5' },
    { userAgent: 'Googlebot', disallow: '/private' },
    { userAgent: 'Googlebot' disallow: '/*.xls$' }
]);

Produces:

User-agent: Googlebot-news
Allow: /
Crawl-delay: 5

User-agent: Googlebot
Disallow: /private

User-agent: Googlebot
Disallow: /*.xls$

Options

Property | Type | Description :----------- |:--------- |:------------------------------------------ userAgent | string | Name of a search engine/bot that the rule applies to allow | string | URL relative to the root domain that should be crawled by the user agent disallow | string | URL relative to the root domain that should not be crawled by the user agent crawlDelay | integer | Time in seconds the search engine/bot should wait before sending the next request sitemap | Array | List of fully-qualified SiteMap URLs

Contributing

Feel free to contribute, either by raising an issue or:

1. Fork it!
2. Create your feature branch: git checkout -b my-new-feature
3. Commit your changes: git commit -m 'Add some feature'
4. Push to the branch: git push origin my-new-feature
5. Submit a pull request

History

For change-log, check releases.

License

Licensed under MIT License © John Doherty