top-crawler-agents
v1.0.28
Published
A list of common crawler agents used on Internet..
Downloads
280
Maintainers
Readme
top-crawler-agents
A list of common crawler user agents useful for retrieving metadata from links derivated from crawler-user-agents.
Some websites (such as Twitter) only return rich HTML metadata if you are identified as popular crawler (like Slack, or WhatsApp).
The list is derivated from crawler-user-agents after applying a script, so it's always up-to-date to latest changes.
Install
$ npm install top-crawler-agents --save
Usage
const uniqueRandomArray = require('unique-random-array')
const randomCrawlerAgent = uniqueRandomArray(require('top-crawler-agents'))
console.log(randomCrawlerAgent())
// => 'Slackbot-LinkExpanding (+https://api.slack.com/robots)'
Related
- top-user-agents – A list of most common User Agent used on Internet.
top-crawler-agents © Kiko Beats, released under the MIT License. Authored and maintained by Kiko Beats with help from contributors.
kikobeats.com · GitHub Kiko Beats · X @Kikobeats