detect-bot-crawler
v1.0.0
Published
A Node.js package to detect and block potential scraping bots based on request patterns, including User-Agent analysis, rate-limiting, and IP reputation checks.
Downloads
3
Maintainers
Readme
Detect-Bot
A Node.js package to detect and block potential scraping bots based on request patterns, including User-Agent analysis, rate-limiting, and IP reputation checks. This package helps protect your web application from malicious scraping bots.
Features
- User-Agent detection: Detects common bot user-agents.
- Rate-limiting: Identifies IPs that make an excessive number of requests in a short period.
- Optional IP check: Can integrate with an external API to check if the request comes from a known malicious IP.
- Custom bot patterns: Allows you to define your own regex patterns to detect bots.
Installation
Install the package via npm:
npm install detect-bot
##Usage Integrate the bot detection package into your Node.js or Express application. Here's a quick example:
const express = require('express'); const BotDetector = require('detect-bot'); // Replace with your package name
const app = express(); const botDetector = new BotDetector({ apiEndpoint: 'https://example.com/check-ip' });
app.use(async (req, res, next) => {
const botCheck = await botDetector.detect(req);
if (botCheck.isBot) {
console.log(Bot detected: ${botCheck.reason}
);
res.status(403).send('Access denied');
} else {
next();
}
});
app.get('/', (req, res) => { res.send('Hello, this page is protected from bots!'); });
app.listen(3000, () => { console.log('Server is running on port 3000'); });
##Configuration When you create a new BotDetector instance, you can pass in optional configuration options:
- botPatterns: An array of regular expressions to detect bot-like User-Agent strings. Default includes patterns for common bots (e.g., /bot/i, /crawl/i, etc.). apiEndpoint: An optional API endpoint to check IP addresses against an external IP reputation service.
const botDetector = new BotDetector({
botPatterns: [/bot/i, /spider/i, /scrape/i], // Customize bot patterns
apiEndpoint: "https://example.com/check-ip", // External API for IP lookup
});
##API detect(req) Detect if the request is from a bot. The method returns a promise that resolves with an object containing the detection result:
isBot: Boolean indicating if the request is likely from a bot. reason: The reason why the request was flagged as a bot (e.g., "Bot-like User-Agent", "Excessive request rate"). Example:
const detection = await botDetector.detect(req);
if (detection.isBot) {
console.log(`Bot detected: ${detection.reason}`);
}
##Testing To run the tests for this package:
npm test
##MIT License.
How to Customize:
- Update the package name in the
README.md
to reflect the actual package name you will use in npm. - Customize the description, and if you plan to add more advanced features (e.g., machine learning detection or IP blacklisting), include them in the
Features
section.
This should give you a solid base to get started!