npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

detect-bot-crawler

v1.0.0

Published

A Node.js package to detect and block potential scraping bots based on request patterns, including User-Agent analysis, rate-limiting, and IP reputation checks.

Downloads

3

Readme

Detect-Bot

A Node.js package to detect and block potential scraping bots based on request patterns, including User-Agent analysis, rate-limiting, and IP reputation checks. This package helps protect your web application from malicious scraping bots.

Features

  • User-Agent detection: Detects common bot user-agents.
  • Rate-limiting: Identifies IPs that make an excessive number of requests in a short period.
  • Optional IP check: Can integrate with an external API to check if the request comes from a known malicious IP.
  • Custom bot patterns: Allows you to define your own regex patterns to detect bots.

Installation

Install the package via npm:

npm install detect-bot

##Usage Integrate the bot detection package into your Node.js or Express application. Here's a quick example:

const express = require('express'); const BotDetector = require('detect-bot'); // Replace with your package name

const app = express(); const botDetector = new BotDetector({ apiEndpoint: 'https://example.com/check-ip' });

app.use(async (req, res, next) => { const botCheck = await botDetector.detect(req); if (botCheck.isBot) { console.log(Bot detected: ${botCheck.reason}); res.status(403).send('Access denied'); } else { next(); } });

app.get('/', (req, res) => { res.send('Hello, this page is protected from bots!'); });

app.listen(3000, () => { console.log('Server is running on port 3000'); });

##Configuration When you create a new BotDetector instance, you can pass in optional configuration options:

  • botPatterns: An array of regular expressions to detect bot-like User-Agent strings. Default includes patterns for common bots (e.g., /bot/i, /crawl/i, etc.). apiEndpoint: An optional API endpoint to check IP addresses against an external IP reputation service.
const botDetector = new BotDetector({
  botPatterns: [/bot/i, /spider/i, /scrape/i], // Customize bot patterns
  apiEndpoint: "https://example.com/check-ip", // External API for IP lookup
});

##API detect(req) Detect if the request is from a bot. The method returns a promise that resolves with an object containing the detection result:

isBot: Boolean indicating if the request is likely from a bot. reason: The reason why the request was flagged as a bot (e.g., "Bot-like User-Agent", "Excessive request rate"). Example:

const detection = await botDetector.detect(req);
if (detection.isBot) {
  console.log(`Bot detected: ${detection.reason}`);
}

##Testing To run the tests for this package:

npm test

##MIT License.

How to Customize:

  • Update the package name in the README.md to reflect the actual package name you will use in npm.
  • Customize the description, and if you plan to add more advanced features (e.g., machine learning detection or IP blacklisting), include them in the Features section.

This should give you a solid base to get started!