npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

scrapman

v2.3.1

Published

Retrieve real (with Javascript executed) HTML code from an URL, ultra fast and supports multiple parallel loading of webs

Downloads

12

Readme

Scrapman

Ski-bi dibby dib yo da dub dub Yo da dub dub Ski-bi dibby dib yo da dub dub Yo da dub dub I'm the Scrapman!

###THE FASTEST SCRAPPER EVER*... AND IT SUPPORTS PARALLEL REQUESTS (*arguably)

Scrapman is a blazingly fast real (with Javascript executed) HTML scrapper, built from the ground up to support parallel fetches, with this you can get the HTML code for 50+ URLs in seconds (~30 seconds).

On NodeJS you can easily use request to fetch the HTML from a page, but what if the page you are trying to load is NOT a static HTML page, but it has dynamic content added with Javascript? What do you do then? Well, you use The Scrapman.

It uses Electron to dynamically load web pages into several <webview> within a single Chromium instance. This is why it fetches the HTML exactly as you would see it if you inspect the page with DevTools.

This is NOT an browser automation tool (yet), it's a node module that gives you the processed HTML from an URL, it focuses on multiple parallel operations and speed.

##USAGE

1.- Install it

npm install scrapman -S

2.- Require it

var scrapman = require("scrapman");

3.- Use it (as many times as you need)

Single URL request

scrapman.load("http://google.com", function(results){
	//results contains the HTML obtained from the url
	console.log(results);
});

Parallel URL requests

//yes, you can use it within a loop.
for(var i=1; i<=50; i++){
    scrapman.load("https://www.website.com/page/" + i, function(results){
        console.log(results);
    });
}

##API

###- scrapman.load(url, callback)

####url Type: String

The URL from which the HTML code is going to be obtained.

####callback(results) Type: Function

The callback function to be executed when the loading is done. The loaded HTML will be in the results parameter.

###- scrapman.configure(config)

####config The configuration object can set the following values

  • maxConcurrentOperations: Integer - The intensity of processing, how many URLs can be loaded at the same time, default: 50

  • wait: Integer - The amount of milliseconds to wait before returning the HTML code of a webpage after it has been completely loaded, default: 0

Questions

Feel free to open Issues to ask questions about using this package, PRs are very welcomed and encouraged.

SE HABLA ESPAÑOL

License

MIT © Daniel Nieto