npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

puppeteer-page-proxy-abowcut

v1.2.15

Published

Additional Node.js module to use with 'puppeteer' for setting proxies per page basis. Altered by abowcut

Downloads

132

Readme

puppeteer-page-proxy-abowcut

Additional Node.js module to use with puppeteer for setting proxies per page basis. Additional changes made to the original library to allow retrying a 4xx/5xx response with a second proxy.

Forwards intercepted requests from the browser to Node.js where it handles the request then returns the response to the browser, changing the proxy as a result.

Features

  • Proxy per page and proxy per request
  • Supports http, https, socks4 and socks5 proxies
  • Supports authentication
  • Handles cookies

Installation

npm i puppeteer-page-proxy-abowcut

API

PageProxy(pageOrReq, proxy)

  • pageOrReq <object> 'Page' or 'Request' object to set a proxy for.
  • proxy <string|object> Proxy to use in the current page.
    • Begins with a protocol (e.g. http://, https://, socks://)
    • In the case of proxy per request, this can be an object with optional properties for overriding requests:
      url, method, postData, headers
      See httpRequest.continue for more info about the above properties.

PageProxy.lookup(page[, lookupService, isJSON, timeout])

  • page <object> 'Page' object to execute the request on.
  • lookupService <string> External lookup service to request data from.
    • Fetches data from api64.ipify.org by default.
  • isJSON <boolean> Whether to JSON.parse the received response.
    • Defaults to true.
  • timeout <number|string> Time in milliseconds after which the request times out.
    • Defaults to 30000.
  • returns: <Promise> Promise which resolves to the response of the lookup request.

NOTE: By default this method expects a response in JSON format and JSON.parse's it to a usable javascript object. To disable this functionality, set isJSON to false.

Usage

Importing:

const useProxy = require('puppeteer-page-proxy-abowcut');

Proxy per page:

await useProxy(page, 'http://127.0.0.1:80');

To remove proxy, omit or pass in falsy value (e.g null):

await useProxy(page, null);

Proxy per request:

await page.setRequestInterception(true);
page.on('request', async request => {
    await useProxy(request, 'https://127.0.0.1:443');
});

The request object itself is passed as the first argument. The proxy can now be changed every request.

Using it along with other interception methods:

await page.setRequestInterception(true);
page.on('request', async request => {
    if (request.resourceType() === 'image') {
        request.abort();
    } else {
        await useProxy(request, 'socks4://127.0.0.1:1080');
    }
});

Overriding requests:

await page.setRequestInterception(true);
page.on('request', async request => {
    await useProxy(request, {
        proxy: 'socks5://127.0.0.1:1080',
        url: 'https://example.com',
        method: 'POST',
        postData: '404',
        headers: {
            accept: 'text/html'
        }
    });
});

NOTE: It is necessary to set page.setRequestInterception to true when setting proxies per request, otherwise the function will fail.

Authenticating:

const proxy = 'https://user:pass@host:port';

IP lookup:

// 1. Waits until done, 'then' continues
const data = await useProxy.lookup(page1);
    console.log(data.ip);

// 2. Executes and 'comes back' once done
useProxy.lookup(page2).then(data => {
    console.log(data.ip);
});

In case of any CORS errors, use --disable-web-security launch flag:

const browser = await puppeteer.launch({
    args: ['--disable-web-security']
});

FAQ

How does this module work?

It takes over the task of requesting content from the browser to do it internally via a requests library instead. Requests that are normally made by the browser, are thus made by Node. The IP's are changed by routing the requests through the specified proxy servers using *-proxy-agent's. When Node gets a response back from the server, it's forwarded to the browser for completion/rendering.

Why am I getting "Request is already handled!"?

This happens when there is an attempt to handle the same request more than once. An intercepted request is handled by either httpRequest.abort, httpRequest.continue or httpRequest.respond methods. Each of these methods 'send' the request to its destination. A request that has already reached its destination cannot be intercepted or handled.

Why does the browser show "Your connection to this site is not secure"?

Because direct requests from the browser to the server are being intercepted by Node, making the establishment of a secure connection between them impossible. However, the requests aren't made by the browser, they are made by Node. All https requests made through Node using this module are secure. This is evidenced by the connection property of the response object:

connection: TLSSocket {
    _tlsOptions: {
        secureContext: [SecureContext],
        requestCert: true,
        rejectUnauthorized: true,
    },
    _secureEstablished: true,
    authorized: true,
    encrypted: true,
}

You can think of the warning as a false positive.

Dependencies