npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

paging-async-generator

v1.0.2

Published

Provides a nice way to iterate over asynchronous calls without waiting for them while managing the size of the concurrent calls

Downloads

3

Readme

paging-async-generator

Provides a nice way to iterate over asynchronous calls without waiting for them while managing the size of the concurrent calls

This library has two special use cases that most has encountered when dealing with web pages or programs:

  1. Having to load lots of ressources from remote servers or asynchronous source and facing the contention on the client side or server side
  2. Having to load data in advance of the user needing it in a paginated environment

This library is relying on async generator to allow the developer to freely iterate at its own speed and need through the content of the loaded resource

Install

like any other library

npm install paging-async-generator

Get started

Available in typescript, the library will be demonstrated in javascript for everybody convenience, assuming that typescript developpers will be able to translate.

const {pagingGenerator: pIt} = require('paging-async-generator');

const http = axios.default.create({validateStatus: s=>s>=200 && s<300});
// replace with your function to access external resource
// it is an asynchronous function can optionally can receive the page and index of the call progress
async function todo(page, index) {
  const res = await http.get(`https://jsonplaceholder.typicode.com/todos/${page*10+index+1}`)
  return res.data && res.data.id? res.data : undefined;
}

// call the todo function by blocks of 5 items per page and make them available in the for
// continue until the return value of the todo function is undefined
for await (let res of pIt(todo, 5)) {
  // res contains the value, page, index and possibly and error
  if (res.error) console.error("on", res.page, res.index, "fail", res.error.message)
  else console.log("on", res.page, res.index, "task", res.value.id, res.value.title)
}
// this should list the 200 items of the todos' collection.

Controlled iteration

With the same above example, say we need to read the items on reaction to the user action

const {pagingGenerator: pIt} = require('paging-async-generator');

const http = axios.default.create({validateStatus: s=>s>=200 && s<300});
// replace with your function to access external resource
// it is an asynchronous function than can optionally receive the page and index of the call progress
async function todo(page, index) {
  const res = await http.get(`https://jsonplaceholder.typicode.com/todos/${page*10+index+1}`)
  return res.data && res.data.id? res.data : undefined;
}

// Page initialization
const pageSize = 10;
const cursor = pIt(todo, pageSize);

const el = document.getElementById("your-button")
el..addEventListener("click",  async ()=>{
  // executes on each user click
  const res = await cursor.next();
  if (res.done) return; // you have no more item to read
  if (res.value.error) console.error("The call failed", res.value.error.message)
  else console.log("new task", res.value.id, res.value.title)
})

You can easilly replace the click event by a page scrolling or anything else that fits your UI

Execution logic

The mode of execution of the iterator returned by pIt has the following logic

  1. initiate a page
    • launch pageSize asynchronous request in //
    • wait for the first available value and returns it (yield)
    • wait for subsequent values in the submitted result and return (yield) them when requested
    • if all values of the pages have been retrieved, finish the page
    • if a returned value was the last (islast()) then finish the page and finish the iteration
  2. if the iteraton is not finished, prepare a new page and continue on 1.

Yielded values are made of the actual result from the asyncFunc, possible error caught, page number and index in page. Errors can be silently processed (ie returned) or thrown depending on the options.

Configuration options

The function creating the iterator pIt(asyncFunc, pageSize, options) accepts a 3rd parameter options that defaults as follow

const defaultOptions:PagingOptions = {
  isLast:(v)=>v==undefined,         // called after each call of the `asyncFunc` with the result to determine if the value should be considered as the last to retrieved
  isError:(v)=>v instanceof Error,  // called after each call of the `asyncFunc` with the result to determine if the value is to be considered as an error
  keepOrder: false,                 // tells the iterator to return the result in the order they were called. If false, the results in a page appear in the order they completed (faster reponses first). If true, the results appear in the order they were made.
  maxErrors: -1,                    // the max numbers of errors to accept before the iterator returns an error. -1 means no error counting. 
  throwOnMaxError: false            // indicates to throw if more than maxErrors have been detected or to quietly finish the iteration.
}

Example using the errors configuration Using the maxErrors and throwOnMaxError is usefull when dealing with low quality servers or unexpectedly unavailable servers. It is often hard to make the difference between a connection problem and a server being down. So with this configuration you can keep testing the connection a few times before really failing.

This example has a function randomFail() that returns a value 90% of the time or an exception

const timer = require('timers/promises')
async function randomFail(page, index) {
  const random = Math.random()*1000;
  await timer.setTimeout(random)
  if (page>5) return undefined; // stop returning after 5 pages
  if (random < 100) throw new Error("Failed");
  return random;
}

const {pagingGenerator: pIt} = require('paging-async-generator');

try {
  const options = {
    maxErrors: 3,
    throwOnMaxError: true
  }
  for await (let res of pIt(randomFail, 10, options)) {
    if (res.error) console.error("error"); // error 1, 2 and 3 can captured here
    else console.log("Value", res.value);
  }
} catch(error) {
  console.error("Catch part", error.message); // too many errors raised just after the 3rd error
}

This will output something like this

0 9 Error
0 0 Value 182.48736758281137
0 1 Value 220.38580312813582
0 4 Value 308.9175972240954
0 3 Value 400.9297899326623
0 5 Value 488.29201828411703
0 2 Value 535.7719212554144
0 8 Value 568.9978143365527
0 7 Value 951.4650640197817
0 6 Value 985.1677185623045
1 0 Error
1 7 Value 409.609336297486
1 2 Value 482.0072511571525
1 1 Value 535.9657542279799
1 6 Value 573.3811808896756
1 8 Value 703.6885387235554
1 4 Value 722.9073244025273
1 9 Value 734.6408818444008
1 5 Value 866.4749323388459
1 3 Value 907.5992228641494
2 8 Error
Catch part max errors reached

Passing an array of functions

In some case you may need to call different functions for each iteration while not necessarilly processing the response in the order they appear in the process.

Contrary to Promise.all() that calls all and only return when all are called, with the paging-async-generator you can start processing before all responses are available

In the below example, let's assume functions that are long to compute and are executed on cloud but some parts can be run in //

function add(x, y) {
  return new Promise (r=>{
    setTimeout(()=>r(x+y), 100)
  })
}
function mul(x, y) {
  return new Promise (r=>{
    setTimeout(()=>r(x*y), 200) // multiplication takes longer
  })
}

const array = [
  add.bind(null, 1, 2), // 3
  add.bind(null, 5, 6), // 11
  mul.bind(null, 10, 10), // 100
  add.bind(null, 100, 10), // 110
];

for await (let v of pIt(array, array.length)) {
  console.log("computed:", v.value)
}

outputs: (note that the multiplication is displayed after the additions since it takes more time to compute and the results are not ordered)

computed: 3
computed: 11
computed: 110
computed: 100