combine-pagination 🦑

combine-pagination is a JavaScript library for paginating across multiple data sources at once, whilst retaining the sort order.

• Great for Infinity Scroll: easily support multiple data sources in your infinity scroll.
• Retain order: your data is always in order, even when it comes from different sources.
• Service agnostic: work with any data service, whether REST, GraphQL, Algolia or another.
• Mix-and-match services: mix data services as you see fit, making one query from GraphQL and one from Algolia.
• Efficient: only fetch data when needed for that data source.

Used in production at https://wellpaid.io.

Installation

npm i combine-pagination --save

or

yarn add combine-pagination

Quick examples

If you already understand the problem, here are some quick examples for paginating across multiple data sets in different scenarios.

Generic

Paginate data from two generic data sets.

const combinedGetters = combinePagination({
  getters: [page => getDataA(page), page => getDataB(page)],
  sortKey: "popularity"
});

const pageOne = await combinedGetters.getNext();
const pageTwo = await combinedGetters.getNext();

Algolia - custom sorting

Paginate data from two distinct algolia queries where the Algolia Index is sorted by a custom field. Each query is using a different keyword.

const index = algoliasearch({
  //...
}).initIndex("hats");

const combinedGetters = combinePagination({
  getters: [
    async page =>
      (await index.query({ page, hitsPerPage: 15, query: "Baseball cap" }))
        .hits,
    async page =>
      (await index.query({ page, hitsPerPage: 15, query: "Top hat" })).hits
  ],
  sortKey: "popularity"
});

const pageOne = await combinedGetters.getNext();
const pageTwo = await combinedGetters.getNext();

Algolia - default sorting

Paginate data from two distinct algolia queries where the Algolia Index is sorted using Algolia's default criteria.

This approach uses a custom sort method that attempts to match Algolia's sorting algorithm.

The sortAlgolia sort method used in this example is experimental. You might need to implement your own if using a custom ranking method.

import { sortAlgolia } from "combine-pagination";

const index = algoliasearch({
  getRankingInfo: true // Ask algolia for ranking info
  //...
}).initIndex("hats");

const combinedGetters = combinePagination({
  getters: [
    async page =>
      (await index.query({ page, hitsPerPage: 15, query: "Baseball cap" }))
        .hits,
    async page =>
      (await index.query({ page, hitsPerPage: 15, query: "Top hat" })).hits
  ],
  sort: sortAlgolia
});

The Problem

Suppose you have two data sets, modernHats and oldHats, and you want to combine them into one data set sorted by popularity called hats:

const modernHats = [
  {
    name: "Baseball Cap",
    popularity: 95
  },
  {
    name: "Beanie",
    popularity: 50
  },
  {
    name: "Flat Cap",
    popularity: 20
  }
];

const oldHats = [
  {
    name: "Top Hat",
    popularity: 85
  },
  {
    name: "Beret",
    popularity: 15
  },
  {
    name: "Bowler Hat",
    popularity: 9
  }
];

In this example, we’ll be paginating 2 results at a time. Let’s create a getter to paginate our data:

const getData = (data, page) => data.slice((page - 1) * 2, page * 2);

Note, in reality you probably already have a data getting with pagination support to retrieve the data via a server.

Now let's get our data. Without using combine-pagination, we might be tempted to just paginate each, sort and combine them. This is what NOT to do:

const modernHatsPage = getData(modernHats, 0);
const oldHatsPage = getData(oldHats, 0);

const hats = [...modernHatsPage, ...oldHatsPage].sort(
  (a, b) => a.popularity - b.popularity
);

This will result in hats that looks like this

[
  {
    name: "Baseball Cap",
    popularity: 95
  },
  {
    name: "Top Hat",
    popularity: 85
  },
  {
    name: "Beanie",
    popularity: 50
  },
  {
    name: "Beret",
    popularity: 15
  }
];

This looks fine, until you query the second page, which will look like this

[
  {
    name: "Flat Cap",
    popularity: 20
  },
  {
    name: "Bowler Hat",
    popularity: 9
  }
];

If we combine these results, you’ll notice that now the results are out of order. Sure, we could re-sort our entire data set, but this has some problems:

1. Reordering UI is confusing - if we’re rendering hats in a UI, such as an infinity scroll, it will cause the UI to reorder and confuse the user.
2. Inefficient sort - re-sorting the entire data set on each pagination is highly inefficient.
3. Unnecessary data request - depending on the order of the data, getting both data sets at once might be unnecessary, especially if a network request is involved.

The Solution

Using a technique (currently) called Framed Range Intersection, we can conservatively hold back trailing data from the first page that we think might overlap with subsequent pages. In the example above, it would mean holding back "Beret" until the next page is retrieved.

combine-pagination implements this technique. Let's try again using the above data set:

import combinePagination from "combine-pagination";

const combinedGetters = combinePagination({
  getters: [page => getData(modernHats, page), page => getData(oldHats, page)],
  sortKey: "popularity"
});

And query the first page:

const page = await combinedGetters.getNext();

Resulting in:

[
  {
    name: "Baseball Cap",
    popularity: 95
  },
  {
    name: "Top Hat",
    popularity: 85
  },
  {
    name: "Beanie",
    popularity: 50
  }
];

As expected, we only received three results. combine-pagination is only showing intersecting data, holding "Beret" back until it receives the next data set. Because of this, you can't define exactly how many results you want to receive. See Fuzzy Pagination.

The second time we run getNext(), we get the next set of data, but this time in the correct order:

[
  {
    name: "Flat Cap",
    popularity: 20
  },
  {
    name: "Beret",
    popularity: 15
  },
  {
    name: "Bowler Hat",
    popularity: 9
  }
];

combine-pagination noticed that "Beret", which was held back from the first set of results, intersects "Flat Cap" and "Bowler Hat", so has inserted it and sorted the page.

That's it. Each time you call getNext(), you'll retreive the next set of sorted data until the getters are exhausted.

Use cases

• Using infinity scroll across multiple data sources.
• Paginating across multiple Algolia queries, such as one geo location query and one not.
• Paginating across different services.

Framed Range Intersecting

Intersecting ranges is a technique for finding values that overlap in two sets of data. For example:

• Intersection of [0, 3] & [2, 4] is [2, 3]
• Intersection of [-1, 34] & [0, 4] is [0, 4]
• Intersection of [0, 3] & [4, 4] is empty set

combine-pagination uses a technique called Framed Range Intersecting (name is WIP), a type of intersecting that determines the leading data set, and intersects the other data sets within that.

Unlike normal range intersecting:

• The first value is the first value of the leading data set.
• The last value is either the last value of the leading data set, or the the last value of the intersecting data set that finishes first.
• Values in multiple data sets are duplicated.

For example:

• Framed Intersection of [0, 3] & [2, 4] is [0, 2, 3]
• Framed Intersection of [-1, 34] & [0, 4] is [-1, 0, 4]
• Framed Intersection of [0, 3] & [4, 4] is [0, 3]
• Framed Intersection of [0, 3] & [2, 4] & [1, 2] is [0, 1, 2, 2]

Fuzzy Pagination

Each time you execute getNext(), you can't be sure how many results you're going to receive. We call this Fuzzy Pagination, which returns 0 - n results for any given page with page size n. This technique is best suited for infinity scroll type use cases.

In normal pagination, you would receive n results for each page, only receiving 0 - n results on the final page.

License

MIT © Chris Villa