GitHub Scanner

Using @mktcodelib/graphql-fetch-all and adds IndexedDB storage and Web Workers on top of it. Install and use it in your project like this:

npm i @mktcodelib/github-scanner

cp node_modules/@mktcodelib/github-scanner/dist/github-scan-worker.js public/github-scan-worker.js

import { Scanner } from '@mktcodelib/github-scanner';

const scanner = new Scanner({ accessToken: "<access token>" });

// Fetch all data from paginated query
const result = await scanner.scan(query, variables);

// Fetch data with intermediate results using a callback
const result = await scanner.scan(
  query,
  variables,
  (intermediateResult) => {
    const { scanId, data, paginators, variables } = intermediateResult;
    // ...
  }
);

// Continue an unfinished scan
const result = await scanner.continueScan(scanId, (intermediateResult) => {
  // ...
});

Database

GitHub Scanner uses IndexedDB to store the fetched data in the browser. The database schema is defined in the db.ts file. Dexie.js is being used as a simple and efficient way to interact with IndexedDB.

You can access the database directly using the exported db instance.

import { db } from '@mktcodelib/github-scanner';

db.scans.toArray().then((scans) => {
  console.log(scans);
});

Web Workers

Each scan is performed in a web worker to avoid blocking the main thread. The worker file is provided by the package and needs to be copied to a location that is accessible by the browser, e.g.:

cp node_modules/@mktcodelib/github-scanner/dist/github-scan-worker.js public/github-scan-worker.js

If you want to use a different filename or path, you can pass it to the Scanner constructor:

const scanner = new Scanner({
  accessToken: "<access token>",
  workerPath: '/path/to/worker.js', // default: '/github-scan-worker.js'
});

The web worker file is responsible for fetching the data and storing it in the database. It communicates with the main thread using postMessage and onmessage to send and receive updates.