Gatsby cdn search plugin

This plugin is in beta and still in work

Give us any feedback, open issues for any questions or ideas

Mongo query compatible search plugin for gatsby.

It is a no cost way to add search to your site.

Key technology is http2/http3 protocols, CDN, mongo-like query and N-GRAM search.

The plugin supports mongo-like query syntax with custom n-gram search.

Idea of this plugin is simple.

• calculated indices in build phase of gatsby apps. (n-gram, text-lex, simple)
• split the indices by chunk
• create range diapason indices as "table of contents" the chunk
• save all chunk and "table of contents" on CDN as assets
• in runtime plugin restore indices and efficiently on-demand loaded chunk over http2 protocol
• http2 multiplexing multiple requests over a single TCP connection.

The plugin has native support React via Hook "useCdnCursorQuery".

Also, you can trace your request with log-level.

import { useCdnCursorQuery, log } from 'gatsby-cdn-search-plugin'
log.enableAll(); // full logging 

Live demo

Kaggle dataset "Used Car Auction Prices" 500 000 row

live demo

source code on github

source of data

Plugins config

    plugins = [
    'gatsby-plugin-offline',
    {
      resolve: require.resolve("./cdn-indice-plugin"),
      options: {
        id: 'cars',
        chunkSize: 6000,
        dataChunkSize: 60,
        indices: [  
          {
              id: 'model',
              column: 'model',
              type: 'simple', 
          },
          { id: 'make', column: 'make' },
          { id: 'year', column: 'year' },
          { id: 'state', column: 'state' },
          { 
              id: 'id-state',
              column: 'state',
              algoritm: 'english',
              type: 'text-lex' 
          },
          { 
              id: 'ngram',
              type: "n-gram",
              actuationLimit: 1,
              actuationLimitAuto: false, 
              gramLen: 3, 
              toLowcase: true, 
              algoritm: 'english',
              stopWords: ["and"],
              columns: ['model', 'make', 'color'] 
          }
        ],
        idAttr: 'id',
        normalizer: ({ data }) => { 
          return data.recentCars
            .map(( {id, ...node} ) => ({ id: id.replace('Car__',''), ...node }));
        },
        graphQL: `query MyQuery { 
          recentCars(cursor: 0, limit: 500000){
              id
              color
              make
              mmr
              model
              seller
              sellingprice
              state
              transmission
              trim
              vin
              year
        }
      }`
      }
    }
    ]

Usage stateful React hook

import { useCdnCursorStatelessQuery, log } from 'gatsby-cdn-search-plugin'
log.enableAll(); // full logging 


const makeQuery = (search) => { // Different query strategy. It is depends of length search word
  if (search.length >= 4) {
    return { $ngram: search }; // n-gram 
  } else if (!!search.length) {
    return {
      $or: [
        { model: { $regex: new RegExp(`^${search}`, 'i'), }, }, // regexp by two columns
        { make: { $regex: new RegExp(`^${search}`, 'i'), }, }
      ],
    };
  } else {
    return undefined;
  }
}
const [state, dispatch] = useReducer(reducer, initialState);

const query = useMemo(() => makeQuery(state.search), [state.search]);

const {hasNext, next, fetching, all, page} = useCdnCursorStatefulQuery('cars', query, {year: 1}, 0, 30); // hook return cursor of data

const load = useMemo(() => {
    if (hasNext && !fetching) {
          next(); // load next slice of data 
    }}, [hasNext, fetching, next]);

Usage stateless React hook (more complicated)

import { useCdnCursorStatelessQuery, log } from 'gatsby-cdn-search-plugin'
log.enableAll(); // full logging 

const initialState = {
    loading: false,
    search: '',
    list: [],
    page: 0,
};

function reducer(state, action) {
    switch (action.type) {
        case 'pageUp':
            return { ...state, page: state.page + 1 }
        case 'type':
            return { ...state, search: action.value }
        case 'loading':
            return { ...state, loading: true }
        case 'load':
            return { ...state, loading: false, list: action.list, page: 0 };
        case 'indice':
            return { ...state, indice: action.value }
        case 'loadMore':
            return { ...state, loading: false, list: [...state.list, ...action.list] };
        default:
            throw new Error();
    }
}

const makeQuery = (search) => { // Different query strategy. It is depends of length search word
  if (search.length >= 4) {
    return { $ngram: search, year: { $lte: 2014 } }; // n-gram and  year <= 2014
  } else if (!!search.length) {
    return {
      $or: [
        { model: { $regex: new RegExp(`^${search}`, 'i'), }, }, // regexp by two columns
        { make: { $regex: new RegExp(`^${search}`, 'i'), }, }
      ],
    };
  } else {
    return { year: { $lte: 2014 } }; // only date predicate
  }
}
const [state, dispatch] = useReducer(reducer, initialState);

const query = useMemo(() => makeQuery(state.search), [state.search]);

const cursor = useCdnCursorQuery('cars', query, {year: 1}, 0, 30); // hook return cursor of data

  useEffect(() => {
    (async () => {
        let list = await cursor.next(); // load first slice of data
        dispatch({ type: 'load', list });
    })();
  }, [state.search, cursor])

  useEffect(() => {
    (async () => {
      if (await cursor.hasNext()) {
        let list = await cursor.next(); // load next slice of data 
        dispatch({ type: 'loadMore', list })
      }
    })()
  }, [state.page]);

Usage find api exactly

      import { restoreDb } from 'gatsby-cdn-search-plugin'

      const db = await restoreDb('cars');
      let result;
      if (search.length >= 4) {
        result = await db.find({ $ngram: search, year: { $gte: 2014 } }, undefined, 0, offset);
      } else if (!!search.length) {
        result = await db.find({ model: { $regex: new RegExp(`^${search}`, 'i'), }, year: { $gte: 2014 } }, undefined, 0, offset);
      } else {
        result = await db.find({ year: { $gte: 2014 } }, undefined, 0, offset);
      }

Usage cursor api exactly

      import { restoreDb } from 'gatsby-cdn-search-plugin'

      let cursor;
      const searchFetch = async (search, skip = 0, limit = 30) => {
        const db = await restoreDb('cars');
        if(cursor){
          cursor.finish();
        }
        if (search.length >= 4) {
          cursor =  db.cursor({ $ngram: search, year: { $gte: 2014 } }, undefined, skip, limit);
        } else if (!!search.length) {
          cursor =  db.cursor({
            $or: [
              { model: { $regex: new RegExp(`^${search}`, 'i'), }, },
              { make: { $regex: new RegExp(`^${search}`, 'i'), }, }
            ],
          }, undefined, skip, limit);
        } else {
          cursor = db.cursor({ year: { $gte: 2014 } }, undefined, skip, limit);
        }
        return await cursor.next();
      }

Plugin options

| Options name | Type | Required | Default value | Description | |---------------|---------------------------------------------------------------------------|----------|---------------|------------------------------------------------------------------------------------------------------------------------------------| | id | String | True | None | Unique id database collection. The first parameter in React Hook useCdnCursorQuery. | | chunkSize | Number | False | 500 | Indices chunk size. This affects the number of indices files. You should select this parameters depends of size of your collection | | dataChunkSize | Number | False | 25 | Data chunk size. This affects the number of data files. | | idAttr | String | True | None | Primary id attribute name. | | normalizer | Function({data: any}): Row[] | True | None | It is callback for handle data from graphQl | | graphQL | graphQL string | True | None | Graphql query for fetching data | | indices | Array<Union<NgramIndicesOption ,TextLexIndicesOption ,SimpleIndicesOption>> | True | None | Secondary indices. Add column available to search. |

NgramIndicesOption

| Options name | Type | Required | Default value | Description | |--------------------|----------|----------|---------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | type | "n-gram" | True | simple | Indices using the N-Gram algorithm for search with typos. Also, support "The Porter Stemming Algorithm" for zipping indices. You can search by this column with specific not Mongo predicate. {$ngram: "search"} | | id | String | True | None | Unique id of indices. Uses as operator name in the search. Query example for id "ngram" is {$ngram: "search"} | | actuationLimit | Number | True | None | Minimum match n-gram in search. [color] -> [col, olo, lor] | | actuationLimitAuto | Boolean | False | False | If option equal true option actuationLimit doesn't work. Actuation limit n-gram calculates auto by the size of the search word. | | gramLen: 3 | Number | False | 3 | Size of. Example if n-gram equal 3 "color" was split to "col", "olo", "lor" | | toLowcase | Boolean | False | False | Case sensitive search | | stem | String | False | None | Preprocess indices with "The Porter Stemming Algorithm". Available values 'english', 'russian', ... | | columns | String[] | True | None | Indexing columns | | stopWords | String[] | False | None | The words exclude for search |

SimpleIndicesOption

| Options name | Type | Required | Default value | Description | |--------------|------------|----------|---------------|-----------------------------------------------------------------------------------------------------------------------| | type | "text-lex" | True | simple | The Porter Stemming Algorithm. You can search by this column with specific not Mongo predicate. {$lex: "search"} | | id | String | True | None | Unique id of indices. Uses as operator name in the search. Query example for id "ngram" is {$lex: "search"} | | algoritm | String | False | None | Preprocess indices with "The Porter Stemming Algorithm". Available values 'english', 'russian', ... | | column | String | True | None | Indexing column |

TextLexIndicesOption

| Options name | Type | Required | Default value | Description | |--------------|----------|----------|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | type | "simple" | True | "simple" | Default value simple indices by one column only. You can search by this column with regular Mongo predicate lt, gt, eq, ... etc. Also, support regexp by "start with regexp" | | id | String | True | None | Unique id of indices. Uses as operator name in the search. Query example for id "ngram" is {$lex: "search"} | | column | String | True | None | Indexing column |