npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

fmr-searchkit

v2.0.5

Published

Sets up a search service for your feathers-mongo server that provides a rich search experience while reusing the security and business logic in your Feathers hooks. Use with the React library FMR-SearchKit-React.

Downloads

36

Readme

FMR-SearchKit

Build rich mongodb search interfaces with cross collection lookups while preserving the security and business logic in your FeathersJS hooks.

The Problem

FeathersJS imposes great structure on a web application project with the hooks system. In the before hooks for a service one can transform data, check permissions and perform other tasks appropriate before a DB operation, and in the after hooks one can perform side effects and implement further transformations.

Let's take query restriction and data redaction as examples of each. A service can have a before hook that restricts the user to their own records by adding their _id to the query. And anther service might have an after hook to redact passwords or other credentials.

Now what happens if you want to build a client side web page or app screen that uses the full power of the MongoDB aggregation framework as applied to your collections? This goes beyond simple CRUD operations as supported by feathers-mongodb and so you'll need to implement your own servce. But in this service you'll need to re-implement all of that same restriction and redaction logic, or you'll need to make sure your users have special permission to use the service. All of this leads to additional highly coupled code in your project, leading to maintainability problems and security leaks and eventually higher costs down the road.

The Solution

This is what FMR-Searchkit is for. It's a drop-in solution for building a highly functional cross-collection search backed by a full MongoDB aggregation that re-uses the before and after hoooks from your feathers services while building the aggregations. Does your user service redact passwords in an after find hook? Searches of your user collection will too, right out of the box - whether user data appears as the main rows of your search as names on a facet filter. Does your personal_info collection restrict people to their own records in a before find hook? So will FMR-Searchkit searches of that collection, right out of the box. And so on.

See it in action

You can use the searchkit right now at my site The Logic Resource Center where it powers the Sequent, Derivation and Tableaux searches.

Installation

Install fmr-searchkit with your favorite package manager.

Use

The module assumes the following

  • In the folder for each service you will expose through the searchkit there is a standard Feathers hooks file
  • In the each of the same folders there is a BSON schema file called schema for the Mongo collection backing the service.
  • Make the db connection available at app.mongodb.db as below

In app.ts:

import mongodb, { ObjectId } from 'mongodb'
import { searchkit } from 'fmr-searchkit'

const client = await MongoClient.connect(/* your connection args */)

app.mongodb = { db: client.db() }

app.configure(
  searchkit({
    ObjectId,
    services: [
      'service1',
      'service2',
      'service3',
      'service4'
    ],
    servicesPath: `${__dirname}/services/`
  })
)

You'll likely want to put some of your own hooks on the new search and schema services this creates, but other than that this is really it - you now have a highly functional search service that reuses the before and after all and find hooks on your services, even on other searches that run lookups and subqueries to the underlying collections.

API

The search service takes a POST or Feathers create with a JSON stringified object specifying the search. The properties of this object:

| Prop | Required | Default | Comment | | -------- | ------- | ------- | --------- | | collection | true | none | base collection from which results will be drawn | | sortField | false | none | | | sortDir | false | none | asc or desc | | page | false | 1 | | | pageSize | false | 100 | | | include | false | none | Returns all fields when unset | | includeSchema | false | false | Return base schema with the response | | filters | false | none | Array of Filter. See ./types | | charts | false | none | Array of Chart. See ./types | | lookup | false | none | Keyed object of Lookup. See ./types | | id | false | none | Not used server-side |

Use

So how do you use it? This is where FMR-Searchkit-React comes in. It allows your to build a wide variety of client- and server-rendered searches with React 18 and full Next 13 support. As you can see here the searchkit ships now with about 20 filters and charts. Instructions on how to use them are at FMR-Searchkit-React

A note on indexing, $lookup and denormalization

The sad truth is that Mongo shines on single collections when the queries have been anticipated with good indexing and struggles with lookups to other collections when you get beyond maybe 20,000 records, often in ways that defy indexing. So what's the point?

You should think of the lookup functionality here as a tool for prototying and offline layouts. There are plenty of applications in the course of a normal web development career where collections are small or users tolerate longer response times. Launching a new product or feature at all but the biggest companies would be an example, as would basic offline analytics functionality for employees.

When you need real performance you need to denormalize. Luckily the plan is simple: after every mutation of the records in your collection you run all necessary lookups and write the populated record to another collection. Just change your layout to search the new collection and you're done. The live data stays small and queries of the denormalized data perfom. So the lifecycle is like this:

  • prototype and explore with free use of lookup.
  • at launch, index properly for the layouts you actually build - all filters will need to be supported by an index and in fact indexes will increase factorially in number with filters so choose wisely.
  • when indexing still isn't getting you by drop in the denormalizeAndUpsert hook provided here and adjust the consuming searchkit layout to point at the new collection with appropriate adjustments.