Loopback Aggregate mixin for MongoDB

Give models the ability to query native MongoDB aggregates and build instances from results.

Highlights

• Accepts both Loopback filter's features and pipeline stages, it will merge in a single parsed pipeline to aggregate.
• Accepts relations' fields within the root where, it will be handled as $lookup stages.
• Refactor the logic from Loopback which is responsible for building the model instances and take advantage of it.
• Supports both callbacks and promises.

This Loopback mixin is intended to be used together with MongoDB connector. Works for Loopback 2 and 3.

How to install

Install the package through NPM

npm i -S @aliatech/loopback-mongo-aggregate-mixin

Install the package through Yarn

yarn add --prod @aliatech/loopback-mongo-distinct-mixin

Basic configuration

Include the mixin in server/model-config.json. Example for Loopback 3:

{
  "_meta": {
    "sources": [
      "loopback/common/models",
      "loopback/server/models",
      "../common/models",
      "./models"
    ],
    "mixins": [
      "loopback/common/mixins",
      "../node_modules/@aliatech/loopback-mongo-aggregate-mixin/lib",
      "../common/mixins"
    ]
  }
}

Enable the mixin in your model definition, ie person.json.

{
  "name": "Person",
  "properties": {
    "name": "string"
  },
  "mixins": {
    "Aggregate": true
  }
}

Usage

Invoke aggregate method passing either:

• A regular Loopback filter (where, fields, include, order, skip, limit)
• An aggregate pipeline
• A combination of both

Basic example

Find a random sample of 3 persons born after 1980:

app.models.Person.aggregate({
  where: {birthDate: {gt: new Date('1980')}},
  aggregate: [{$sample: {size: 3}}],
}, (err, persons) => {
  if (err) return next(err);
  // persons are Person model instances
});

Find where relation properties

Relation properties can be specified in the "where" criteria using dot notation. $lookup stages will be automatically generated to reach those relations and filter the root documents by such criteria. it works like a "LEFT JOIN" feature, however it's still necessary to add the "include" filter if you require the relation to be hydrated.

Example: Bring persons who are part of a team in which there is some person who is born after 2001

app.models.Person.aggregate({
  where: {'team.persons.birthDate': {$gt: new Date('2001')}},
}, (err, persons) => {
  if (err) return next(err);
  // persons are Person model instances
});

Note: It works for hasOne, belongsTo and hasMany. Filtering by embedded properties is not affected and continues to work as usual.

Do not build instances

Some queries are intended to retrieve data that can not be transformed into model instances. aggregate method will attempt to build instances by default, but this behavior can be disabled passing an options object {build: false} as second argument.

Example: Bring count of persons by company

app.models.Person.aggregate({
  aggregate: [{
    $group: {
      _id: '$companyId',
      total: {$sum: 1},
    },
  }],
}, {build: false}, (err, groups) => {
  if (err) return done(err);
  // Each group should be a plain object with just 'id' and 'total' attributes  
});

Build instances on demand

The aggregate result often needs some processing before building the model instances. It's possible to postpone the build phase until the models' data are resolved.

Example: Bring the persons count together with a specific page

Person.aggregate([{
  group: {
    _id: null,
    total: {$sum: 1},
    objects: {$push: '$$ROOT'},
  },
}, {
  project: {
    total: 1,
    items: {$slice: ['$objects', pageStart, pageLength]},
  },
}], {buildLater: true}, (err, [data, build]) => {
  if (err) return next(err);
  // data is a plain structure {total, items} where items is an array of documents, not model instances. 
  build(data.items, (err, persons) => {
    if (err) return next(err);
    // now you got persons as Person model instances
  });
});

• In this case, model documents are not brought as root result, so we could disable the automatic building by just passing the option {build: false}, but in this case, what we really need is the option {buildLater: true}.
• The difference is that buildLater will provide us a build function (together with native documents) to invoke by our hand . Person instances will be finally obtained by calling such function passing data.items.
• Build on demand feature it's available as a model static method Model.buildResult.

Note: Pipeline array can be directly passed as argument. Also stage names can obviate "$" character.

GeoNear example

Combine regular "where" with $geoNear stage. $geoNear will be moved to the pipeline head as MongoDB requires.

app.models.Company.aggregate({
  where: {sector: 'Software'},
  aggregate: [{
    $geoNear: {
      near: {type: 'Point', coordinates: [-0.076132, 51.508530]},
      distanceField: 'distance',
      maxDistance: 5000, // 5Km.
      spherical: true,
    },
  }],
}, (err, companies) => {
  if (err) return done(err);
  // companies are Company model instances
});

Promise support

Methods aggregae and buildResult support either callback or promise usage. All the examples above are made with callbacks. Below it's shown how it's made with promise style.

Example: Find a random sample of 3 persons born after 1980:

app.models.Person.aggregate({
  where: {birthDate: {gt: new Date('1980')}},
  aggregate: [{$sample: {size: 3}}],
}).then((persons) => {
  // persons are Person model instances
}).catch((err) => {
  // handle an error
});

Same example using await

try{
  const persons = await app.models.Person.aggregate({
    where: {birthDate: {gt: new Date('1980')}},
    aggregate: [{$sample: {size: 3}}],
  });
} catch(err) {
  // handle an error
}

Advanced configuration

Enable the mixin passing an options object instead of just true.

Available options:

| Option | Type | Required | Description | | --------------------- | ----------| --------- | ----------------- | | mongodbArgs | object | optional | Set defaults for MongoDB aggregate command options (default {}). Check the official documentation | | build | boolean | optional | Whether to automatically build model instances from aggregate results by default. (default true) | | buildOptions | object | optional | Set defaults for building process options (default {notify: true}) | | buildOptions.notify | boolean | optional | Whether to notify model operation hooks on build by default (default true) |

Any of these options can be replaced on the fly with the following syntax:

app.models.Person(filter, options, callback);

The options argument will be timely merged with the defaults for a single call.

Example: Allow MongoDB to use disk

This is a MongoDB aggregate command option that prevent memory issues on large queries. It can be enabled by default as follows:

{
  "name": "Person",
  "properties": {
    "name": "string"
  },
  "mixins": {
    "Aggregate": {
      "mongodbArgs": {
        "allowDiskUse": true
      }
    }
  }
}

Or just enable the option on the fly for a single call:

app.models.Person(filter, {mongodbArgs: {allowDiskUse: true}}, callback);

Debug

Prepend DEBUG environment when running server or tests to display what pipelines are being sent to MongoDB:

DEBUG=loopback:mixins:aggregate node . # Run server with debug

Testing

Install develop dependences

npm i -D # If you use NPM
yarn install # If you use Yarn

Execute tests

npm test # Without coverage check
npm run test-with-coverage # With coverage check

Credits

Inspired by https://github.com/BoLaMN/loopback-mongo-aggregate-mixin

Developed by Juan Costa for ALIA Technologies