npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

@researchgate/mongoose-avro-schema-generator

v1.1.6

Published

Generates Apache Avro schemas from mongoose schemas.

Downloads

22

Readme

A node module that generates Apache avro schemas from mongoose schemas.

Getting Started

Prerequesites

The Mongoose Avro Schema Generator requires node >= 9.2.0. The only dependency is mongoose >= 5.0.9, a connection to MongoDB is not necessary.

Installation

Using yarn:

yarn add @researchgate/mongoose-avro-schema-generator

Using npm:

npm install @researchgate/mongoose-avro-schema-generator

Quick Start

In order to generate schemas for all registered mongoose models we import the module, create a new MongooseAvroSchemaGenerator and run the generate() method.

Let's first register a simple mongoose model.

let schema = new Schema({
    something: String,
});
mongoose.model('mySchema', schema);

Now we instantiate the Mongoose Avro Schema Generator with a the mongoose instance.

const Generator = require('mongoose-avro-schema-generator');
const mongooseAvroSchemaGenerator = new Generator(mongoose);

Then mongooseAvroSchemaGenerator.generate() will output an array of all generated schemas.

[
    {
        "dbcollection": "myschemas",
        "dbtype": "mongodb",
        "fields": [
            {
                "name": "something",
                "type": ["null", "string"],
                "default": null
            },
            {
                "name": "_id",
                "type": [
                    "null",
                    {
                        "subtype": "objectid",
                        "type": "string"
                    }
                ],
                "default": null
            },
            {
                "name": "__v",
                "type": ["null", "double"],
                "default": null
            }
        ],
        "name": "mySchema",
        "namespace": "some.namespace"
    }
]

Please note that the schema also contains the auto-generated fields _id and _v. In the following chapters we will have a look at further details.

Usage

Generate avro schemas for all registered mongoose models:

mongooseAvroSchemaGenerator.generate();

Restrict the schema generation to a set of models:

mongooseAvroSchemaGenerator.generate(['User', 'Transaction']);

Override the default "mongoose" namespace by setting the second parameter in the constructor:

new Generator(mongoose, 'some.custom.namespace');

Mapping

This section will explain the mapping from mongoose types to avro schemas.

Primitive Types

The following table lists all the primitive types and their mapped equivalent in the avro schema.

| Native/Mongoose Type | Avro Type | | -------------------- | --------- | | String | "string" | | Schema.Types.String | "string" | | Boolean | "boolean" | | Number | "double" | | Buffer | "bytes" | | Date | { "type": "long", "subtype": "date" } | | Schema.Types.ObjectId | { "type": "string", "subtype": "objectid" } |

Please note that Date and Schema.Types.ObjectId are mapped to objects with a more specific subtype.

Warning: The Schema.Types.Mixed type is not supported. The same applies to the equivalent empty object literal {} or the empty array []. Trying to generate a schema from a model with such a type will result in an error.

Arrays

Arrays are mapped to the avro type array. For example the mongoose field some: [Number] is getting mapped to the following field in the avro schema:

{
    "name": "some",
    "type": [
        "null", 
        {
            "type": "array",
            "items": ["null", "double"]
        }
    ],
    "default": null
}

We will later see how the default and null values are generated.

Embedded Documents

Embedded documents are mapped to avro records. For example some: { thing: String } is represented by following avro definition:

{
	"name": "some",
	"type": [
		"null",
		{
			"name": "someEmbedded",
			"type": "record",
			"fields": [{
				"name": "thing",
				"type": ["null", "string"],
				"default": null
			}]
		},
		"record"
	],

	"default": null
}

Please note the name "someEmbedded" of the embedded record which is autogenerated from the parent object with name "some". This is necessary because in an avro schema embedded documents always need a name.

Records can also appear as array items. Like with embedded documents the name will be inferred from the parent object. For example some: [{ thing: String }] will be mapped to an avro schema containing the following field:

{
	"name": "some",
	"type": [
		"null",
		{
			"type": "array",
			"items": {
				"type": [
					"null",
					{
						"name": "someItemEmbedded",
						"type": "record",
						"fields": [{
							"name": "thing",
							"type": ["null", "string"],
							"default": null
						}]
					}
				]
			}
		}
	],
	"default": null
}

Please note the name "someItemEmbedded" of the embedded record which is autogenerated from the parent object with name "some". The same applies for records embedded in arrays of arrays which will have the suffix "ItemItem". In general every array layer appends another "Item" suffix.

Attributes

The Mongoose Avro Schema Generator makes use of the mongoose attributes required and default.

Nullable

The avro type null will be included automatically if no required : true attribute is set for a field in mongoose. Hence the avro schema for the following mongoose schema

let schema = new Schema({
    something: { type: String },
});
mongoose.model('mySchema', schema);

will have a union type of null and string

{
    "name": "something",
    "type": ["null", "string"],
    "default": null
}

However if we add the required attribute

let schema = new Schema({
    something: { type: String, required: true },
});
mongoose.model('mySchema', schema);

the type will be restricted to string.

{
  "name": "something",
  "type": "string"
}

We note that also the default value of null has been removed.

Default Values

Default values are either defined explicitly or are implicitly inferred.

Explicit Defaults

The mongoose schema

new Schema({
    something: { type: String, default: 'foo' },
});

will be transformed into an avro schema containing the following field

{
    "name": "something",
    "type": ["null", "string"],
    "default": "foo"
}

Arrays will always have a default of null.

Important: Functional default values are not supported and will be ignored, i.e. treated as if the field doesn't have any default.

Implicit Defaults

If a field can be null and no explicit default value is defined, the default will be set to null.

Complex Example

Let's register a schema with mongoose.

const mongoose = require('mongoose');
const mongooseAvroSchemaGenerator = require('mongoose-avro-schema-generator');
mongooseAvroSchemaGenerator.init(mongoose);

let schema = new Schema({
    something: { type: [[Number]], default: ['foo'] },
    else: [String]
});
mongoose.model('mySchema', schema);

Then mongoosevroSchemaGenerator.generate(['mySchema'], { namespace: 'some.namespace' }) will return the following avro schema:

[
    {
        "dbcollection": "myschemas",
        "dbtype": "mongodb",
        "type": "record",
        "fields": [
            {
                "name": "something",
                "type": [
                    "null",
                    {
                        "type": "array",
                        "items": [
                            "null",
                            {
                                "type": "array",
                                "items": ["null", "string"]
                            }
                        ]
                    }
                ],
                "default": null
            },
            {
                "name": "else",
                "type": [
                    "null",
                    {
                        "type": "array",
                        "items": ["null", "string"]
                    }  
                ],
                "default": null
            },
            {
                "default": null,
                "name": "_id",
                "type": [
                    "null",
                    {
                        "subtype": "objectid",
                        "type": "string"
                    }
                ]
            },
            {
                "default": null,
                "name": "__v",
                "type": ["null", "double"]
                
            }
        ],
        "name": "mySchema",
        "namespace": "some.namespace"
    }
]

Running tests

Mocha tests for Mongoose Avro Schema Generator can be found in /test. A yarn job is configured to run those tests using yarn test.