Mongodb database adapter for Yjs

Persistent MongoDB storage for y-websocket server. You can use this adapter to easily store and retrieve Yjs documents in/from MongoDB.

Notes:

• This was once a fork of the official y-leveldb but for MongoDB
• This package is not officially supported by the Yjs team.

Use it (Installation)

You need Node version 16 or newer.

It is available at npm.

npm i y-mongodb-provider

Simple Server Example

There are full working server examples in the example-servers directory.

import http from 'http';
import { WebSocketServer } from 'ws';
import * as Y from 'yjs';
import { MongodbPersistence } from 'y-mongodb-provider';
import yUtils from 'y-websocket/bin/utils';

const server = http.createServer((request, response) => {
	response.writeHead(200, { 'Content-Type': 'text/plain' });
	response.end('okay');
});

// y-websocket
const wss = new WebSocketServer({ server });
wss.on('connection', yUtils.setupWSConnection);

/*
 * y-mongodb-provider
 *  with all possible options (see API section below)
 */
const mdb = new MongodbPersistence(createConnectionString('yjstest'), {
	collectionName: 'transactions',
	flushSize: 100,
	multipleCollections: true,
});

/*
 Persistence must have the following signature:
{ bindState: function(string,WSSharedDoc):void, writeState:function(string,WSSharedDoc):Promise }
*/
yUtils.setPersistence({
	bindState: async (docName, ydoc) => {
		// Here you listen to granular document updates and store them in the database
		// You don't have to do this, but it ensures that you don't lose content when the server crashes
		// See https://github.com/yjs/yjs#Document-Updates for documentation on how to encode
		// document updates

		// official default code from: https://github.com/yjs/y-websocket/blob/37887badc1f00326855a29fc6b9197745866c3aa/bin/utils.js#L36
		const persistedYdoc = await mdb.getYDoc(docName);
		const newUpdates = Y.encodeStateAsUpdate(ydoc);
		mdb.storeUpdate(docName, newUpdates);
		Y.applyUpdate(ydoc, Y.encodeStateAsUpdate(persistedYdoc));
		ydoc.on('update', async (update) => {
			mdb.storeUpdate(docName, update);
		});
	},
	writeState: async (docName, ydoc) => {
		// This is called when all connections to the document are closed.
		// In the future, this method might also be called in intervals or after a certain number of updates.
		return new Promise((resolve) => {
			// When the returned Promise resolves, the document will be destroyed.
			// So make sure that the document really has been written to the database.
			resolve();
		});
	},
});

server.listen(port, () => {
	console.log('listening on port:' + port);
});

API

persistence = MongodbPersistence(connectionLink: string, options: object)

Create a y-mongodb-provider persistence instance.

import { MongodbPersistence } from 'y-mongodb-provider';

const persistence = new MongodbPersistence('connectionString', {
	collectionName,
	flushSize,
	multipleCollections,
});

Options:

• collectionName
  • Name of the collection where all documents are stored
  • Default: "yjs-writings"
• flushSize
  • The number of transactions needed until they are merged automatically into one document
  • Default: 400
• multipleCollections
  • When set to true, each document gets an own collection (instead of all documents stored in the same one)
  • When set to true, the option collectionName gets ignored.
  • Default: false
  • Note: When you dont set this setting to true, you should create an index for your MongoDB collection.

persistence.getYDoc(docName: string): Promise<Y.Doc>

Create a Y.Doc instance with the data persistet in MongoDB. Use this to temporarily create a Yjs document to sync changes or extract data.

persistence.storeUpdate(docName: string, update: Uint8Array): Promise

Store a single document update to the database.

persistence.getStateVector(docName: string): Promise<Uint8Array>

The state vector (describing the state of the persisted document - see Yjs docs) is maintained in a separate field and constantly updated.

This allows you to sync changes without actually creating a Yjs document.

persistence.getDiff(docName: string, stateVector: Uint8Array): Promise<Uint8Array>

Get the differences directly from the database. The same as Y.encodeStateAsUpdate(ydoc, stateVector).

persistence.clearDocument(docName: string): Promise

Delete a document, and all associated data from the database.

persistence.setMeta(docName: string, metaKey: string, value: any): Promise

Persist some meta information in the database and associate it with a document. It is up to you what you store here. You could, for example, store credentials here.

persistence.getMeta(docName: string, metaKey: string): Promise<any|undefined>

Retrieve a store meta value from the database. Returns undefined if the metaKey doesn't exist.

persistence.delMeta(docName: string, metaKey: string): Promise

Delete a store meta value.

persistence.getAllDocNames(docName: string): Promise<Array<string>>

Retrieve the names of all stored documents.

persistence.getAllDocStateVectors(docName: string): Promise<Array<{ name:string,clock:number,sv:Uint8Array}

Retrieve the state vectors of all stored documents. You can use this to sync two y-mongodb-provider instances.

!Note: The state vectors might be outdated if the associated document is not yet flushed. So use with caution.

persistence.flushDocument(docName: string): Promise

Internally y-mongodb stores incremental updates. You can merge all document updates to a single entry. You probably never have to use this.

persistence.destroy(): Promise

Close the database connection for a clean exit.

Indexes

It is recommended that you create the following compound index on your MongoDB collection to improve query performance:

db['yjs-writings'].createIndex({
	version: 1,
	docName: 1,
	action: 1,
	clock: 1,
	part: 1,
});

An other example

yUtils.setPersistence({
	bindState: async (docName, ydoc) => {
		const persistedYdoc = await mdb.getYDoc(docName);
		// get the state vector so we can just store the diffs between client and server
		const persistedStateVector = Y.encodeStateVector(persistedYdoc);

		/* we could also retrieve that sv with a mdb function
		 *  however this takes longer;
		 *  it would also flush the document (which merges all updates into one)
		 *   thats prob a good thing, which is why we always do this on document close (see writeState)
		 */
		//const persistedStateVector = await mdb.getStateVector(docName);

		// in the default code the following value gets saved in the db
		//  this however leads to the case that multiple complete Y.Docs are saved in the db (https://github.com/fadiquader/y-mongodb/issues/7)
		//const newUpdates = Y.encodeStateAsUpdate(ydoc);

		// better just get the differences and save those:
		const diff = Y.encodeStateAsUpdate(ydoc, persistedStateVector);

		// store the new data in db (if there is any: empty update is an array of 0s)
		if (diff.reduce((previousValue, currentValue) => previousValue + currentValue, 0) > 0)
			mdb.storeUpdate(docName, diff);

		// send the persisted data to clients
		Y.applyUpdate(ydoc, Y.encodeStateAsUpdate(persistedYdoc));

		// store updates of the document in db
		ydoc.on('update', async (update) => {
			mdb.storeUpdate(docName, update);
		});

		// cleanup some memory
		persistedYdoc.destroy();
	},
	writeState: async (docName, ydoc) => {
		// This is called when all connections to the document are closed.

		// flush document on close to have the smallest possible database
		await mdb.flushDocument(docName);
	},
});

Contributing

We welcome contributions! Please follow these steps to contribute:

1. Fork the repository.
2. Set up your development environment: npm install.
3. Make your changes and ensure tests pass: npm test.
4. Submit a pull request with your changes.

Testing

To run the test suite, use the following command:

npm test

License

y-mongodb-provider is licensed under the MIT License.

[email protected]