JavaScript Database (JSDB)

A zero-dependency, transparent, in-memory, streaming write-on-update JavaScript database for the Small Web that persists to a JavaScript transaction log.

Use case

A small and simple data layer for basic persistence and querying.

Built for Small Web use and used in Kitten, Domain, and Site.js.

This is not for you to farm people for their data. Surveillance capitalists can jog on now.

Features

• Transparent: if you know how to work with arrays and objects and call methods in JavaScript, you already know how to use JSDB. It’s not called JavaScript Database for nothing.

• Automatic: it just works. No configuration.

• 100% code coverage: meticulously tested. Note that this does not mean it is bug free ;)

Limitations

• Small Data: this is for small data, not Big Data™.

• For Node.js: will not work in the browser. (Although data tables are plain ECMAScript Modules – aka ESM; es6 modules – and can be loaded in the browser.)

• Runs on untrusted nodes: this is for data kept on untrusted nodes (servers). Use it judiciously if you must for public data, configuration data, etc. If you want to store personal data or model human communication, consider end-to-end encrypted and/or peer-to-peer replicating data structures instead to protect privacy and freedom of speech.

• In-memory: all data is kept in memory and, without tweaks, cannot exceed 1.4GB in size. While JSDB will work with large datasets, that’s not its primary purpose and it’s definitely not here to help you farm people for their data, so please don’t use it for that. (If that’s what you want, quite literally every other database out there is for your use case so please use one of those instead.)

• Streaming writes on update: writes are streamed to disk to an append-only transaction log as JavaScript statements and are both quick (in the single-digit milliseconds region on a development laptop with an SSD drive) and as safe as we can make them (synchronous at the kernel level).

• No schema, no migrations: again, this is meant to be a very simple persistence, query, and observation layer for local server-side data. If you want schemas and migrations, take a look at nearly every other database out there. (Although, if you want to, you could build both on top of JSDB but this is left as, uhum, an excercise for the reader. If you’re interested in more advanced uses of JSDB, take a look at how it’s being used in Domain.

Note: the limitations are features, not bugs. This is a focused tool for a specific purpose. While feature requests are welcome, I do not foresee extending its application scope.

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Installation

npm install @small-tech/jsdb

Usage

Here’s a quick example to whet your appetite:

import JSDB from '@small-tech/jsdb'

// Create your database in the test folder.
// (This is where your JSDF files – “tables” – will be saved.)
//
const db = JSDB.open('db')

// Create db/people.js table with some initial data if it
// doesn’t already exist.
if (!db.people) {
  db.people = [
    {name: 'Aral', age: 43},
    {name: 'Laura', age: 34}
  ]

  // Correct Laura’s age. (This will automatically update db/people.js)
  db.people[1].age = 33

  // Add Oskar to the family. (This will automatically update db/people.js)
  db.people.push({name: 'Oskar', age: 8})

  // Update Oskar’s name to use his nickname. (This will automatically update db/people.js)
  db.people[2].name = 'Osky'
}

After running the above script, take a look at the resulting database table in the ./db/people.js file.

JavaScript Data Format (JSDF)

JSDB tables are written into JavaScript Data Format (JSDF) files. A JSDF file is a plain JavaScript file in the form of an ECMAScript Module (ESM; es6 module) that comprises an append-only transaction log which creates the table in memory. For our example, it looks like this:

export const _ = [ { name: `Aral`, age: 43 }, { name: `Laura`, age: 34 } ];
_[1]['age'] = 33;
_[2] = { name: `Oskar`, age: 8 };
_[2]['name'] = `Osky`;

It’s just JavaScript!

A JSDF file is just JavaScript. Specifically, it is an ECMAScript Module (ESM; es6 module).

The first line is a single assignment/export of all the data that existed in the table when it was created or last loaded.

Any changes to the table made during the last session that it was open are written, one statement per line, starting with the second line.

Since it’s just a regular JavaScript module, you can simply import a JSDF file as a module in Node.js or even in a browser.

For example, create an index.html file with the following content in the same folder as the other script and serve it locally using a web server like Kitten and you will see the data printed out in your browser:

<h1>People</h1>
<ul id='people'></ul>

<script type="module">
  import { _ as people } from '/db/people.js'

  const peopleList = document.getElementById('people')

  people.forEach(person => {
    const li = document.createElement('li')
    li.innerText = `${person.name} (${person.age} years old)`
    peopleList.appendChild(li)
  })
</script>

Supported and unsupported data types.

Just because it’s JavaScript, it doesn’t mean that you can throw anything into JSDB and expect it to work.

Supported data types

• Number
• Boolean
• String
• Object (including objects with null prototypes)
• Array
• Date
• Custom data types (see below).

Additionally, null and undefined values are persisted as-is.

Security note regarding strings

Strings are automatically sanitised to escape backticks, backslashes, and template placeholder tokens to avoid arbitrary code execution via JavaScript injection attacks.

The relevant areas in the codebase are linked to below.

• String sanitisation code (JSDF class)
• String sanitisation code tests (test/index.js)

If you notice anything we’ve overlooked or if you have suggestions for improvements, please open an issue.

Custom data types

Custom data types (instances of your own classes) are also supported.

During serialisation, class information for custom data types is persisted.

For JSDB to be able to find your custom class during deserialisation, you must:

1. Pass the class in question to JSDB.open() in the options.classes array
2. Implement your custom class’s constructor so it accepts a parameter object as its only argument and populates itself with the properties within it.

🪤 Do not use setter methods (the set accessor) in your custom classes as they may not behave as you expect. Instead of set something () {…} use setSomething () {…} for more predictable behaviour. Learn more

Example

import JSDB from '@small-tech/jsdb'

class Person {
  constructor ({name = 'Jane Doe'}={}) {
    this.name = name
  }
  introduceYourself () {
    console.log(`Hello, I’m ${this.name}.`)
  }
}

const db = JSDB.open('db', { classes: [Person] })

// Initialise the people table if it doesn’t already exist.
if (!db.people) {
  db.people = [
    new Person('Aral'),
    new Person('Laura')
  ]
}

// Will always print out “Hello, I’m Laura.”
// (On the first run and on subsequent runs when the objects are loaded from disk.)
db.people[1].introduceYourself()

💡 If the constructor singature looks confusing to you, please read this explanation of how to provide default values for parameter objects in JavaScript.

💡 If your model class accepts arbitrary properties (i.e., if it’s basically an object with helper methods), you can use this shorthand to assign all the properties in the parameter object to your custom object:

class MyModel {
  constructor(parameters) {
    Object.assign(this, parameters)
  }
  // …
}

If you look in the created db/people.js file, this time you’ll see:

export const _ = [ 
  typeof Person === 'function' ? Reflect.construct(Person, [Object.getOwnPropertyDescriptors({name: 'Aral'})]) : (() => { throw new Error('Person class missing in JSDB.open()'); })(),
  typeof Person === 'function' ? Reflect.construct(Person, [Object.getOwnPropertyDescriptors({name: 'Laura'})]) : (() => { throw new Error('Person class missing in JSDB.open()'); })()
 ];

🪤 If you were to load the database without registering the Person class with JSDB when opening the database, you will get a regular object back and the type information will be lost both in the persisted database and in memory.

To test this, you can run the following code:

import JSDB from '@small-tech/jsdb'
const db = JSDB.open('db')

// Prints out { name: 'Laura' }
console.log(db.people[1])

Remember that if you don’t register classes when importing your database, type information will be lost from the database. To test this, run the first piece of code again and note that you get an error similar to:

db.people[1].introduceYourself()
             ^

TypeError: db.people[1].introduceYourself is not a function
    at file:///var/home/aral/Projects/jsdb/examples/custom-data-types/index.js:24:14
    at ModuleJob.run (node:internal/modules/esm/module_job:192:25)

If you look in the people table, you’ll see that the type information has been lost:

export const _ = [ { 'name': `Aral` }, { 'name': `Laura` } ];

You can find these examples in the examples/custom-data-types folder of the source code.

🪤 Setters

Do not use the setter methods (the set accessor) in your custom classes as they may not behave as you expect. Instead of set something () {…} use setSomething () {…} for more predictable behaviour.

Say you have the following custom object:

class Equation {
  constructor ({x = 1, y = 2}={})

  sum() {
    return this.x + this.y
  }

  set operands ({x, y}) {
    this.x = x
    this.y = y
  } 
}

And you persist it like this:

const equation = new Equation()

db.sums.mySum = equation // Persist it.

console.log('Sum: ', equation.sum()) // 3

equation.operands = {x:2, y: 3}
console.log('Sum: ', equation.sum()) // 5

// Close and reopen the database
await db.close()
db = null

db = JSDB.open(databasePath, { classes: [Equation] })

Console.log('Sum: ', equation.sum()) // 5

So, from this, you might think that everything is working as expected and that the following happened:

1. You created the equation and persisted it, which set x to 1 and y to 2 (both in memory and on disk).
2. Then you set the operands property, which resulted in the setter being called and the setter set the x property to 2 and the y property to 3 (both in memory and on disk).
3. When you reopened the database those x and y properties were read back and populated in memory.

However, here’s what actually happened:

1. You created the equation and persisted it, which set x to 1 and y to 2 (both in memory and on disk).
2. Then you set the operands property, which, in memory resulted in the x and y properties being set. However, on disk, what the proxy persisted was anoperands property, with its value set to {x: 2, y: 3}
3. When you reopened the database, the operands setter was called as the table was being read in and in memory, it set the x and y properties to 2 and 3.

In order to get the behaviour you expected, use a regular method instead of setter like this:

class Equation {
  constructor ({x = 1, y = 2}={})

  sum() {
    return this.x + this.y
  }

  setOperands ({x, y}) {
    this.x = x
    this.y = y
  } 
}

Using setOperand() instead of set operand() means that the behaviour of your apps will be more predictable. Specifically:

• You will be sure that your method is only called when you call it in your app (so, for example, you can reference the database itself in your setter method or in any functions you call from your setter method – for example, to update a remote API as part of your model – without having your application crash because the database does not exist yet when your set accessor is called automatically as the table is being read in).

• Persisted properties and in-memory properties will be the same at all times.

Unsupported data types

If you try to add an instance of an unsupported data type to a JSDB table, you will get a TypeError.

The following data types are currently unsupported but might be supported in the future:

• Map (and WeakMap)
• Set (and WeakSet)
• BigInt

The following intrinsic objects are not supported as they don’t make sense to support:

• Intrinsic objects (DataView, Function, Generator, Promise, Proxy, RegExp, Symbol)
• Binary collections (ArrayBuffer, Float32Array, Float64Array, Int8Array, Int16Array, Int32Array, TypedArray, Uint8Array, Uint16Array, Uint32Array, and Uint8ClampedArray)*
• Error types (Error, EvalError, RangeError, ReferenceError, SyntaxError, TypeError, and URIError)

💡 Methods of custom classes you want to persist are automatically ignored, as are any functions you might attempt to mix into them.

* The TypedArray family of classes are implemented differently from every other data type in JavaScript and do not support being proxied, etc.

Important security note

JSDF is not a data exchange format.

Since JSDF is made up of JavaScript code that is evaluated at run time, you must only load JSDF files from domains that you own and control and have a secure connection to.

Do not load in JSDF files from third parties.

If you need a data exchange format, use JSON.

Rule of thumb:

• JSON is a terrible format for a database but a great format for data exchange.
• JSDF is a terrible format for data exchange but a great format for a JavaScript database.

Note: As of version 3.0.0, data in JSDB is evaluated in V8 Virtual Machine contexts instead of using eval(). However, this should still not be considered a security feature. All the above advice still applies.

Migrating from earlier versions of JSDF

As of JSDB version 5.0.0, JSDB uses JSDF version 3.

What’s new in version 3

• Custom classes are expected to have a constructor that accepts a parameter object as its only argument. The custom class must assign properties on itself that mirror the properties passed in this parameter object.

• The constructor of persisted custom classes is run when deserialising the object and must conform to the form outlined in the first point in order for the object to be properly deserialised.

• Custom objects have their methods ignored by default (instead of throwing for trying to persist a function)

• Custom classes can extend other classes (e.g., EventEmitter).

• Properties that begin with an underscore (_) are treated as private and ignored (note that due to the complexities of native private properties – i.e., #thoseThatBeginWithAHash – in JavaScript when interacting with proxied objects, it is recommended that you do not use native private properties in your custom classes).

• Objects with null prototypes are supported. i.e., objects created with Object.create(null, …).

Version 2 to 3

You only need to carry out a migration if you are persisting instances of custom classes in your database and/or if you are currently persisting properties that begin with an underscore (_).

If you are doing neither of those things, you can skip it and your app will work with JSDB 5+/JSDF 3 as it did with JSDB 2-4/JSDF 2.

A. Migrate your database

If you are persisting custom objects:

1. Make a backup of your database folder just in case.

2. From the jsdb source code folder, run the following command in terminal, substituting <path to database folder> with the path to your database folder:

   node migrate/jsdf-version-2-to-3 <path to database folder>

The script will update the custom object instantiation code in your database tables to the new JSDF 3 syntax.

Refactor your code

1. If you are persisting custom objects, ensure that they conform to the JSDF 3 constructor expectation.

   e.g., The following JSDF 2 custom class:

   class CustomClass {
     x = 1
     y = 1
   
     // …
   }

   Becomes the following in JSDF 3:

   class CustomClass {
     constructor ({x=1 , y=2} = {}) {
       this.x = 1
       this.y = 1
     }
   
     // …
   }

2. If your code relied on persisting properties starting with underscores (_), refactor it so it no longer does. Properties starting with underscores are treated as private and not persisted.

   e.g., the _willIBePersisted property in the following custom class will be persisted in JSDF 2 but not in JSDF 3. Refactoring it to willIBePersisted will allow it to be persisted in JSDF 3.

   class CustomClass {
     _willIBePersisted = 42
   }

   🪤 The database migration script in section A does not automatically handle persisted properties that begin with an underscore. You will need to manually rename these in your database tables.

Version 1 to 2

Version 1 of JSDF was used in the earlier 1.x (CommonJS) version of JSDB and contained a UMD-style declaration.

Please use the 1.x branch if that’s what you’d prefer. That branch will continue to be maintained for as long as it is being used in Site.js

Migrating from version 1 to 2 is simple but is not handled automatically for you by JSDB for performance reasons. For a basic example, see examples/jsdf-version-1.0-to-version-2.0-migration.

JavaScript Query Language (JSQL)

In the browser-based example, above, you loaded the data in directly. When you do that, of course, you are not running it inside JSDB so you cannot update the data or use JavaScript Query Language (JSQL) to query it.

To test out JSQL, open a Node.js command-line interface (run node) from the directory that your scripts are in and enter the following commands:

import JSDB from '@small-tech/jsdb'

// This will load test database with the people table we created earlier.
const db = JSDB.open('db')

// Let’s carry out a query that should find us Osky.
console.log(db.people.where('age').isLessThan(21).get())

Note that you can only run queries on arrays. Attempting to run them on plain or custom objects (that are not subclasses of Array) will result in a TypeError.

Furthermore, queries only make sense when used on arrays of objects. Running a query on an array of simple data types will not throw an error but will return an empty result set.

For details, see the JSQL Reference section.

Compaction

When you load in a JSDB table, JSDB will, by default, compact the JSDF file.

Compaction is important for two reasons; during compaction:

• Deleted data is actually deleted from disk. (Privacy.)
• Old versions of updated data are actually removed. (Again, privacy.)

Compaction might also reduce the size of your tables.

Compaction is a relatively fast process but it does get uniformly slower as the size of your database grows – it has O(N) time complexity as the whole database is recreated.

Compaction is disabled for very large tables. (Hopefully, you will never reach such sizes in your Small Web apps.)

Turning off compaction

You do have the option to override the default behaviour and keep all history. To do this, pass compactOnLoad=false as an option while opening your database:

  import JSDB from '@small-tech/jsdb'
  const db = JSDB.open('db', { compactOnLoad: false })

You might want to do this, for example, if you’re creating a web app that lets you create a drawing and you want to play the drawing back stroke by stroke, etc.

Example

Continuing with the example, above, now that you’ve loaded the file back, look at the ./db/people.js JSDF file again to see how it looks after compaction:

export const _ = [ { name: `Aral`, age: 43 }, { name: `Laura`, age: 33 }, { name: `Osky`, age: 8 } ];

Ah, that is neater. Laura’s record is created with the correct age and Oskar’s name is set to its final value from the outset. And it all happens on the first line, in a single assignment. Any new changes will, just as before, be added starting with the third line.

(You can find these examples in the examples/basic folder of the source code.)

Closing a database

Your database tables will be automatically closed if you exit your script. However, there might be times when you want to manually close a database (for example, to reopen it with different settings, etc.) In that case, you can call the asynchronous close() method on the database proxy.

Here’s what you’d do to close the database in the above example:

async main () {
  // … 🠑 the earlier code from the example, above.

  await db.close()

  // The database and all of its tables are now closed.
  // It is now safe (and allowed) to reopen it.
}

main()

Working with JSON

As mentioned earlier, JSDB writes out its tables as append-only logs of JavaScript statements in what we call JavaScript Data Format (JSDF). This is not the same as JavaScript Object Notation (JSON).

JSON is not a good format for a database but it is excellent – not to mention ubiquitous – for its original use case of data exchange. You can easily find or export datasets in JSON format. And using them in JSDB is effortless. Here’s an example that you can find in the examples/json folder of the source code:

Given a JSON data file of spoken languages by country in the following format:

[
  {
    "country": "Aruba",
    "languages": [
      "Dutch",
      "English",
      "Papiamento",
      "Spanish"
    ]
  },
  {
    "etc.": "…"
  }
]

The following code will load in the file, populate a JSDB table with it, and perform a query on it:

import fs from 'fs'
import JSDB from '@small-tech/jsdb'

const db = JSDB.open('db')

// If the data has not been populated yet, populate it.
if (!db.countries) {
  const countries = JSON.parse(fs.readFileSync('./countries.json', 'utf-8'))
  db.countries = countries
}

// Query the data.
const countriesThatSpeakKurdish = db.countries.where('languages').includes('Kurdish').get()

console.log(countriesThatSpeakKurdish)

When you run it, you should see the following result:

[
  {
    country: 'Iran',
    languages: [
      'Arabic',    'Azerbaijani',
      'Bakhtyari', 'Balochi',
      'Gilaki',    'Kurdish',
      'Luri',      'Mazandarani',
      'Persian',   'Turkmenian'
    ]
  },
  {
    country: 'Iraq',
    languages: [ 'Arabic', 'Assyrian', 'Azerbaijani', 'Kurdish', 'Persian' ]
  },
  { country: 'Syria', languages: [ 'Arabic', 'Kurdish' ] },
  { country: 'Turkey', languages: [ 'Arabic', 'Kurdish', 'Turkish' ] }
]

The code for this example is in the examples/json folder of the source code.

Dispelling the magic and a pointing out a couple of gotchas

Here are a couple of facts to dispel the magic behind what’s going on:

• What we call a database in JSDB is just a regular directory on your file system.
• Inside that directory, you can have zero or more tables.
• A table is a JSDF file.
• A JSDF file is an ECMAScript Module (ESM; es6 module) that exports a root data structure (either an object or an array) that may or may not contain data and a sequence of JavaScript statements that mutate it. It is an append-only transaction log that is compacted at load. JSDF files are valid JavaScript files and should import and run correctly under any JavaScript interpreter that supports ESM.
• When you open a database, you get a Proxy instance back, not an instance of JSDB.
• Similarly, when you reference a table or the data within it, you are referencing proxy objects, not the table instance or the data itself.

How the sausage is made

When you open a database, JSDB loads in any .js files it can find in your database directory. Doing so creates the data structures defined in those files in memory. Alongside, JSDB also creates a structure of proxies that mirrors the data structure and traps (captures) calls to get, set, or delete values. Every time you set or delete a value, the corresponding JavaScript statement is appended to your table on disk.

By calling the where() or whereIsTrue() methods, you start a query. Queries help you search for specific bits of data. They are implemented using the get traps in the proxy.

Gotchas and limitations

Given that a core goal for JSDB is to be transparent, you will mostly feel like you’re working with regular JavaScript collections (objects and arrays) instead of a database. That said, there are a couple of gotchas and limitations that arise from the use of proxies and the impedance mismatch between synchronous data manipulation in JavaScript and the asynchronous nature of file handling:

1. You can only have one copy of a database open at one time. Given that tables are append-only logs, having multiple streams writing to them would corrupt your tables. The JSDB class enforces this by forcing you to use the open() factory method to create or load in your databases.

2. You cannot reassign a value to your tables without first deleting them. Since assignment is a synchronous action and since we cannot safely replace the existing table on disk with a different one synchronously, you must first call the asynchronous delete() method on a table’s internal __table__ instance before assigning a new value for it on the database, thereby creating a new table.

   async main () {
     // … 🠑 the earlier code from the example, above.
   
     await db.people.__table__.delete()
   
     // The people table is now deleted and we can recreate it.
   
     // This is OK.
     db.people = [
       {name: 'Ed Snowden', age: 37}
     ]
   
     // This is NOT OK.
     try {
       db.people = [
         {name: 'Someone else', age: 100}
       ]
     } catch (error) {
       console.log('This throws as we haven’t deleted the table first.')
     }
   }
   
   main()

3. There are certain reserved words you cannot use in your data. This is a trade-off between usability and polluting the mirrored proxy structure. JSDB strives to keep reserved words to a minimum.

   This is the full list:

   | | Reserved words | | -------------------------- | ------------------------------------------------------------------------------ | | As table name | close | | Property names in data | where, whereIsTrue, __table__ |

   Note: You can use the __table__ property from any level of your data to get a reference to the table instance (JSTable instance) that it belongs to. This is mostly for internal use but it’s there if you need it.

Table events

You can listen for the following events on tables:

| Event name | Description | | ---------- | ------------------------------------- | | persist | The table has been persisted to disk. | | delete | The table has been deleted from disk. |

Example

The following handler will get called whenever a change is persisted to disk for the people table:

db.people.__table__.addListener('persist', (table, change) => {
  console.log(`Table ${table.tableName} persisted change ${change.replace('\n', '')} to disk.`)
})

JSQL Reference

The examples in the reference all use the following random dataset. Note, I know nothing about cars, the tags are also arbitrary. Don’t @ me ;)

const cars = [
  { make: "Subaru", model: "Loyale", year: 1991, colour: "Fuscia", tags: ['fun', 'sporty'] },
  { make: "Chevrolet", model: "Suburban 1500", year: 2004, colour: "Turquoise", tags: ['regal', 'expensive'] },
  { make: "Honda", model: "Element", year: 2004, colour: "Orange", tags: ['fun', 'affordable'] },
  { make: "Subaru", model: "Impreza", year: 2011, colour: "Crimson", tags: ['sporty', 'expensive']},
  { make: "Hyundai", model: "Santa Fe", year: 2009, colour: "Turquoise", tags: ['sensible', 'affordable'] },
  { make: "Toyota", model: "Avalon", year: 2005, colour: "Khaki", tags: ['fun', 'affordable']},
  { make: "Mercedes-Benz", model: "600SEL", year: 1992, colour: "Crimson", tags: ['regal', 'expensive', 'fun']},
  { make: "Jaguar", model: "XJ Series", year: 2004, colour: "Red", tags: ['fun', 'expensive', 'sporty']},
  { make: "Isuzu", model: "Hombre Space", year: 2000, colour: "Yellow", tags: ['sporty']},
  { make: "Lexus", model: "LX", year: 1997, colour: "Indigo", tags: ['regal', 'expensive', 'AMAZING'] }
]

Starting a query (the where() method)

const carsMadeIn1991 = db.cars.where('year').is(1991).get()

The where() method starts a query.

You call it on a table reference. It takes a property name (string) as its only argument and returns a query instance.

On the returned query instance, you can call various operators like is() or startsWith().

Finally, to invoke the query you use one one of the invocation methods: get(), getFirst(), or getLast().

The anatomy of a query.

Idiomatically, we chain the operator and invocation calls to the where call and write our queries out in a single line as shown above. However, you can split the three parts up, should you so wish. Here’s such an example, for academic purposes.

This starts the query and returns an incomplete query object:

const incompleteCarYearQuery = db.cars.where('year')

Once you call an operator on a query, it is considered complete:

const completeCarYearQuery = incompleteCarYearQuery.is(1991)

To execute a completed query, you can use one of the invocation methods: get(), getFirst(), or getLast().

Note that get() returns an array of results (which might be an empty array) while getFirst() and getLast() return a single result (which may be undefined).

const resultOfCarYearQuery = completeCarYearQuery.get()

Here are the three parts of a query shown together:

const incompleteCarYearQuery = db.cars.where('year')
const completeCarYearQuery = incompleteCarYearQuery.is(1991)
const resultOfCarYearQuery = completeCarYearQuery.get()

Again, idiomatically, we chain the operator and invocation calls to the where() call and write our queries out in a single line like this:

const carsMadeIn1991 = db.cars.where('year').is(1991).get()

Connectives (and() and or())

You can chain conditions onto a query using the connectives and() and or(). Using a connective transforms a completed query back into an incomplete query awaiting an operator. e.g.,

const veryOldOrOrangeCars = db.cars.where('year').isLessThan(2000).or('colour').is('Orange').get()

Example

const carsThatAreFunAndSporty = db.cars.where('tags').includes('fun').and('tags').includes('sporty').get()

Result

[
  { make: "Subaru", model: "Loyale", year: 1991, colour: "Fuscia", tags: ['fun', 'sporty'] },
  { make: "Jaguar", model: "XJ Series", year: 2004, colour: "Red", tags: ['fun', 'expensive', 'sporty']},
]

Custom queries (whereIsTrue())

For more complex queries – for example, if you need to include parenthetical grouping – you can compose your JSQL by hand. To do so, you call the whereIsTrue() method on a table instead of the where() method and you pass it a full JSQL query string. A completed query is returned.

When writing your custom JSQL query, prefix property names with valueOf..

Note that custom queries are inherently less safe as you are responsible for sanitising input at the application level to avoid leaking sensitive data. (Basic sanitisation to avoid arbitrary code execution is handled for you by JSDB). Make sure you read through the Security considerations with queries](#security-considerations-with-queries) section if you’re going to use custom queries.

Example

const customQueryResult = db.cars.whereIsTrue(`(valueOf.tags.includes('fun') && valueOf.tags.includes('affordable')) || (valueOf.tags.includes('regal') && valueOf.tags.includes('expensive'))`).get()

Result

[
  { make: 'Chevrolet', model: 'Suburban 1500', year: 2004, colour: 'Turquoise', tags: [ 'regal', 'expensive' ] },
  { make: 'Honda', model: 'Element', year: 2004, colour: 'Orange', tags: [ 'fun', 'affordable' ] },
  { make: 'Toyota', model: 'Avalon', year: 2005, colour: 'Khaki', tags: [ 'fun', 'affordable' ] },
  { make: 'Mercedes-Benz', model: '600SEL', year: 1992, colour: 'Crimson', tags: [ 'regal', 'expensive', 'fun' ] },
  { make: 'Lexus', model: 'LX', year: 1997, colour: 'Indigo', tags: [ 'regal', 'expensive', 'AMAZING' ] }
]

Relational operators

• is(), isEqualTo(), equals()
• isNot(), doesNotEqual()
• isGreaterThan()
• isGreaterThanOrEqualTo()
• isLessThan()
• isLessThanOrEqualTo()

Note: operators listed on the same line are aliases and may be used interchangeably (e.g., isNot() and doesNotEqual()).

Example (is)

const carWhereYearIs1991 = db.cars.where('year').is(1991).getFirst()

Result (is)

{ make: "Subaru", model: "Loyale", year: 1991, colour: "Fuscia", tags: ['fun', 'sporty'] }

Example (isNot)

const carsWhereYearIsNot1991 = db.cars.where('year').isNot(1991).get()

Result (isNot)

[
  { make: "Chevrolet", model: "Suburban 1500", year: 2004, colour: "Turquoise", tags: ['regal', 'expensive'] },
  { make: "Honda", model: "Element", year: 2004, colour: "Orange", tags: ['fun', 'affordable'] },
  { make: "Subaru", model: "Impreza", year: 2011, colour: "Crimson", tags: ['sporty', 'expensive']},
  { make: "Hyundai", model: "Santa Fe", year: 2009, colour: "Turquoise", tags: ['sensible', 'affordable'] },
  { make: "Toyota", model: "Avalon", year: 2005, colour: "Khaki", tags: ['fun', 'affordable'] },
  { make: "Mercedes-Benz", model: "600SEL", year: 1992, colour: "Crimson", tags: ['regal', 'expensive', 'fun'] },
  { make: "Jaguar", model: "XJ Series", year: 2004, colour: "Red", tags: ['fun', 'expensive', 'sporty'] },
  { make: "Isuzu", model: "Hombre Space", year: 2000, colour: "Yellow", tags: ['sporty'] },
  { make: "Lexus", model: "LX", year: 1997, colour: "Indigo", tags: ['regal', 'expensive', 'AMAZING'] }
]

Note how getFirst() returns the first item (in this case, an object) whereas get() returns the whole array of results.

The other relational operators work the same way and as expected.

String subset comparison operators

• startsWith()
• endsWith()
• includes()
• startsWithCaseInsensitive()
• endsWithCaseInsensitive()
• includesCaseInsensitive()

The string subset comparison operators carry out case sensitive string subset comparisons. They also have case insensitive versions that you can use.

Example (includes() and includesCaseInsensitive())

const result1 = db.cars.where('make').includes('su').get()
const result2 = db.cars.where('make').includes('SU').get()
const result3 = db.cars.where('make').includesCaseInsensitive('SU')

Result 1

[
  { make: "Isuzu", model: "Hombre Space", year: 2000, colour: "Yellow", tags: ['sporty']}
]

Since includes() is case sensitive, the string 'su' matches only the make Isuzu.

Result 2

[]

Again, since includes() is case sensitive, the string 'SU' doesn’t match the make of any of the entries.

Result 3

[
  { make: "Subaru", model: "Impreza", year: 2011, colour: "Crimson", tags: ['sporty', 'expensive'] },
  { make: "Isuzu", model: "Hombre Space", year: 2000, colour: "Yellow", tags: ['sporty'] }
]

Here, includesCaseInsensitive('SU') matches both the Subaru and Isuzu makes due to the case-insensitive string comparison.

Array inclusion check operator

• includes()

The includes() array inclusion check operator can also be used to check for the existence of an object (or scalar value) in an array.

Note that the includesCaseInsensitive() string operator cannot be used for this purpose and will throw an error if you try.

Example (includes() array inclusion check):

const carsThatAreRegal = db.cars.where('tags').includes('regal').get()

Result (includes() array inclusion check)

[
  { make: "Chevrolet", model: "Suburban 1500", year: 2004, colour: "Turquoise", tags: ['regal', 'expensive'] },
  { make: "Mercedes-Benz", model: "600SEL", year: 1992, colour: "Crimson", tags: ['regal', 'expensive', 'fun']},
  { make: "Lexus", model: "LX", year: 1997, colour: "Indigo", tags: ['regal', 'expensive', 'AMAZING'] }
]

Security considerations with queries

JSDB (as of version 1.1.0), attempts to carry out basic sanitisation of your queries for you to avoid a play date with Little Bobby Tables.

That said, you should still sanitise your queries at the application level, if you’re using custom queries via whereIsTrue(). Basic sanitisation will protect you from arbitrary code execution but it will not protect you from, for example, someone passing || valueOf.admin === true to attempt to access private information. You should be vigilant in your sanitisation when using whereIsTrue() and stick to using where() whenever possible.

The current sanitisation strategy is two-fold and is executed at time of query execution:

1. Remove dangerous characters (statement terminators, etc.):

   • Semi-colon (;)
   • Backslash (\)
   • Backtick (`)
   • Plus sign (+)
   • Dollar sign ($)
   • Curly brackets ({})

   Reasoning: remove symbols that could be used to create valid code so that if our sieve (see below) doesn’t catch an attempt, the code will throw an error when executed, which we can catch and handle.

2. Use a sieve to remove expected input. If our sieve contains any leftover material, we immediately return an empty result set without executing the query.

During query execution, if the query throws (due to an injection attempt that was neutralised at Step 1 but made it through the sieve), we simply catch the error and return an empty result set.

The relevant areas in the codebase are linked to below.

• Query sanitisation code (QueryProxy class)
• Query sanitisation code (QuerySanitiser class)
• Query sanitisation code tests (test/index.js)

If you notice anything we’ve overlooked or if you have suggestions for improvements, please open an issue.

Performance characteristics

• The time complexity of reads and writes are both O(1).
• Reads are fast (take fraction of a millisecond and are about an order of magnitude slower than direct memory reads).
• Writes are fast (in the order of a couple of milliseconds on tests on a dev machine).
• Initial table load time and full table write/compaction times are O(N) and increase linearly as your table size grows.

Suggested limits

• Break up your database into multiple tables whenever possible.
• Keep your table sizes under 100MB.

Hard limits

• Your database size is limited by available memory.
• If your database size is larger than > ~1.3GB, you should start your node process with a larger heap size than the default (~1.4GB). E.g., to set aside 8GB of heap space:

node --max-old-space-size=8192 why-is-my-database-so-large-i-hope-im-not-doing-anything-shady.js

Memory Usage

The reason JSDB is fast is because it keeps the whole database in memory. Also, to provide a transparent persistence and query API, it maintains a parallel object structure of proxies. This means that the amount of memory used will be multiples of the size of your database on disk and exhibits O(N) memory complexity.

Initial load time and full table write/compaction both exhibit O(N) time complexity.

For example, here’s just one sample from a development laptop using the simple performance example in the examples/performance folder of the source code which creates random records that are around ~2KB in size each:

| Number of records | Table size on disk | Memory used | Initial load time | Full table write/compaction time | | ----------------- | ------------------ | ----------- | ----------------- | -------------------------------- | | 1,000 | 2.5MB | 15.8MB | 85ms | 45ms | | 10,000 | 25MB | 121.4MB | 845ms | 400ms | | 100,000 | 250MB | 1.2GB | 11 seconds | 4.9 seconds |

(The baseline app used about 14.6MB without any table in memory. The memory used column subtracts that from the total reported memory so as not to skew the smaller dataset results.)

Note: For tables > 500GB, compaction is turned off and a line-by-line streaming load strategy is implemented. If you foresee your tables being this large, you (a) are probably doing something nasty (and won’t mind me pointing it out if you’re not) and (b) should turn off compaction from the start for best performance. Keeping compaction off from the start will decrease initial table load times. Again, don’t use this to invade people’s privacy or profile them.

Development

Please open an issue before starting to work on pull requests.

Testing

1. Clone this repository.
2. npm install
3. npm test

For code coverage, run npm run coverage.

Note: lib/LineByLine.js is excluded from coverage as it is the inlined version of n-readlines. The tests for it can be found as part of that library.

Also, as JSDB has no runtime dependencies, you only have to run npm install if you want to run the test or make a distribution build.

Building

You can now build a 32KB distribution version of the module:

npm run build

Find the distribution build in dist/index.js.

To run the tests on the distribution build, use npm run test-dist.

Ideas for the future.

• [ ] Implement transactions.
• [ ] ╰─ Ensure 100% code coverage for transactions.
• [ ] ╰─ Document transactions.
• [ ] ╰─ Add transaction example.
• [ ] Implement indices.
• [ ] ╰─ Ensure 100% code coverage for indices.
• [ ] ╰─ Document indices.
• [ ] ╰─ Add indices example.

Related projects, inspiration, etc.

• Initial brainstorming (query language)
• SWX (Native SWF data format and related tools for Flash that I created in 2007 by reverse engineering SWF bytecode.)
• proxy-fun
• filejson
• Declaraoids
• ScunMEngine

Like this? Fund us!

Small Technology Foundation is a tiny, independent not-for-profit.

We exist in part thanks to patronage by people like you. If you share our vision and want to support our work, please become a patron or donate to us today and help us continue to exist.

Copyright

© 2020-present Aral Balkan, Small Technology Foundation.