als-meta-base

The Als MetaBase is a database system designed to store data using a unique approach where data that fits within the filename constraints is stored in the filename itself, while the remaining data is stored within the file content. This method leverages the typical usage patterns where the majority of database entries are short enough to fit within filename limits (255 characters for most systems and 1000+ characters for systems like Linux). As a result, the system can achieve high performance since it mostly deals with file names stored in-memory, minimizing the need to read file content.

The system converts all data to base64 for storage, or encrypts it if encryption is enabled, ensuring data integrity and security. Currently, the database supports local disk storage, but is architectured to allow the replacement of file handling methods, with future plans to support cloud storage solutions like AWS S3.

Key Features

• Efficient Data Storage: Utilizes filenames for storing data, reducing the need for file content access.
• Encryption Support: Offers optional data encryption for enhanced security.
• Flexible Storage Backend: Designed with the capability to substitute local file handling with other storage backends like cloud-based systems.
• Scalability: Handles a large amount of data efficiently due to minimal disk read operations.
• Multiple databases

Installation

To install als-meta-base, run the following command in your project directory:

npm install als-meta-base

This command will download and install the als-meta-base library and its dependencies into your project.

Creating a Database Instance

als-meta-base provides a flexible system for managing databases. You can either use a default database instance or create custom database instances as needed.

Using the Default Database Instance

By default, als-meta-base provides a pre-configured database instance which you can use directly without additional setup. Here's how you can access and use the default database instance:

const { Model, Schema:{ string, number } } = require('als-meta-base');

// Accessing and modifying the default database instance directory
Model.db.rootDir = 'path/to/your/directory';

// Using the default database instance
const User = new Model('User', {
  name: [String,string(3,20,'No name')],
  age: [Number,number(0,120)]
});

// Adding data to the User model
const newUser = User.create({ name: "John Doe", age: 30 });
await newUser.save();

Creating Custom Database Instances

You can also create multiple custom database instances if you need separate databases for different parts of your application:

const { DB, Model, Schema:{ string, number } } = require('als-meta-base');

// Creating a custom database instance
const customDB = new DB({
  rootDir: "/path/to/custom/db",
  maxAge: 1000 * 60 * 60 * 24 // 1 day in milliseconds
});

// Defining a model for the custom database
const Product = new Model('Product', {
  title: [String,string(3,20)],
  price: [Number,number(0)]
}, false, customDB);

// Adding data to the Product model
const newProduct = Product.create({ title: "Widget", price: 25.99 });
await newProduct.save();

In both cases, you can define models using the Model class by specifying the model name and schema. The Schema class is used to define the structure of the data.

Configuration Options

When creating a database instance, you can customize its behavior using various configuration options. Here's a list of available options:

• rootDir: The directory where database files will be stored.
• cryptDir: The directory for storing encryption keys if encryption is enabled.
• logger: Function used for logging errors (default is console.error).
• maxAge: Maximum age in milliseconds for data to be kept in memory before being cleared.
• maxFilenameLength: Maximum length of the filename for storing data.
• clearAfter: Number of actions after which the database should clean up old data based on maxAge.

These options allow you to tailor the database to your specific needs.

Using the Model Class

The Model class in als-meta-base is used to define and interact with data models within your database. Each model corresponds to a collection of data with a defined schema.

Defining a Model

To define a model, you need to specify a model name and a schema. The schema defines the structure and constraints of the data. Here's an example of defining a model:

const { Model, Schema:{ string, number } } = require('als-meta-base');

const User = new Model('User', {
  name: [String, string(3, 20, 'No name')], // Name must be a string between 3 and 20 characters
  age: [Number, number(0, 120)] // Age must be a number between 0 and 120
});

Model Parameters

When defining a model, several parameters can be specified to configure its behavior and structure. Here's a breakdown of the parameters used in the Model constructor:

• name (String): The name of the model, which corresponds to the collection name in the database.
• schema (Object): Defines the structure of data within the model. Each key in the schema object corresponds to a field in the model, and the value defines the type of the field and any constraints or default values.

Each field in the schema can have the following attributes:

• Type: The data type of the field (e.g., String, Number). This is mandatory for each field.
• Validator: A function or set of functions that validate the field's value. Validators can check for things like string length, numeric range, etc.
• Default Value: An optional default value for the field if no value is specified.

Here is an expanded example of a model with detailed parameter descriptions:

const { Model, Schema:{ string, number } } = require('als-meta-base');

const User = new Model('User', {
  name: [String, string(3, 20, 'No name')], // Name must be a string between 3 and 20 characters, default 'No name'
  age: [Number, number(0, 120)], // Age must be a number between 0 and 120
  email: [String, string(5, 50)], // Email must be a string between 5 and 50 characters
  isActive: [Boolean, boolean(), true] // isActive is a boolean, default true
});

This setup allows for flexibility and ensures that each record adheres to the defined structure and constraints before being saved to the database.

Creating a Model Instance

Once a model is defined, it can be instantiated and used to interact with the corresponding data in the database:

const newUser = User.create({ name: "Alice", age: 25, email: "[email protected]", isActive: false });
await newUser.save(); // Persist the new user to the database

This method creates a new record in the database based on the model's schema, with the ability to include any or all fields defined in the schema.

Note

The methods used to manipulate data (such as create, find, findOneAndUpdate, findOneAndDelete) ensure that the operations respect the schema's constraints and the database's integrity.

Schema Validation

The Model class uses als-schema for data validation. The schema defines data types, constraints, and default values for each field in your model. This ensures that your data adheres to the specified structure and validation rules before it is saved to the database.

For a detailed description of how to define and customize schemas, refer to the official als-schema documentation available at als-schema npm package.

Creating a New Record

Once a model is defined, you can create new records using the create method, which initializes a new instance of the model with the given parameters:

const newUser = User.create({ name: "Alice", age: 25 });
await newUser.save(); // Persist the new user to the database

Finding Records

To find records in the database, you can use the find method, which returns a query object that can be further refined:

const users = await User.find({ age: { $gt: 18 } }).exec(); // Finds all users older than 18

Updating Records

To update records, use the findOneAndUpdate method. It updates the first record that matches the given query:

await User.findOneAndUpdate({ name: "Alice" }, { age: 26 }); // Updates Alice's age to 26

Deleting Records

To delete a record, use the findOneAndDelete method. It removes the first record that matches the query:

await User.findOneAndDelete({ name: "Alice" }); // Deletes the record with the name Alice

Additional Query Methods

• skip(n): Skips the first n records.
• limit(n): Limits the result to n records.
• sort(prop): Sorts the results by the property prop. Prefix with - for descending order.

Each of these methods can be chained to refine the query:

const limitedUsers = await User.find().sort('-age').limit(5).exec(); // Finds the oldest 5 users

This section of the documentation covers the basic functionalities provided by the Model class, enabling you to effectively interact with your data models.

Working with Queries

In als-meta-base, the Query functionality is indirectly accessible through methods such as find, findOneAndUpdate, and findOneAndDelete. These methods facilitate searching, updating, and deleting records within your database.

The find Method

The find method is used to retrieve a collection of records that match a specified query. This method is typically used for retrieving multiple records based on criteria defined in the where parameter.

// Retrieve all users where age is greater than 18
const adultUsers = await User.find({ age: { $gt: 18 } }).exec();

The result is a collection of instances that match the criteria.

The findOneAndUpdate Method

The findOneAndUpdate method updates the first record that matches the specified where criteria and returns the updated instance.

// Update the age of the first user named Alice to 26
await User.findOneAndUpdate({ name: "Alice" }, { age: 26 });

This method modifies a single record and the operation directly affects the database.

The findOneAndDelete Method

The findOneAndDelete method removes the first record matching the where criteria from the database.

// Delete the first record with the name Alice
await User.findOneAndDelete({ name: "Alice" });

This method deletes a single record based on the provided criteria.

Using Query Conditions

The first parameter in these methods acts as a where clause, which specifies the conditions that the records must meet. The conditions are defined using a JSON-like syntax, allowing for complex querying capabilities. als-meta-base uses als-json-filter for compiling these where conditions into executable queries. For detailed information on creating complex queries with als-json-filter, refer to its documentation at als-json-filter npm package.

Here is an example of a more complex query using multiple conditions:

// Find users named Alice or Bob who are older than 25
const users = await User.find({
  $or: [
    { name: "Alice", age: { $gt: 25 } },
    { name: "Bob", age: { $gt: 25 } }
  ]
}).exec();

Each method uses these query conditions to perform the respective database operations effectively and efficiently.

Query Results

• The find method returns a collection of model instances that match the query criteria.
• Both findOneAndUpdate and findOneAndDelete methods work with single instances; the former updates and the latter deletes the instance matching the criteria.

Refining Queries

In als-meta-base, you can refine your database queries using several chainable methods provided by the Query class. These methods allow for detailed and flexible queries, adjusting the results to fit your specific needs.

The where Method

The where method is used to specify the search criteria for the query. It can be called multiple times, with each call refining the search conditions:

const users = User.find().where({ age: { $gt: 18 } }).where({ active: true });

Each subsequent use of where will add more conditions to the query, effectively using a logical AND between conditions.

The skip Method

The skip method is used to skip over a specific number of records in the result set:

const skippedUsers = User.find().skip(10); // Skips the first 10 records

This is particularly useful for implementing pagination.

The limit Method

The limit method limits the number of records returned by the query:

const limitedUsers = User.find().limit(5); // Limits the result to 5 records

This method is also essential for pagination and controlling the size of the result set.

The select Method

The select method allows you to specify a subset of fields to be included in the result, which can significantly reduce the amount of data retrieved and processed, thus improving performance when full records are not required. You can also exclude specific fields by prefixing the field name with a '-' character.

const selectedUsers = User.find().select(['name', 'email']); // Returns only the 'name' and 'email' of the users
const usersWithoutEmail = User.find().select(['-email']); // Returns all fields except 'email'

This feature is particularly useful in situations where you want to protect sensitive information like passwords or personal identifiers from being exposed in the query results, or simply to optimize data transfer and processing by excluding unnecessary fields.

Excluding Fields with select

When using the select method, you can also exclude fields from the results by prefixing them with a '-'. This is useful for hiding sensitive information or omitting fields that are not necessary for a particular operation.

// Example of excluding a field
const userProfiles = User.find().select(['name', 'age', '-password']); // Fetches all fields except 'password'

This method modifies the query to include or exclude fields dynamically, based on your requirements, giving you fine-grained control over the data returned by your query.

The sort Method

The sort method orders the results based on a specified field. To specify descending order, prefix the field name with a '-' character:

const sortedUsers = User.find().sort('-age'); // Sorts users by age in descending order

Without the prefix, results are sorted in ascending order by default.

The first Method

The first method is a shortcut that retrieves only the first record matching the query criteria:

const firstUser = await User.find({ age: { $gt: 18 } }).first(); // Returns the first user older than 18

This method is a convenient way to retrieve a single record without needing to specify limit(1).

Example of a Combined Query

Here's how you can combine these methods to create a complex query:

const users = await User.find({ age: { $gt: 18 }, active: true })
   .where({ name: 'Alex' })
   .sort('-age')
   .skip(10)
   .limit(10)
   .select(['name', 'age', 'active'])
   .exec();

This query will find active users named Alex over the age of 18, skip the first 10 records, limit the results to 10 entries, select only specified fields, and sort them by age in descending order.

By using these methods, you can craft precise queries to retrieve, update, or delete data according to your application's requirements.

Working with Collections

The exec method of the Query class returns a collection of items when retrieving data from the database. This collection is not just a simple array; it is enhanced with additional methods to facilitate operations on the entire collection or individual items.

Collection Structure

The collection is an array-like object where each item is accessed via a getter function. This setup ensures that the data for each item is retrieved only when it is specifically accessed, improving performance and memory usage by delaying data loading.

Available Methods in Collection

The collection provides several methods that allow you to work with the items more efficiently:

array()

Converts the collection into a standard array of items. This can be useful when you need to use array methods that are not directly available on the collection object.

const usersArray = usersCollection.array(); // Converts collection to a standard array

first()

Returns the first item in the collection, or null if the collection is empty. This is useful for queries that are expected to return a single item.

const firstUser = usersCollection.first(); // Retrieves the first user from the collection

remove()

Performs an asynchronous operation to remove all items in the collection from the database. This method returns a promise that resolves when all items have been successfully removed.

await usersCollection.remove(); // Removes all users in the collection from the database

update(newParams)

Updates all items in the collection with the new parameters specified in newParams. This is an asynchronous operation that applies the updates and saves the changes to the database.

await usersCollection.update({ active: false }); // Sets the 'active' status to false for all users in the collection

Each of these methods provides a way to interact with the collection as a whole or to manipulate multiple items at once, making batch operations simple and efficient.

Usage Example

Here is an example of using a collection returned by the exec method:

const activeUsers = await User.find({ active: true }).exec();

// Convert the collection to an array to use standard array getter
const userArray = activeUsers.array;

// Get the first user from the collection
const firstActiveUser = activeUsers.first();

// Update all users in the collection
await activeUsers.update({ lastLogin: new Date() });

// Remove all users from the collection
await activeUsers.remove();

Working with Proxy Objects

Each item returned in the collection from the exec method is a proxy object, not a plain JavaScript object. This design allows for more control over the interaction with the data fields and ensures any changes are consistently managed and persisted back to the database.

Proxy Object Behavior

Each proxy object represents a single record from the database, with fields accessible as properties. However, these properties have enhanced behavior:

• Lazy Loading: Properties are loaded lazily, meaning the data is only fetched when a property is accessed. This improves performance, especially when working with large datasets.
• Automatic Saving: Changes to any property are tracked, and the save() method can be invoked to persist changes back to the database asynchronously.
• Controlled Access: Only properties defined in the model's schema are accessible. Access to undefined properties will return undefined.

Methods of Proxy Objects

Each proxy object includes several methods that facilitate working with the underlying data:

save()

Saves any changes made to the properties of the proxy object back to the database. This method is asynchronous.

const user = await User.find().first();
user.name = "Alice";
await user.save(); // Persists the change to the database

remove()

Removes the record represented by the proxy object from the database. This method is also asynchronous.

const user = await User.find().first();
await user.remove(); // Deletes the user from the database

Interaction with Proxy Objects

Proxy objects can be manipulated much like regular JavaScript objects, but with the added benefit of integrated database operations:

const user = await User.find({ name: "John" }).first();
console.log(user.age); // Accesses the 'age' property

user.email = "[email protected]"; // Sets a new value for 'email'
await user.save(); // Saves the updated user back to the database

Restrictions

• Immutable Properties: The id property and any other properties not included in the model's schema cannot be modified.
• Selective Fields: If fields are selectively loaded using the select() method in a query, only those fields can be accessed or modified in the proxy object.

This setup ensures data integrity and consistent behavior across all operations, while still providing the flexibility and performance benefits of on-demand data loading and saving.