als-store Library

als-store is a Node.js library that revolutionizes file management by treating files and directories akin to a database. It provides a unique approach to handling file systems with capabilities typically seen in database management systems.

Capabilities of als-store:

• Efficient File Management: Offers a simplified and effective approach to managing files and directories. Users can easily create, read, update, and delete files, as well as organize and manipulate directories.
• Advanced Querying and Filtering: Enables complex operations such as sorting, filtering, and limiting files within directories, catering to specific requirements and improving data retrieval efficiency.
• Schema-Based File Name Handling: Supports custom schemas for file names, allowing for structured and validated file naming conventions.
• Caching for Performance: Implements an intelligent caching mechanism RLU, significantly reducing file system access times and enhancing overall performance.

Use Cases:

• Content Management Systems: Ideal for systems that require organized file storage and retrieval.

• Data Processing Applications: Beneficial for applications that need to process and manipulate a large number of files.

• General Application Development: Versatile for any Node.js application that involves file management.

• als-store Library

  • Change log
  • Install and Import
  • Usage
    • Examples
  • Store class api
    • Key Functionalities:
    • Store Class Constructor
      • Constructor Parameters and Validation:
      • Additional Actions in the Constructor:
      • Usage example
    • Method: get()
      • Overview:
      • Functionality:
      • Implementation Details:
      • API
      • Example
    • Method: first()
      • Overview:
      • Functionality:
      • Implementation Details:
    • Method: file()
      • Overview:
    • File Filtering Methods
      • Method: skip(skip)
      • Method: limit(limit)
      • Method: level(level)
      • Method: dir(dir)
      • Method: filter(...fns)
      • Example
    • Sorting Methods in Store Class
      • Sorting Properties
      • Specifying Sorting Order
    • Usage Examples
    • Method: remove()
      • Overview:
      • Functionality:
      • Usage Examples:
      • Method Implementation:
    • Important Notes:
    • Method: renameDir(oldDirName, newDirName)
      • Overview:
      • Usage with dir Method:
      • Functionality:
      • Method Implementation:
      • Important Notes:
    • Method: create(name, value)
      • Overview:
      • Usage with dir Method:
      • Method Implementation:
      • Examples
    • Method: values()
      • Overview:
      • Caching:
      • Asynchronous File Reading:
      • Customizing Parallelism:
      • Method Implementation:
    • Usage Example:
    • Class: File
      • Overview:
      • Caching Mechanism:
      • File Operations:
      • Reading File Content:
      • Static Data:
      • File Naming and Schema:
    • File Class Constructor
      • Overview:
      • Constructor Parameters:
      • Functionality Description:
      • Usage Example:
    • Getters and Setters in the File Class
      • schema (Getter and Setter)
      • name (Getter and Setter)
      • params (Getter)
      • $name (Getter)
      • fullPath and $fullPath (Getters)
      • buffer (Getter)
      • value (Getter and Setter)
      • json (Getter and Setter)
      • stats (Getter)
    • Asynchronous Methods in the File Class
      • async getStats()
      • async getValue()
      • async save()
      • async remove()
    • File Name Conversion in the File Class of als-store
      • Schema Dependency
      • Handling Complex Data
      • Operating System Constraints
      • Cross-platform Compatibility
      • Reversibility of the Process
      • Practical Applications

Change log

• updated als-promises - throw more specific error
• value setter - can get buffer as value
• fixed updating stats after saving
• in create(name,value) - name can include dir, like - some/filename.ext
• get(value=false), first(value=false) - value parameter (use instead values())
• new method - store.file(filename,value=false)
• empty/undefined dir store.dir('') should work now
• file.save() - return file

Install and Import

Install using npm:

npm i als-store

Require in your project:

const { Store, File } = require('als-store')

Usage

Examples

Example for merging docs:

const {Store} = require('als-store')
const root = new Store({dirPath:__dirname});

(async function() {

// get all files with values from docs folder
   const { results } = await root.dir('docs').values().get()
   
// Merge the content
   const content = results.map(({value}) => value).join('\n');

// Create new file readme with merged connent and save it
   await root.create('readme.md',content).save()

})()

Example for user management:

const { string, email, lowercase, id } = require('als-schema')
const {Store} = require('./index')
const { join } = require('path')

// Create user schema
const schema = {
   id,
   name: [lowercase, string(1, 25)],
   email: [lowercase, email],
}

// Create user's store
const dirPath = join(__dirname, 'users')
const userStore = new Store({ dirPath, schema })

// Save user
const userData = {
   id:undefined,
   name: 'Alex',
   email: '[email protected]'
}
const userBio = 'some bio'
const user = userStore.create(userData,userBio)

console.log(user.name) // Alex.alex@mail[.]com
console.log(user.$name) // alex.alex@mail[.]com.60d215t2817459018
console.log(user.params) // { name: 'alex', email: '[email protected]', id: '60d215t2817459018' }
const {id} = user.params

await user.save()

// find user
await userStore.filter(({params}) => params.id === id).first()

Store class api

The Store class is a central component of the als-store library, designed to manage files and directories in a way that resembles database operations. It offers a structured and efficient approach to file system interaction, making it a powerful tool for various file management tasks in Node.js applications.

Key Functionalities:

1. Query Building: Similar to constructing queries in a database, Store allows users to build queries for file retrieval. This includes filtering, sorting, and limiting the files returned, providing precise control over the file selection process.

2. Caching Mechanism: Store implements an advanced caching system (through File cache system), enhancing performance by reducing the frequency and cost of file system access.

3. Asynchronous File Operations: All operations in Store are asynchronous, ensuring non-blocking execution and better performance, especially in I/O-intensive environments.

4. Directory Management: Beyond individual files, Store can create, delete, and rename directories, providing comprehensive control over the file system structure.

5. Schema Integration: When used with a schema, Store can validate file names and contents, ensuring that files adhere to predefined formats and standards. This feature adds a layer of data integrity, similar to schema validation in databases.

Store Class Constructor

The Store constructor creates a new instance of the Store class, which represents a file store with specific structure and functionality. It initializes the store with various configuration options provided through the params object.

Constructor Parameters and Validation:

• params: An object containing the following configuration parameters:
  • dirPath (String, required, 3-255 characters): Specifies the path to the directory where files will be stored. This parameter is validated to ensure it is a string with a length between 3 and 255 characters.
  • schema (Object, optional): A schema object used for validating files in the store. If provided, a new instance of Schema (als-schema package) is created with the provided schema. If not provided, no schema validation is used.
  • maxAge (Number, optional, >= 1000): Specifies the maximum lifetime of a file in milliseconds. It is validated to be an optional numeric parameter with a minimum value of 1000.
  • maxActiveAge (Number, optional, >= 1000): Specifies the maximum active time of a file in milliseconds. Similar to maxAge, it is validated as an optional numeric parameter with a minimum value of 1000.

Additional Actions in the Constructor:

• Calls ensureDirSync(dirPath) to ensure the existence of the directory at the specified path.
• Initializes internal variables and states of the store, such as results, errors arrays, and the options object.

This constructor allows flexible and scalable setup of a file store, catering to various use cases with customizable parameters.

Usage example

const Store = require('als-store');
const params = {
  dirPath = __dirname,
  schema = {name:String,ext:String,ext1:String},
  maxAge = 1000*60*60*30,
  maxActiveAge = 1000*60*60*7
}
const store = new Store(params)

Method: get()

Overview:

The get(value=false) method is an asynchronous function responsible for retrieving files from the store based on the current options set in the Store instance. It filters, sorts, and limits the results according to the specified criteria.

if value parameter set to true (false by default), return files with values.

Functionality:

• Retrieval and Filtering: Retrieves files from the specified directory (and subdirectories, if needed) based on the depth level, filtering criteria, and sorting options.
• Validation and Sorting: After retrieving the files, it performs additional operations like validating file stats, applying filters, and sorting the results if required.
• Handling Values and Removing Outdated Files: Optionally retrieves the values of the files and removes outdated files based on maxAge and maxActiveAge settings.

Implementation Details:

• Resets the internal state before starting the retrieval process.
• Combines multiple conditions like filters, sorting, and pagination (limit and skip) to process the file list.
• Uses promises to handle asynchronous operations efficiently.
• Filters and sorts the results based on the provided options.
• Optionally loads the content of the files if the values option is set.
• Performs cleanup of outdated files.

API

get method return Store instance which initialize store.results and store.errors as empty arrays before get runs. All results added to store.results and errors to store.errors and can be available after promise is resolved. Each result is instance of File class.

Example

const {results=[],errors=[]} = await store.get()

// or

await store.get()
const {results=[],errors=[]} = store

Method: first()

Overview:

The first(value=false) method is an asynchronous function that retrieves the first file from the store based on the current options, similar to the get() method but specifically for the first file.

if value parameter set to true (false by default), return file with value.

Functionality:

• This method is essentially a convenience function that sets the limit to 1, calls the get() method, and returns the first file (instance of File) in the results.

Implementation Details:

• Sets the limit option to 1 to ensure only the first file is fetched.
• Calls the get() method to retrieve the file.
• Returns the first file in the results array or null if no files are found.

Method: file()

Overview:

The first(filename,value=false) method is an asynchronous function that retrieves the first file from the store by filename parameter.

if value parameter set to true (false by default), return file with value.

File Filtering Methods

Method: skip(skip)

• Purpose: Sets the number of files to skip before starting to collect files in the results.
• Parameters:
  • skip (Number, required): The number of files to skip.
• Returns: The Store instance for method chaining.
• Description: This method sets the skip option in the internal options object, determining how many files will be skipped in the retrieval process.

Method: limit(limit)

• Purpose: Specifies the maximum number of files to include in the results.
• Parameters:
  • limit (Number, required): The maximum number of files to retrieve.
• Returns: The Store instance for method chaining.
• Description: This method sets the limit option in the internal options object, controlling the maximum number of files that can be retrieved in one call.

Method: level(level)

• Purpose: Sets the depth level for file retrieval in directory substructures.
• Parameters:
  • level (Number, required): The depth level for file retrieval.
• Returns: The Store instance for method chaining.
• Description: This method configures how deep the file retrieval process will go into the directory structure. A level of 0 means only the current directory, 1 includes one level of subdirectories, and so on.

Method: dir(dir)

• Purpose: Specifies a subdirectory within the main directory for file operations.
• Parameters:
  • dir (String, required, 1-255 characters): The subdirectory path.
• Returns: The Store instance for method chaining.
• Description: This method sets a specific subdirectory path within the main directory where file operations (like retrieval and storage) will be focused.
  • The subdirectory can be relative path to directory, like somedir/anotherdir

Method: filter(...fns)

• Purpose: Applies custom filter functions to the file retrieval process.
• Parameters:
  • ...fns (Function, required): One or more functions used as filters.
• Returns: The Store instance for method chaining.
• Description: This method allows adding custom filter functions to the internal options.filters array. Each filter function is applied in sequence to the file list during the retrieval process.

Example

const jsFilter = (file) => file.name.endsWith('.js')
const sizeFilter = (file) => file.stats.size <= 1024
await store
   .skip(10) // skip first 10 files
   .limit(50) // limit to 50 files
   .dir('some-dir') // include files only from some-dir folder
   .level(2) // don't include files up to level 2
   .filter(jsFilter,sizeFilter) // include only js files smaller than 1024 bytes
   get()
console.log(store.results)

Sorting Methods in Store Class

Sorting Properties

• Purpose: To define how files in the store can be sorted.
• Available Properties for Sorting:
  • name: Sorts files based on their names.
  • stats.size, stats.birthtime, etc.: Sorts files based on specific statistics like size, creation time, etc.
  • Any other custom property defined in the files.

Specifying Sorting Order

• Ascending Order: By default, files are sorted in ascending order based on the chosen property.
• Descending Order: To sort in descending order, you can use either of the following methods:
  • Prefix the property name with a minus sign (-). For example, -name sorts files by name in descending order.
  • Use the desc() method after specifying the sorting property. For example, if you have set the sorting property as name, calling desc() will sort the files by name in descending order.

Usage Examples

1. Ascending Order:

   • .sort('name'): Sorts files by name in ascending order.
   • .sort('stats.size'): Sorts files by size in ascending order.

2. Descending Order:

   • .sort('-name'): Sorts files by name in descending order.
   • .sort('stats.size').desc(): Sorts files by size in descending order.

These sorting methods allow users to organize files in the Store according to specific criteria, either in ascending or descending order, enhancing the flexibility and usability of the file management system.

Method: remove()

Overview:

The remove() method in the Store class is designed to delete files or entire directories from the store. It offers flexibility to remove files based on various criteria, such as directory location, nesting level, and custom filters.

Functionality:

• General Removal: If no specific options are set, remove() will delete all files within the store.
• Directory-Specific Removal: When used in conjunction with the dir() method, remove() will delete all files within the specified directory.
• Level-Based Removal: When used with the level() method, it will delete files up to a specified nested directory level.
• Filtered Removal: Can be combined with filter(), skip(), limit(), and other methods to remove files that match specific criteria.

Usage Examples:

1. Remove All Files in Store:

   await store.remove(); // Removes all files from the store

2. Remove Files in a Specific Directory:

   await store.dir('/some-dir').remove(); // Removes all files in the 'some-dir' folder

3. Remove Files Up to a Nested Level:

   await store.level(2).remove(); // Removes all files up to nested level 2

4. Remove Files Based on Filters:

   // Example: Using filter, skip, and limit
   await store.filter(/* some filter function */).skip(10).limit(5).remove();
   // Removes files that match the filter, skipping the first 10 and limiting to the next 5

Method Implementation:

• Flexible Criteria: The method checks the current configuration of the Store instance to determine which files or directories should be removed.
• Asynchronous Operation: The removal operation is asynchronous, ensuring that the process does not block other operations in the application.

Important Notes:

• Caution: This method performs deletion operations that cannot be undone. It is crucial to ensure that the correct filters and options are set before executing this method.
• Performance Considerations: The method's performance may vary based on the number of files to be removed and the file system's characteristics.

Method: renameDir(oldDirName, newDirName)

Overview:

The renameDir method in the Store class is used for renaming a directory within the store. It allows for the modification of directory names, providing flexibility in managing the directory structure.

Usage with dir Method:

• Setting Directory Path: The renameDir method can be preceded by the dir method to specify the path to the directory that is to be renamed.
• Example:

  store.dir('some/path/to/dir').renameDir('old-value', 'new-value');

Functionality:

• Directory Existence Check: Before renaming, the method checks for the existence of both the old directory (to ensure it exists) and the new directory (to ensure it does not already exist).
• Renaming Process: If the checks pass, the method proceeds to rename the old directory to the new directory name.
• Cache Update: After renaming, the store's internal cache is updated to reflect the change in directory name. This ensures that subsequent operations on the store will recognize the new directory structure.

Method Implementation:

• Parameters:
  • oldDirName (String, required): The current name of the directory to be renamed.
  • newDirName (String, required): The new name for the directory.
• Asynchronous Operation: The renaming operation is asynchronous, allowing other processes to run concurrently without blocking.
• Error Handling: If either directory does not meet the existence criteria, the method will not proceed with the renaming to prevent any unintended data loss or conflicts.

Important Notes:

• Correct Usage: It's essential to ensure that the directory names provided are accurate and that the new directory name does not conflict with any existing directory names.
• Impact on Store: Renaming a directory will affect all files and subdirectories within it. Users should be cautious to avoid disrupting the file structure unintentionally.

This renameDir method provides an efficient way to manage the directory structure within the Store class, with built-in checks and cache updating for consistency and safety.

Method: create(name, value)

Overview:

The create method in the Store class is used for creating new file within the store. Under the hood, it creates new instance of File class and return it.

Usage with dir Method:

• Setting Directory Path: The create method can be preceded by the dir method to specify the path to the directory that is to be saved.
• Example:

  store.dir('some/path/to/dir').create('filename', 'value');

Method Implementation:

• Parameters:
  • name (String/Object, required): The current name of the file or parameters if schema presented.
    • Can include directory path - like some/dir/name.ext
    • Error Handling:
      • if name is object and schema presented, error validation functions can throw errors.
      • if name is string and it's length less than 1 or more than 255
  • value (String, optional): The value of the files.

Examples

Using filename:

const file = store.create('some.txt','some value')
file.save()

Using object:

// schema {name:String,ext:String}
const file = store.create({name:'some',ext:'txt'},'some value')
file.save()

Method: values()

Overview:

The values() method is used to enable the retrieval of file content in the Store class. This method configures the store to load the content of each file when the get() method is called.

Caching:

• Cache Size: The content of each file retrieved by this method is cached. The size of this cache is configurable and plays a crucial role in the performance and memory usage of the application.
• Cache Behavior: Once the cache reaches its set limit, older entries may be evicted to make room for new ones, following a cache eviction policy (e.g., Least Recently Used).

Asynchronous File Reading:

• Parallel Processing: The reading of file contents is performed asynchronously, allowing for non-blocking I/O operations.
• Grouped Processing: By default, files are read in parallel in groups of 100 processes at a time. This parallelism is managed to optimize performance while avoiding excessive resource usage.

Customizing Parallelism:

• Adjustable Parallel Promises: The number of parallel processes can be customized by modifying the Store.paralelPromises static property.
• Example Usage: Setting Store.paralelPromises = 50 would change the parallel processing to handle 50 files at a time.

Method Implementation:

• Enabling Values: By invoking this method (values()), the store is configured to load and cache the contents of files when get() is executed.
• Asynchronous Behavior: The method ensures that file content loading is done asynchronously, maintaining application responsiveness.

Usage Example:

const store = new Store({ dirPath: '/path/to/dir' });
const {results,errors} = await store.values().get()

Class: File

Overview:

The File class in the provided codebase is designed for managing files, offering functionalities like creation, deletion, and renaming. It also enables reading and interpreting file content in various formats and handles file metadata. A key feature of this class is its caching mechanism, enhancing performance and efficiency.

Caching Mechanism:

• Cache Management: The File class maintains a cache for storing file data and metadata.
• Configurable Cache Size: The cache size is adjustable and is set to 50% of the available free memory by default. This can be modified by changing File.maxCacheSize.
• Cache Operations: The cache supports operations like clearing, adding, and removing entries, ensuring efficient memory management.

File Operations:

• Creating Files: New files can be created with specified names and paths.
• Deleting Files: Files can be deleted both from the file system and the cache.
• Renaming Files: The class allows renaming of files, updating both the file system and the cache to reflect the changes.

Reading File Content:

• Buffer: Reads file content as a buffer.
• String Value: Interprets file content as a string.
• JSON: Parses file content as JSON, allowing for easy manipulation of JSON data.

Static Data:

• Default Static Properties: The class includes static properties like stats, which by default include 'size', 'atimeMs', 'mtimeMs', and 'birthtimeMs'.
• Managing Static Data: These static properties are used to manage and access file metadata.

File Naming and Schema:

• Schema-Based Naming: The file name can encode data if a schema is provided for interpreting these names.
• Schema Validation: The schema can also be used for validating files, ensuring that they conform to expected formats and standards.

File Class Constructor

Overview:

The File class constructor creates an instance of File, designed to manage a file within a specified directory. It supports validation, caching, and data management of the file.

Constructor Parameters:

1. path (String, required): The path to the directory where the file is located. This parameter specifies the file's location in the file system but does not include the file name itself.

2. name (String, required): The name of the file. It is used for identifying and managing the file within its directory.

3. schema (Schema, optional): An instance of the Schema class used for generating file parameters from its name and vice versa. The schema allows for the transformation of the file name into a structured set of data (e.g., {name: 'Alex Smith', age: 25} from the file name Alex Smith.25) and ensures data validation.

Functionality Description:

• When creating a File instance, the constructor initializes key properties of the file, such as path, name, and size.
• If a schema is provided for the file, it is used to generate parameters from the file name and to convert parameters back into a name.
• The constructor also manages the addition of the file to the cache, updating the size of cached data and ensuring efficient memory usage.

Usage Example:

const file = new File('/path/to/directory', 'example.txt', userSchema);

Getters and Setters in the File Class

schema (Getter and Setter)

• Getter: Returns the current schema of the file, which is used for data validation and management.
• Setter: Sets a new schema for the file. If provided, the schema is used to transform the file name into parameters and vice versa, allowing the organization and validation of file data according to a specified format.

name (Getter and Setter)

• Getter: Returns the current name of the file.
• Setter: Sets a new name for the file. Validates whether the new file name meets the standards. On successful name change, updates the cache size and removes the old name from the cache.

params (Getter)

• Getter: Returns the file's parameters extracted from its name using the current schema.

$name (Getter)

• Getter: Returns the file name, which can be constructed from the file's parameters if a schema is provided.

fullPath and $fullPath (Getters)

• fullPath Getter: Returns the full path to the file, combining the specified path and the current file name. Validates the generated path.
• $fullPath Getter: Similar to fullPath, but uses $name (potentially modified name) to form the path.

buffer (Getter)

• Getter: Returns the file's content buffer, if it has been loaded.

value (Getter and Setter)

• Getter: Returns the file's content as a utf-8 string, if available.
• Setter: Sets a new string value for the file, converting it to a buffer and marking the file as modified.
  • Can get value as buffer

json (Getter and Setter)

• Getter: Returns the file's content converted to JSON, if available.
• Setter: Sets new file content in JSON format, converting it to a string before saving.

stats (Getter)

• Getter: Provides file statistics, such as size, access time, modification time, and creation time, if they have been loaded.

Asynchronous Methods in the File Class

async getStats()

• Purpose: Retrieves and updates the statistical information of the file such as size, access time, modification time, and creation time.
• Functionality:
  • Checks if statistics are already loaded to avoid unnecessary file system access.
  • Retrieves stats using stat from fs-extra if the file exists.
  • Updates the internal #stats property with relevant statistical data.
  • Depends on File.stats, which is a configurable array defining which statistical properties to load and cache.
  • Only the properties specified in File.stats are stored in the cache, making it customizable based on application needs.
• Usage: Typically called to get updated file information, especially after file operations that might change these statistics. It allows for selective caching of file statistics as defined in File.stats.

async getValue()

• Purpose: Loads and returns the content of the file.
• Functionality:
  • Checks if the file content is already loaded in the cache.
  • Reads the file content asynchronously using readFile from fs-extra.
  • Updates the internal #value property with the file content.
• Usage: Useful for accessing the file content without directly interacting with the file system, leveraging the internal caching mechanism.

async save()

• Purpose: Saves changes to the file, including content and metadata.
• Functionality:
  • Handles new file creation or updating existing files.
  • Manages file renaming and ensures synchronization with the cache.
  • Writes the file content to the disk, updating the file system.
  • Retrieves the latest file stats after saving.
• Usage: Called to persist modifications to the file, ensuring that changes in content or metadata are accurately reflected in the file system.
• returns instance of file

async remove()

• Purpose: Deletes the file from the file system and removes its data from the cache.
• Functionality:
  • Uses unlink from fs-extra to delete the file.
  • Handles errors during the deletion process, capturing them in an errors array.
  • Cleans up the cache by calling removeFromCache.
• Usage: Utilized to permanently delete a file, effectively removing it from both the file system and the internal cache.

File Name Conversion in the File Class of als-store

Schema Dependency

• Conditional Conversion: The sophisticated file name conversion process is triggered only when a schema is provided. In the absence of a schema, files are saved with their original names without undergoing this conversion.

Handling Complex Data

• Conversion Logic: Complex data types (like numbers, strings, objects) are encoded into a string format that complies with file naming standards.
• Special Cases Handling: Specific values such as undefined, null, and NaN are converted to recognizable string representations.

Operating System Constraints

• Windows Restrictions: Windows has specific rules for file names, disallowing certain characters and reserved names.
• Conversion on Windows:
  • Illegal characters are replaced with _x followed by their ASCII hex code.
  • Reserved names are transformed using _r and a hex-encoded representation of each character.

Cross-platform Compatibility

• Uniform Handling: Ensures consistent handling across different platforms, including Linux and macOS.
• Dot Handling: Dots (.), often used in file extensions, are replaced with [.] to maintain clarity.

Reversibility of the Process

• Decoding: The conversion process is reversible, allowing the original data to be reconstructed from the file name by reversing the encoding steps.

Practical Applications

• Schema-Based File Naming: Facilitates storing structured data in file names, enabling a seamless mapping between data and valid file names.
• Data Integrity and Accessibility: Maintains data integrity and accessibility across platforms by ensuring valid and meaningful file names.
• Automated Process: The entire conversion is handled automatically, abstracting complexities from the user.