File System

This project contains some helper functions and types for working with the file system.

All functions that handle text files expect the text to be encoded in UTF-8.

Installation

npm install --save @gmjs/file-system

API Listing

Async

Basic File Operations

• readTextAsync - Reads the entire contents of a text file.
• writeTextAsync - Writes text to a file.
• readBinaryAsync - Reads the entire contents of a binary file into a Buffer.
• writeBinaryAsync - Writes binary data to a file.
• createFileAsync - Creates a file.
• existsAsync - Checks if a file exists.
• ensureDirAsync - Creates a directory and any necessary parent directories if they do not exist.

Find

• findFsEntriesAsync - Searches a directory for file system entries, recursively.

Sync

Basic File Operations

• readTextSync - Synchronously reads the entire contents of a text file.
• writeTextSync - Synchronously writes text to a file.
• readBinarySync - Synchronously reads the entire contents of a binary file into a Buffer.
• writeBinarySync - Synchronously writes binary data to a file.
• createFileSync - Synchronously creates a file.
• existsSync - Synchronously checks if a file exists.
• ensureDirSync - Synchronously creates a directory and any necessary parent directories if they do not exist.

Find

• findFsEntriesSync - Synchronously searches a directory for file system entries, recursively.

Observable

Find

• fromFindFsEntries - Creates an observable of results of file search for a given directory, recursively.

API

Async

Basic File Operations

readTextAsync

Reads the entire contents of a text file.

Accepts a path parameter, returns a Promise<string>.

async function printFileContent(): Promise<void> {
  const content = await readTextAsync('path/to/file.txt');
  console.log(content);
}

writeTextAsync

Writes text to a file. Overwrites the file if it already exists.

Accepts path and content parameters, returns a Promise<void>. content is a string.

async function writeToFile(): Promise<void> {
  await writeTextAsync('path/to/file.txt', 'Hello, World!');
}

readBinaryAsync

Reads the entire contents of a binary file into a Buffer.

Accepts a path parameter, returns a Promise<Buffer>.

async function printFileContent(): Promise<void> {
  const content = await readBinaryAsync('path/to/file.bin');
  console.log(content.toString());
}

writeBinaryAsync

Writes binary data to a file. Overwrites the file if it already exists.

Accepts path and content parameters, returns a Promise<void>. content is a Buffer.

async function writeToFile(): Promise<void> {
  const content = Buffer.from([0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x2c, 0x20, 0x57, 0x6f, 0x72, 0x6c, 0x64, 0x21]);
  await writeBinaryAsync('path/to/file.bin', content);
}

createFileAsync

Creates a file. Often used with writeTextAsync or writeBinaryAsync to ensure that a file and ancestor directories exist before writing to it.

Accepts a path parameter, returns a Promise<void>.

async function createFile(): Promise<void> {
  const path = 'path/to/file.txt';
  await createFileAsync(path);
  await writeTextAsync(path, 'Hello, World!');
}

existsAsync

Checks if a file exists.

Accepts a path parameter, returns a Promise<boolean>.

async function checkFileExists(): Promise<void> {
  const path = 'path/to/file.txt';
  const exists = await existsAsync(path);
  if (exists) {
    const content = await readTextAsync(path);
    console.log(content);
  } else {
    console.log('File does not exist.');
  }
}

ensureDirAsync

Creates a directory and any necessary parent directories if they do not exist.

Accepts a path parameter, returns a Promise<void>.

async function createPathIfNotExists(): Promise<void> {
  await ensureDirAsync('path/to/directory');
}

Find

findFsEntriesAsync

Searches a directory for file system entries, recursively. Search can be limited by depth.

Accepts a directory parameter and an optional options parameter of type FindOptions, returns a Promise<readonly FilePathStats[]>.

For more details on how the search works, see the FindOptions section.

async function printFilePaths(): Promise<void> {
  const files = await findFsEntriesAsync(
    'path/to/directory',
    { depthLimit: 2 },
  );
  for (const file of files) {
    console.log(file.path);
  }
}

Sync

Basic File Operations

readTextSync

Synchronously reads the entire contents of a text file.

Accepts a path parameter, returns a string.

function printFileContent(): void {
  const content = readTextSync('path/to/file.txt');
  console.log(content);
}

writeTextSync

Synchronously writes text to a file. Overwrites the file if it already exists.

Accepts path and content parameters. content is a string.

function writeToFile(): void {
  writeTextSync('path/to/file.txt', 'Hello, World!');
}

readBinarySync

Synchronously reads the entire contents of a binary file into a Buffer.

Accepts a path parameter, returns a Buffer.

function printFileContent(): void {
  const content = readBinarySync('path/to/file.bin');
  console.log(content.toString());
}

writeBinarySync

Synchronously writes binary data to a file. Overwrites the file if it already exists.

Accepts path and content parameters. content is a Buffer.

function writeToFile(): void {
  const content = Buffer.from([0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x2c, 0x20, 0x57, 0x6f, 0x72, 0x6c, 0x64, 0x21]);
  writeBinarySync('path/to/file.bin', content);
}

createFileSync

Synchronously creates a file. Often used with writeTextSync or writeBinarySync to ensure that a file and ancestor directories exist before writing to it.

Accepts a path parameter.

function createFile(): void {
  const path = 'path/to/file.txt';
  createFileSync(path);
  writeTextSync(path, 'Hello, World!');
}

existsSync

Synchronously checks if a file exists.

Accepts a path parameter, returns a boolean.

function checkFileExists(): void {
  const path = 'path/to/file.txt';
  const exists = existsSync(path);
  if (exists) {
    const content = readTextSync(path);
    console.log(content);
  } else {
    console.log('File does not exist.');
  }
}

ensureDirSync

Synchronouly creates a directory and any necessary parent directories if they do not exist.

Accepts a path parameter.

function createPathIfNotExists(): void {
  ensureDirSync('path/to/directory');
}

Find

findFsEntriesSync

Synchronously searches a directory for file system entries, recursivelys. Search can be limited by depth.

Accepts a directory parameter and an optional options parameter of type FindOptions, returns a readonly FilePathStats[].

For more details on how the search works, see the FindOptions section.

function printFilePaths(): void {
  const files = findFsEntriesSync(
    'path/to/directory',
    { depthLimit: 2 },
  );
  for (const file of files) {
    console.log(file.path);
  }
}

Observable

Find

fromFindFsEntries

Creates an observable of results of file search for a given directory. Search is recursive.

Accepts a directory parameter and an optional options parameter of type FindOptions, returns an Observable<FilePathStats>.

For more details on how the search works, see the FindOptions section.

fromFindFsEntries(
  'path/to/directory',
  { depthLimit: 2 },
).subscribe({
  next: (file) => console.log(file.path),
  complete: () => console.log('Search complete.'),
});

Types

FilePathStats

Information about a file. This is the return value of the find functions.

interface FilePathStats {
  readonly path: string;
  readonly stats: Stats; // Node.js fs.Stats
}

path contains the file system entry path relative to the root search directory.

stats contains the file system stats of the file system entry. This is a Node.js fs.Stats object, and will contain information about whether the entry is a file or directory, etc.

If we have the following file structure:

- root/
  - file1.txt
  - file2.txt
  - directory1/
    - file3.txt
    - file4.txt
  - directory2/
    - file5.txt

And the search root is root/, the FilePathStats objects for the files will have the following path values:

file1.txt
file2.txt
directory1/
directory1/file3.txt
directory1/file4.txt
directory2/
directory2/file5.txt

Any directory path will be suffixed with a / - this can be used to differentiate between files and directories without having to check the stats object, and it can be used when speciying sort and filter options.

The search will be done in a depth-first manner.

FindOptions

Input options when searching for files:

• depthLimit: number | undefined
  • The maximum depth to search.
  • Default is undefined.
  • If value is 0 or undefined, there is no depth limit (limit is inifnite).
• sort: FindSortOptions | undefined - See FindSortOptions.
• filter: FindFilterOptions | undefined - See FindFilterOptions.

export interface FindOptions {
  readonly depthLimit?: number;
  readonly sort?: FindSortOptions;
  readonly filter?: FindFilterOptions;
}

FindSortOptions

NOTE: Sorting is done on a per-directory basis, not globally across all files returned by the find function. To sort globally, you can simply implement sorting at the call site, outside of the find function.

Sorting options are a discriminated union of two sorting systems:

• comparer - Accepts a custom comparer function.
• predefined - Accepts predefined sorting strategies.

comparer

Completely flexible, allows you to specify any logic in a custom comparer function.

For example, to sort first files, then directories, and each of those by name ascending, you can use the following comparer:

const comparer: FindSortComparer = (a, b) => {
  if (a.stats.isFile() && b.stats.isDirectory()) {
    return 1;
  }
  if (a.stats.isDirectory() && b.stats.isDirectory()) {
    return -1;
  }
  return a.path.localeCompare(b.path);
};

predefined

Allows the use of predefined sorting strategies.

Two parameters need to be specified:

• typeOrder: FileSortTypeOrder - The sort order by file system type (files and directories):
  • file-first - Files first, then directories.
  • directory-first - Directories first, then files.
  • none - No sorting by type, files and directories are mixed.
• direction: FileSortDirection - The sort direction of file path:
  • asc - Ascending (by file path).
  • desc - Descending (by file path).

To achieve the similar sorting as in the comparer example above (there are some small differences in how locale compare is done), you can use the following predefined parameters:

const params: FindSortPredefinedParameters = {
  typeOrder: 'file-first',
  direction: 'asc',
};

FindSortOptions Implemetation

export const LIST_OF_FIND_SORT_OPTIONS_KINDS = [
  'comparer',
  'predefined',
] as const;

export type FindSortOptionsKind =
  (typeof LIST_OF_FIND_SORT_OPTIONS_KINDS)[number];

export interface FindSortOptionsBase {
  readonly kind: FindSortOptionsKind;
}

export interface FindSortOptionsComparer extends FindSortOptionsBase {
  readonly kind: 'comparer';
  readonly comparer: FindSortComparer;
}

export interface FindSortOptionsPredefined extends FindSortOptionsBase {
  readonly kind: 'predefined';
  readonly params: FindSortPredefinedParameters;
}

export type FindSortOptions =
  | FindSortOptionsComparer
  | FindSortOptionsPredefined;

// comparer
export type FindSortComparer = (a: FilePathStats, b: FilePathStats) => number;

// predefined
export interface FindSortPredefinedParameters {
  readonly typeOrder: FileSortTypeOrder;
  readonly direction: FileSortDirection;
}

export type FileSortTypeOrder = 'file-first' | 'directory-first' | 'none';
export type FileSortDirection = 'asc' | 'desc';

FindFilterOptions

NOTE: Filtering is done on a per-directory basis, not globally across all files returned by the find function. To filter globally, you can simply implement filtering at the call site, outside of the find function.

Caveats

Related to the above - if a directory is filtered out, it will prevent the search from entering that directory. This can be useful for performance reasons, as it can prevent the search from entering large directories that we may want to eventually filter out entirely. Example is a node_modules directory in JavaScript or TypeScript projects.

This can cause some unexpected behavior if you are not aware of it.

Lets say we have the following monorepo file structure:

- root/
  - package.json
  - libs/
    - lib1/
      // ...
      package.json
    - lib2/
      // ...
      package.json

If we want to search for all package.json files in the monorepo, we can use the following filter:

const filter: FindFilterOptions = {
  kind: 'string',
  params: {
    include: [
      { endsWith: '/' }, // make sure that directories are not filtered out
      { endsWith: 'package.json' },
    ],
  },
};

It would return the following paths:

package.json
libs/
libs/lib1/
libs/lib1/package.json
libs/lib2/
libs/lib2/package.json

You could then further filter out the directories at the call site if that is what you wanted.

If you just specified this filter:

const filter: FindFilterOptions = {
  kind: 'string',
  params: {
    include: [
      { endsWith: 'package.json' },
    ],
  },
};

It would just return the root package.json file:

package.json

The reason is that no directory would match endsWith: 'package.json', would be filtered out, and the search would not enter them to find the package.json files inside.

Description

Filtering options are a discriminated union of three filtering systems:

• predicate - Accepts a custom predicate function.
• string - Accepts string filtering parameters.
• regex - Accepts regex filtering parameters.

predicate

Completely flexible, allows you to specify any logic in a custom predicate function.

For example, to filter out all files that don't end with .txt, you can use the following predicate:

const predicate: FindFilterPredicate = (entry) => {
  return entry.path.endsWith('.txt');
};

string

NOTE: Path used for comparison is the FilePathStats path property. This is the path relative from the search root directory. If a file system entry is a directory, the path will end with a /.

See FindFilterStringParameters below for the structure of the options.

It allows specification of include and exclude arrays.

Each array can contain multiple FindFilterStringParametersByType objects, each of which can contain the following properties:

• startsWith: string | undefined - The file path must start with this string.
• endsWith: string | undefined - The file path must end with this string.
• contains: string | undefined - The file path must contain this string.
• equals: string | undefined - The file path must be equal to this string.

If no properties are specified (you pass in an empty array), all files will be included.

If you specify more than one property, all properties must match for the file to be included (AND logic is used).

If more than one of the above objects are specified (in the array of include or exclude), OR logic is used between the items (meaning file system entries are included or excluded if they are matched by any filter element in the array).

include and exclude arrays function the same way except that:

• include is used to include files that match the criteria, and exclude is used to exclude files that match the criteria (obviously).
• If include is not specified, all files are included by default, whereas if exclude is not specified, no files are excluded by default.

First any files that match the include criteria are included, then any files that match the exclude criteria are removed from those included files.

Example:

a.txt
a.bin
b.txt
b.bin
c.txt
c.bin

const params: FindFilterStringParameters = {
  include: [
    { startsWith: 'a', endsWith: '.txt' },
    { endsWith: '.bin' },
  ],
  exclude: [
    { startsWith: 'c' },
  ],
};

Would return:

a.txt
a.bin
b.bin

1. First include entry would include a.txt.
2. Second include entry would include a.bin, b.bin, and c.bin.
3. First (and only) exclude entry would exclude c.bin, and would have excluded c.txt if it had previously been included.
4. That leaves a.txt, a.bin, and b.bin.

regex

Works exactly the same as string, but each item in the include and exclude arrays is a RegExp object instead of a FindFilterStringParametersByType object.

FindFilterOptions Implemetation

export const LIST_OF_FIND_FILTER_OPTIONS_KINDS = [
  'predicate',
  'string',
  'regex',
] as const;

export type FindFilterOptionsKind =
  (typeof LIST_OF_FIND_FILTER_OPTIONS_KINDS)[number];

export interface FindFilterOptionsBase {
  readonly kind: FindFilterOptionsKind;
}

export interface FindFilterOptionsPredicate extends FindFilterOptionsBase {
  readonly kind: 'predicate';
  readonly predicate: FindFilterPredicate;
}

export interface FindFilterOptionsString extends FindFilterOptionsBase {
  readonly kind: 'string';
  readonly params: FindFilterStringParameters;
}

export interface FindFilterOptionsRegex extends FindFilterOptionsBase {
  readonly kind: 'regex';
  readonly params: FindFilterRegexParameters;
}

export type FindFilterOptions =
  | FindFilterOptionsPredicate
  | FindFilterOptionsString
  | FindFilterOptionsRegex;

// predicate
export type FindFilterPredicate = (entry: FilePathStats) => boolean;

// string
export interface FindFilterStringParameters {
  readonly include?: readonly FindFilterStringParametersByType[];
  readonly exclude?: readonly FindFilterStringParametersByType[];
}

export interface FindFilterStringParametersByType {
  readonly startsWith?: string;
  readonly endsWith?: string;
  readonly contains?: string;
  readonly equals?: string;
}

// regex
export interface FindFilterRegexParameters {
  readonly include?: readonly RegExp[];
  readonly exclude?: readonly RegExp[];
}