ReliableTXT-IO

About this package

This package is the Node.js-specific part mentioned in the environment-independent ReliableTXT package (You will find more information there about ReliableTXT in general). This package uses Node.js's file system module and offers simple classes to load and save ReliableTXT files. It offers stream reader and writer classes to read and write ReliableTXT files line-by-line.

If you want to get a first impression on how to use this package, you can watch this video. But always check the changelog of the presented packages for possible changes, that are not reflected in the video.

Getting started

First get the ReliableTXT-IO package installed with a package manager of your choice. If you are using NPM just run the following command:

npm install @stenway/reliabletxt-io

As a first example how to use the package, let's create a ReliableTxtDocument and save the document as a file using the static ReliableTxtFile class and its static method saveSync. This method takes the document and a filepath string as arguments. After the file was written, we load it with the static method loadSync and print its content to the console.

import { ReliableTxtDocument } from '@stenway/reliabletxt'
import { ReliableTxtFile } from '@stenway/reliabletxt-io'

const filePath = "Test.txt"
const document = new ReliableTxtDocument("Hello world")
ReliableTxtFile.saveSync(document, filePath)

const loadedDocument = ReliableTxtFile.loadSync(filePath)
console.log(loadedDocument)

The package always offers synchronous and asynchronous versions of the methods and classes. The synchronous methods always have the 'Sync' suffix and the asynchronous ones come without the suffix. The asynchronous verion of the previous example would therefor look like this:

const filePath = "Test.txt"
const document = new ReliableTxtDocument("Hello world")
await ReliableTxtFile.save(document, filePath)

const loadedDocument = await ReliableTxtFile.load(filePath)
console.log(loadedDocument)

If you directly want to write text to a file, without using the ReliableTxtDocument class, you can use the static methods writeAllTextSync or writeAllText of the ReliableTxtFile class. The default encoding is UTF-8. If you want to use another encoding, you can specify it as an argument.

const content = "A\nB"
ReliableTxtFile.writeAllTextSync(content, "TestUtf8.txt")
ReliableTxtFile.writeAllTextSync(content, "TestUtf16.txt", ReliableTxtEncoding.Utf16)
ReliableTxtFile.writeAllTextSync(content, "TestUtf16R.txt", ReliableTxtEncoding.Utf16Reverse)
ReliableTxtFile.writeAllTextSync(content, "TestUtf32.txt", ReliableTxtEncoding.Utf32)

const text1 = ReliableTxtFile.readAllTextSync("TestUtf8.txt")
const text2 = ReliableTxtFile.readAllTextSync("TestUtf16.txt")
const text3 = ReliableTxtFile.readAllTextSync("TestUtf16R.txt")
const text4 = ReliableTxtFile.readAllTextSync("TestUtf32.txt")

The readAllTextSync or readAllText methods of the ReliableTxtFile class load the file and return only the textual content, without the information which encoding was used.

Using the ReliableTxtDocument class and the loadSync method we can retrieve the information which encoding was used via the encoding property:

const document = new ReliableTxtDocument("A\nB", ReliableTxtEncoding.Utf16)
ReliableTxtFile.saveSync(document, "TestUtf16.txt")

const loadedDocument = ReliableTxtFile.loadSync("TestUtf16.txt")
console.log(loadedDocument.encoding)

The ReliableTxtFile class also offers some comfort methods, like in the following example, which shows the use of the static method writeAllLinesSync. The method takes an array of line strings, joins them with the line feed character and writes the resulting string as ReliableTXT file.

ReliableTxtFile.writeAllLinesSync(["A", "B"], "TestUtf8.txt")
ReliableTxtFile.writeAllLinesSync(["A", "B"], "TestUtf16.txt")

const text1 = ReliableTxtFile.readAllTextSync("TestUtf8.txt")
const text2 = ReliableTxtFile.readAllTextSync("TestUtf16.txt")

With the static readAllLinesSync method we can directly get the lines array back again:

const lines1 = ReliableTxtFile.readAllLinesSync("TestUtf8.txt")
const lines2 = ReliableTxtFile.readAllLinesSync("TestUtf16.txt")

If you want to append to an existing ReliableTXT file use the appendAllText or appendAllLines methods of the ReliableTxtFile class:

ReliableTxtFile.writeAllTextSync("A\nB", "Append.txt")
ReliableTxtFile.appendAllTextSync("C\nD", "Append.txt")
ReliableTxtFile.appendAllLinesSync(["E", "F"], "Append.txt")

Big files

Be aware of string length limitations in Node.js / V8. Depending on the version, the maximum allowed string lengths might vary. Strings might be limited to roughly 512 MB. There might also be limits to the string length that the provided UTF-16 decoder can process. This limit might be 128 MB.

To test this out, you can play around with the following code, that creates big text files with various encodings:

import { ReliableTxtEncoding } from "@stenway/reliabletxt"
import { ReliableTxtFile } from "@stenway/reliabletxt-io"

function writeBigFiles() {
	// create a big string

	console.log("Creating string")
	const maxStringLength = 512 * 1024 * 1024 - 24 // (1 << 29) - 24
	const maxUtf16DecoderStringLength = 128 * 1024 * 1024 - 1 // 134_217_727 
	const text: string = "a".repeat(maxStringLength-1)

	// write big files

	console.log("Write Utf8")
	ReliableTxtFile.writeAllTextSync(text, "BigUtf8.txt")

	console.log("Write Utf16")
	ReliableTxtFile.writeAllTextSync(text, "BigUtf16.txt", ReliableTxtEncoding.Utf16)

	console.log("Write Utf32")
	ReliableTxtFile.writeAllTextSync(text, "BigUtf32.txt", ReliableTxtEncoding.Utf32)
}

writeBigFiles()

function readBigFiles() {
	let text = ReliableTxtFile.readAllTextSync("BigUtf8.txt")
	console.log(text.length)

	text = ReliableTxtFile.readAllTextSync("BigUtf16.txt")
	console.log(text.length)

	text = ReliableTxtFile.readAllTextSync("BigUtf32.txt")
	console.log(text.length)
}

readBigFiles()

console.log("Done")

Streaming classes

This package also contains classes to read and write ReliableTXT files line-by-line. The following example shows the use of the SyncReliableTxtStreamWriter to write a big file and the use of the ReliableTxtStreamReader to read it. Both classes are instatiated using a static create method. The writer offers the writeLine method and the reader offers the readLine method, which either returns the line string or null, if the end of the file was reached.

import { SyncReliableTxtStreamWriter, SyncReliableTxtStreamReader } from "@stenway/reliabletxt-io"

const longLine = "a".repeat(1024*1024)

// write

const writer = SyncReliableTxtStreamWriter.create("Test.txt")
try {
	for (let i: number = 0; i<1000; i++) {
		writer.writeLine(`Line ${(i+1).toString().padStart(10, "0")}: ${longLine}`)
	}
} finally {
	writer.close()
}

// read

const reader = SyncReliableTxtStreamReader.create("Test.txt")
try {
	let count = 0
	while (true) {
		const line = reader.readLine()
		if (line === null) { break }
		count++
	}
	console.log(`Count: ${count}`)
} finally {
	writer.close()
}

Related packages

The Stenway Text File Format Stack defines a set of formats that are built upon ReliableTXT. Check out the following related io-packages:

• WSV-IO
• SML-IO
• TBL-IO
• OML-IO