WSV-IO

About this package

This package is the Node.js-specific part mentioned in the environment-independent WSV package (You will find more information there about WSV in general). This package uses Node.js's file system module and offers simple classes to load and save WSV files. It offers stream reader and writer classes to read and write WSV files line-by-line.

If you want to get a first impression on how to use this package, you can watch this video. But always check the changelog of the presented packages for possible changes, that are not reflected in the video.

Getting started

First get the WSV-IO package installed with a package manager of your choice. If you are using NPM just run the following command:

npm install @stenway/wsv-io

We import the WsvFile class from the WSV IO package and parse an example WSV string to get a WsvDocument object. We will pass this document as an argument to the static method saveSync of the WsvFile class. We also specify the file name Test.wsv.

import { WsvDocument } from '@stenway/wsv'
import { WsvFile } from '@stenway/wsv-io'

const filePath = "Test.wsv"
const document = WsvDocument.parse("a b\nc d #My comment")
WsvFile.saveSync(document, filePath)

After running the code, the file was created like this:

a b
c d #My comment

The file will be written with the default ReliableTXT encoding which is UTF-8.

We now change the encoding property of the document to the ReliableTXT encoding UTF-16 and save the document as another file. To use the ReliableTxtEncoding enum, we need to install the ReliableTXT package as well.

document.encoding = ReliableTxtEncoding.Utf16
WsvFile.saveSync(document, "TestUtf16.wsv")

If we wanna save our file and strip all unneccassary whitespace characters and comments away, we can pass another argument to the saveSync method.

WsvFile.saveSync(document, "TestWithoutComment.wsv", false)

And now the comment of the original document is gone:

a b
c d

Loading WSV files

To load a WSV file, we use the static method loadSync and we only need to provide the file path. An import aspect of the load method is, that we will never have to specifiy the encoding with which the file should be loaded, because a WSV file is a ReliableTXT file and thus the encoding is always automatically detected by reading the preamble bytes at the start of the file.

const loadedDocument = WsvFile.loadSync("Test.wsv")
const loadedDocumentStr = loadedDocument.toString()
console.log(loadedDocument.encoding)

When we run the code, we can see that the file encoding UTF-8 was correctly detected.

We now do the same with the UTF-16 encoded WSV file:

const loadedDocumentUtf16 = WsvFile.loadSync("TestUtf16.wsv")
const loadedDocumentUtf16Str = loadedDocumentUtf16.toString()
console.log(loadedDocumentUtf16.encoding)

And it will display the correctly detected UTF-16 encoding.

We can also load our first file, but without the comment, again by specifying the second argument as false:

const loadedDocumentWithoutComment = WsvFile.loadSync("Test.wsv", false)
const loadedDocumentWithoutCommentStr = loadedDocumentWithoutComment.toString()

Appending to a WSV file

A very helpful method, is the static method appendSync of the WsvFile class which will either create a new WSV file when the specified file does not exist, or will append the lines of a WsvDocument to an existing file. The method will take care of the encoding, so that there won't be any mismatches. Meaning you cannot unintentionally append UTF-16 encoded lines, to a UTF-8 encoded file.

WsvFile.appendSync(loadedDocument, "Append.wsv")
WsvFile.appendSync(loadedDocumentUtf16, "Append.wsv")

Big files

Concerning big WSV files, you can watch the ReliableTXT NPM package video which shows the problem of string and file size limits of V8 and Node.js which lead to certain restrictions when it comes to writing and reading bigger files in one take. The WSV IO package uses the ReliableTXT IO package so the same limits apply here as well, concerning the saveSync and loadSync method. The limitations might change from version to version of Node.js. With version 20 of Node.js you can say roughly, a 500MB WSV file can be written and read in one take, but if you need bigger files you should use the stream writer and reader classes or the appendSync method for writing.

Streaming classes

We'll now have a look the synchronous WsvStreamWriter class. We first create a writer object by calling the constructor and passing the file name as an argument. We then iterate a hundred times and write a line using the WriteLine method. After that we close the writer and will execute the example.

import { SyncWsvStreamReader, SyncWsvStreamWriter } from "@stenway/wsv-io"
import { WsvLine } from "@stenway/wsv"

const writer = SyncWsvStreamWriter.create("Stream.wsv")
try {
	for (let i=0; i<100; i++) {
		writer.writeLine(new WsvLine([i.toString(), "a", "b", "c"]))
	}
} finally {
	writer.close()
}

To read the file line by line, we now create a synchronous WsvStreamReader class. The readLine method returns a WsvLine object or returns null, if the end of the file was reached. We count the number of lines and print the number after closing the reader.

const reader = SyncWsvStreamReader.create("Stream.wsv")
try {
	let count = 0
	while (true) {
		const line = reader.readLine()
		if (line === null) { break }
		count++
	}
	console.log(`Count: ${count}`)
} finally {
	writer.close()
}

Asynchronous IO

Like the ReliableTXT IO package the WSV-IO package offers synchronous and asynchronous methods and classes. Synchronous methods always have the 'Sync' suffix, like saveSync or loadSync. To use the asynchronous versions just use the save and load methods of the WsvFile class like in the following example:

const document = WsvDocument.parse("a b\nc")
await WsvFile.save(document, "Test.wsv")
const loadedDocument = await WsvFile.load("Test.wsv")

An asynchronous stream reader analogously can be created like this:

const reader = await WsvStreamReader.create("Stream.wsv")
try {
	const line = await reader.readLine()
} finally {
	await reader.close()
}

BinaryWSV Files

BinaryWSV is the binary representation of WSV documents. It starts with the magic code 'BW1'. BinaryWSV is made for scenarios, where parsing speed of the textual representation might be a limitation. Learn more about the format in this video or read more in the documentation of the WSV package.

This package offers functionality to load and save BinaryWSV files and to stream read and write them line-by-line. The static class BinaryWsvFile offers both synchronous and asynchronous static methods to load, save and append. In the following example the usage of the saveSync and loadSync method is shown:

const document = WsvDocument.parse("a b\nc")
BinaryWsvFile.saveSync(document, "Test.bwsv")
const loadedDocument = BinaryWsvFile.loadSync("Test.bwsv")

The classes and methods are analogous to the ones of the textual representation. A stream writer for a BinaryWSV file would simply look like this:

const writer = SyncBinaryWsvStreamWriter.create("Stream.bwsv")
try {
	for (let i=0; i<100; i++) {
		writer.writeLine(new WsvLine([i.toString(), "a", "b", "c"]))
	}
} finally {
	writer.close()
}