nexline
v1.2.2
Published
Read file, stream, string, buffer line by line without putting them all in memory. It supports cool features like `custom line separator`, `various encodings`, `reverse mode`, `iterable protocol`
Downloads
644
Maintainers
Readme
Nexline
Read file, stream, string, buffer line by line without putting them all in memory.
It supports cool features below.
- Lightweight
- Iterable protocol
- Handle large file with small memory footprint
- Support various input types (file, stream, string, buffer)
- Support various encodings
- Support reverse mode
- Support custom, multiple line separators
- Support multiple inputs
Why I made this?
Node.js's default readline module is great but it's pause()
method does not work immediately.
I made wrapper to fix that issue before, but it was not enough.
Even if pause()
works immediately, it is still inconvenient.
If I want to execute async function over line by line in large file, I have to call pause()
and resume()
at every line.
I needed better way to do that without putting them all in memory.
Breaking changes in 1.0.0
lineSeparator
default value is changed from['\n', '\r\n']
to'\n'
for better performance- If you want to handle both CRLF and LF, set
lineSeparator: ['\n', '\r\n']
- If you want to handle both CRLF and LF, set
- Support reverse mode.
- Support file descriptor as input
- Improve performance
- Optimize memory usage
Install with npm
Required Node.js version >= 8.0.0
npm install nexline
How to use
Use next()
method
const nexline = require('nexline');
async function main () {
const nl = nexline(...);
while(true) {
const line = await nl.next();
console.log(line);
if (line === null) break; // If all data is read, returns null
}
}
Use as iterator ( Requires node >= 10 )
const nexline = require('nexline');
async function main () {
const nl = nexline(...);
// nexline is iterable
for await (const line of nl) console.log(line);
}
Use file as input
- Don't forget to close file descriptor after finish
- You can use
autoCloseFile: true
to close file descriptor automatically
const nl = nexline({
input: fs.openSync(path_to_file, 'r'),
// autoCloseFile: true,
});
Use stream as input
const nl = nexline({
input: fs.createReadStream(path_to_file),
});
Use string as input
const nl = nexline({
input: 'foo\nbar\nbaz',
});
Use buffer as input
const nl = nexline({
input: Buffer.from('foo\nbar\nbaz'),
});
Use multiple input
const nl = nexline({
input: [
fs.openSync(path_to_file1, 'r'),
fs.createReadStream(path_to_file2),
'foo\nbar\nbaz',
Buffer.from('foo\nbar\nbaz'),
],
});
Use other encodings
See encodings supported by iconv-lite
const nl = nexline({
input: fs.openSync(path_to_file, 'r'),
encoding: 'cp949',
});
Use other lineSeparator
const nl = nexline({
input: 'foo;bar;baz',
lineSeparator: ';',
});
Use multiple lineSeparator
const nl = nexline({
input: 'foo\r\nbar\nbaz',
lineSeparator: ['\n', '\r\n'], // You can handle both LF and CRLF like this.
});
Reverse mode
const nl = nexline({
input: fs.openSync(path_to_file, 'r'), // NOTE: You cannot use stream in reverse mode.
reverse: true,
});
Methods
| Name | Description |
| ------------- | --------------- |
| next() | async, It returns line until all data is read, after then returns null
|
| close() | Close nexline |
Options
| Name | Default | Description | | ------------- | --------------------------- | --------------- | | input | undefined | Required. File descriptor, stream, string, buffer, You can provide multiple inputs using array | | lineSeparator | '\n' | Any string more than one character. You can provide multiple line separator using array | | encoding | 'utf8' | See encodings supported by iconv-lite | | reverse | false | Reverse mode, Cannot use this option with stream input because stream cannot be read from end. Use file descriptor instead | | autoCloseFile | false | Automatically close file descriptor after all data is read |
Contribution
Pull requests and stars are always welcome. For bugs and feature requests, please create an issue
License
Copyright © 2019 Youngho Jin. Released under the MIT License