node-stream-csv
v0.0.2
Published
Open a large CSV file line-by-line, applying headers to each row
Downloads
2
Readme
streamCSV
v0.0.2
Read a CSV or TSV file line-by-line, applying headers to each row and guessing the correct type for each entry. Convenient for very large files.
##Installation
npm install node-stream-csv
How to use
streamCSV
takes three arguments: A filename, a callback for each row, and a callback when it's done. The last is optional.
Examples
These are all examples from test/test.js
var streamCSV = require("node-stream-csv");
streamCSV("state_population.tsv", function(state) {
console.log(state);
});
/* Input
SUMLEV REGION DIVISION STATE NAME POPESTIMATE2013 POPEST18PLUS2013 PCNT_POPEST18PLUS
10 0 0 0 United States 316128839 242542967 76.7
40 3 6 1 Alabama 4833722 3722241 77
40 4 9 2 Alaska 735132 547000 74.4
*/
/* Output:
{ SUMLEV: 10,
REGION: 0,
DIVISION: 0,
STATE: 0,
NAME: 'United States',
POPESTIMATE2013: 316128839,
POPEST18PLUS2013: 242542967,
PCNT_POPEST18PLUS: 76.7 }
{ SUMLEV: 40,
REGION: 3,
DIVISION: 6,
STATE: 1,
NAME: 'Alabama',
POPESTIMATE2013: 4833722,
POPEST18PLUS2013: 3722241,
PCNT_POPEST18PLUS: 77 }
{ SUMLEV: 40,
REGION: 4,
DIVISION: 9,
STATE: 2,
NAME: 'Alaska',
POPESTIMATE2013: 735132,
POPEST18PLUS2013: 547000,
PCNT_POPEST18PLUS: 74.4 }
*/
Note that the module correctly interpreted the numbers as numbers. If you don't want this behavior, such as when you've got leading zeroes you want to preserve, pass an options object to the first argument with dontguess
:
streamCSV(
{
filename: "county_population.csv",
dontguess: true
},
function(datum, c) {
process.stdout.write("Read " + c + " lines from county file.\r");
},
function() { // on close
cb();
}
);
/* Output
{ SUMLEV: '050',
REGION: '3',
DIVISION: '6',
STATE: '01',
COUNTY: '001',
STNAME: 'Alabama',
CTYNAME: 'Autauga County',
CENSUS2010POP: '54571'
// etc etc
}
##Options
dontguess
: No type guessing, just import every data point as a string. You can also pass a comma-separated list of fields not to guess onnoheader
: The first line is a regular line of data, not a set of headers. Just give me back arrays of each line.sparse
: If there is a header but the field value is blank, don't include it in the object for that rowmode
: "csv" or "tsv". If unspecified, guesses from file extension.
##Dependencies
This module uses d3's csv parser on each line.
It uses a fork of maleck13's readline module to convert the filestream into line-by-line events. The fork allows for the fact that most readline modules, including the core module, often miss the last line of the file. The fork is distributed in this repo without the git files.