with-csv
v2.0.0
Published
A CSV file manipulation library with a fluent API à la Lodash
Downloads
646
Readme
withCSV
It's like lodash for CSV files !
withCSV
is a Node.js library to consume and produce CSV files with clean and readable code, without sacrificing performance. It features :
📜 A fluent API similar to lodash chainable methods, treating your CSV like the array of objects it really is
🏋️ Based on a robust parsing library using Node streams. It is stupid fast and memory-efficient by default, for you to go crazy on large files 🪐
🖋 Equipped with a streaming CSV stringifying library fit for writing large volumes of data to disk
⚙ Barely 300 lines of Typescript
⏳ Support for asynchronous callbacks
Installing
withCSV
can be installed using your package manager of choice :
npm install with-csv
# or
yarn add with-csv
Usage
Example
Given the following CSV file :
id,name,phone,flag,category
1,Joe,0612345678,true,6
2,Jack,0698765421,false,12
3,Mark,0645631256,true,54
4,Valerie,0645631256,true,12
import { withCSV } from 'with-csv'
const result = await withCSV('my.csv')
.columns(['name', 'phone', 'flag'])
// row (below) is automatically typed as {name: string, phone: string, flag: string}
.filter(row => row.flag === 'true')
.map(row => `${row.name}: ${row.phone}`)
// value (below) has been typed as the output of .map , which is a string
.filter(value => value.startsWith('J'))
// At this point the CSV file hasn't yet been read
// It will be read by the terminator method `rows` (below)
.rows()
console.log(result)
// [
// "Joe: 0612345678",
// "Jack: 0698765421"
// ]
// You can also use withCSV to produce CSV files after treatment
await withCSV('my.csv')
.columns(['name', 'phone', 'flag'])
// row (below) is automatically typed as {name: string, phone: string, flag: string}
.filter(row => row.flag === 'true')
.toCSV('your.csv')
Initialization
withCSV(csvFile, options): Returns an instance of withCSV configured with the provided CSV file and options. At this stage the CSV file is not opened yet.
- csvSource: The path to the CSV file
- options (optional): A csv-parse options object
The withCSV
instance exposes the methods columns which takes as input an array of column names. This allows withCSV
to infer the type of the rows.
Querying API
Once you have selected your columns, you can start manipulating your CSV data using the Querying API. It consists of two categories of methods :
⛓️ chainable methods which are stacked in a pipeline through which every row will be processed one by one
🚧 terminator methods which will trigger the reading of the file, and the processing of each row through the pipeline
Only one terminator method can be present. It will return a promise which resolves to the output of your pipeline.
Unless otherwise specified, the methods signature are always the same as their corresponding method in the javascript Array prototype.
The only major differences are :
- All methods in the querying API accept asynchronous callbacks
- Array methods such as
filter
,map
etc... will not receive the whole array as their last argument. This is by design as the CSV file is never held fully in memory.
⛓️ Chainable methods
map(callback): maps each record to a new shape. The output will be typed accordingly.
pick(keys): picks a subset of properties from the records. keys
is described in the lodash.pick documentation.
filter(callback): filters out records.
forEach(callback): this is executed on each record, but doesn't alter the data of the rows.
uniq(iterator): deduplicates records from your CSV file. It can accept as argument :
- A column name to deduplicate on that column
- An array of column anmes to deduplicate on the combination of those columns
- A callback returning a string to deduplicate on the value of that string
🚧 Terminator methods
The following methods only ever consume one row at a time so they are safe to use on very large files
process: Executes the pipeline on all the rows, but without outputting any data. This is useful for example when your pipeline is based on forEach
and you want to discard the final output data.
find(callback): returns the first matching record
findIndex(callback): returns the index of the first matching record
every(callback): returns true
if all records match
some(callback): returns true
if at least one record matches
includes(value): returns true
if the final output contains the value passed. This uses lodash.isEqual
so the value can be a primitive, object, array, etc...
count(): returns the number of rows at the end of the pipeline
first(limit): returns the first elements of the result up to a maximum of limit
toCSV(csvTarget, options): writes the result to a file or a stream
- if
csvTarget
is a string, a file at this path will be created - if
csvTarget
is a WriteStream, the data will be piped directly to it options
are documented in thecsv-stringify
documentation page
The following methods consume the entirety of your CSV file and the resulting output will be stored in memory. Very large files should be adequately filtered beforehand or you may max out your machine's memory.
last(limit): returns the last elements of the result up to a maximum of limit
skip(offset): returns the whole result but omits the offset
first items
key(property, filterUndefined): returns an array of the values of that property
for each row. If filterUnderfined
is true, then only defined values will be returned.
toJSON(replacer, spaces): returns the final result of the query pipeline, as a JSON string. replacer
and spaces
are documented in the JSON.stringify signature.
rows(): returns all the rows of the result, as an array of objects
Contributing & Testing
Feel free to write us about any bug you may find, or submit a PR if you have fresh ideas for the library !
This project does its testing with a ghetto Jest clone written in <60 lines, and which includes basic benchmarking. You can launch the test suite & benchmarks with npm test
or yarn test
(it should take around 1 minute).
Here are the benchmark results on a decently powerful machine. Small sample contains 100 rows, medium contains 100 000 and large contains 2 000 000.
Benchmark Import and export : small: 12.515ms
Benchmark Import and export : medium: 390.564ms
Benchmark Import and export : large: 8.820s
1 => Import and export
Benchmark 1 map : small: 3.122s
Benchmark 1 map : medium: 576.674ms
Benchmark 1 map : large: 9.265s
2 => 1 map
Benchmark 4 chained map : small: 2.094ms
Benchmark 4 chained map : medium: 509.942ms
Benchmark 4 chained map : large: 11.925s
3 => 4 chained map
Benchmark uniq : small: 2.786ms
Benchmark uniq : medium: 667.513ms
Benchmark uniq : large: 15.352s
4 => uniq
Made with 💖 @ Ambler HQ