csv-split-easy

Splits the CSV string into array of arrays, each representing a row of columns

Table of Contents

• Install
• Idea
• API
• The algorithm
• Contributing
• Licence

Install

npm i csv-split-easy

// consume its main function as a CommonJS require:
const splitEasy = require('csv-split-easy')
// or as a ES module:
import splitEasy from 'csv-split-easy'

Here's what you'll get:

Type | Key in package.json | Path | Size ----------------|-----------------------|-------|-------- Main export - CommonJS version, transpiled to ES5, contains require and module.exports | main | dist/csv-split-easy.cjs.js | 8 KB ES module build that Webpack/Rollup understands. Untranspiled ES6 code with import/export. | module | dist/csv-split-easy.esm.js | 7 KB UMD build for browsers, transpiled, minified, containing iife's and has all dependencies baked-in | browser | dist/csv-split-easy.umd.js | 31 KB

⬆  back to top

Idea

Split a string representing a CSV file into an array of arrays, so that we can traverse it later.

Acceptance Criteria:

• It should accept CSV's with or without a header row
• Header row might have different amount of columns than the rest of the rows
• Content (not header) rows might be offset and have different amount of columns from the rest
• There can be various line break types (\n, \r, \n\r or \n\n\n\n\n\n\r\r\r\r\r\n\n\n\n or whatever)
• It should ignore any values wrapped with double quotes
• It should interpret commas as part of the value if it is wrapped in double quotes
• It should accept empty fields and output them as empty strings
• It should automatically detect (dot/comma) and remove thousand separators from digit-only cells
• Minimal dependencies and 100% unit test code coverage in all ways: per-branch, per-statement, per-function and per-line.

Outside of the scope:

• Trimming the values of leading and trailing empty space. Just use String.prototype.trim()
• Parsing numeric values. Parse them yourself. It's outside of the scope of this lib.
• Smart detection of the offset columns. See csv-fix-offset
• Sorting rows of double-entry, accounting CSV's. See csv-sort

⬆  back to top

API

splitEasy(str[, opts])

String-in, an array of arrays-out. Empty values, same as numbers too, are set as empty strings.

Options object

Defaults:

    {
      removeThousandSeparatorsFromNumbers: true,
      padSingleDecimalPlaceNumbers: true,
      forceUKStyle: false
    }

options object's key | Type | Obligatory? | Default | Description --------------------------------------|----------|-------------|-------------|---------------------- { | | | | removeThousandSeparatorsFromNumbers | Boolean | no | true | Should remove thousand separators? 1,000,000 → 1000000? Or Swiss-style, 1'000'000 → 1000000? Or Russian-style, 1 000 000 → 1000000? padSingleDecimalPlaceNumbers | Boolean | no | true | Should we pad one decimal place numbers with zero? 100.2 → 100.20? forceUKStyle | Boolean | no | false | Should we convert the decimal separator commas into dots? 1,5 → 1.5? } | | | |

⬆  back to top

Returns

Returns an array of arrays. When there's nothing given, returns [['']]

There's always one array within the main array and there's always something there, at least an empty string.

For example

const splitEasy = require('csv-split-easy')
const fs = require('fs')
const path = require('path')
// let's say our CSV sits in the root:
// its contents, let's say, are:
// 'Product Name,Main Price,Discounted Price\nTestarossa (Type F110),"100,000","90,000"\nF50,"2,500,000","1,800,000"'
const testCSVFile = fs.readFileSync(path.join(__dirname, './csv_test.csv'), 'utf8')

let source = splitEasy(testCSVFile)
console.log('source = ' + JSON.stringify(source, null, 4))
// => [
//        [
//            "Product Name",
//            "Main Price",
//            "Discounted Price"
//        ],
//        [
//            "Testarossa (Type F110)",
//            "100000",
//            "90000"
//        ],
//        [
//            "F50",
//            "2500000",
//            "1800000"
//        ]
//    ]

⬆  back to top

The algorithm

CSV files, especially accounting-ones, are different from just any files. We assume that we don't want any empty rows in the parsed arrays. It means, conventional string splitting libraries would be inefficient here because after splitting, we'd have to clean up any empty rows.

The second requirement is that any of the column values in CSV can be wrapped with double quotes. We have to support mixed, wrapped and not wrapped-value CSV's because Metro bank used to produce these when I banked with them back in 2015.

The third requirement is that any of the values can be wrapped with double quotes and have commas within as values.

The requirements mentioned above pretty much rule out the conventional regex-based split algorithms. You can just split by /\r?\n/ but later you'll need to clean up possible empty rows. You can't string.split each row by comma because that comma might be a value, you need to check for wrapping double quotes first!

So, the best algorithm is a single for-loop traversal on the input string, detecting and array.pushing the values one by one. It worked very well on email-remove-unused-css where I remove unused CSS from an HTML template within around 2.5 times more characters "travelled" than there are in the file. Traversing as a string also worked well on html-img-alt which needs only a single traversal through the string to fix all the img tag alt attributes and clean all the crap in/around them.

⬆  back to top

Contributing

• If you want a new feature in this package or you would like us to change some of its functionality, raise an issue on this repo.

• If you tried to use this library but it misbehaves, or you need an advice setting it up, and its readme doesn't make sense, just document it and raise an issue on this repo.

• If you would like to add or change some features, just fork it, hack away, and file a pull request. We'll do our best to merge it quickly. Code style is airbnb-base, only without semicolons. If you use a good code editor, it will pick up the established ESLint setup.

⬆  back to top

Licence

MIT License (MIT)

Copyright © 2018 Codsen Ltd, Roy Revelt