csv-parser

Permissive CSV parser.

• Accepts both \" and "" as escape sequences for double-quotes.
• Accepts either , or ; as column separator.

Approach

A state machine is defined, with the following states:

ClearBuffer
ActiveBuffer
MaybeWithinQuotes
WithinQuotes
MaybeDoneWithQuotes
MaybeEscapeNext_InsideQuotes
MaybeEscapeNext_OutsideQuotes

The initial state is ClearBuffer. To parse the CSV, the text is read one character at a time. Based on the current state and the character code, one or more actions are applied and then the machine moves into another state.

The actions are:

push_to_buffer
push_quote_to_buffer
push_backslash_to_buffer
flush_buffer
create_new_row

These actions modify a text buffer and a 2D array of rows containing cells. push_to_buffer adds the recently read character to the buffer; flush_buffer puts the buffer's contents into a cell and clears the buffer; create_new_row concatenates a new empty "row" onto the 2D array.

The result is a 2D array of cells, grouped by row.

The full state machine can be viewed in Lucidchart.

Determining the separator

Semicolon-separated .csv files are sometimes used, especially in locales where the comma is used as the decimal point in writing numbers. To determine the separator of a particular CSV, we run the same state machine over the first 5 lines of the file -- but instead of keeping a buffer, we keep a list of how many commas and semicolons were seen in each line (not counting those inside quoted strings) and use the results to infer which is the separator.

This separator determination is used as the first step of the CSV parser algorithm; it determines the separator used in the main step.