CSV PARSER ON STEROIDS

CSV files parser on TypeScript with Sequelizer. For easier migration.

Install npm modules

npm install

Then configure a database according to the .env.example file

In config.ts you will also see sample default values...

We can define different options upon reading a file e.g:

interface CSVParserOptions {
    mapKeyIndexes?: number[];
    mapKeySeparator?: string;
    classPath?: string;
    classObjectGetterName?: string;
    delimiter?: string;
    rootDir?: string;
    rootEnv?: string;
    fileExtension?: string;
   mergeMapKeyValues?: boolean;
}

1. mapKeyIndexes

   • After reading a file a Map<any|number, any> is generated by default it has numeric index as a key, but we can assign a key generated by the properties.
   • If we want records to have a custom key from CSV file e.g: email we can set mapKeyIndexes: [3] where 3 is the index in CSV for email.
   • We can also have combined key e.g: [0, 3] which will give us ${ID}-${EMAIL}
   • Note: the keys are separated by the Dash symbol if you want to override the default you can change the value of mapKeySeparator key.

2. classPath && classObjectGetterName

   • After parsing the data from CSV you can cast it to a certain Class which by default can be created inside entities' folder. e.g: { classPath: '../entities/Profile' }
   • If you want value of the map to be some certain property or getter inside the Class you can provide a value for classObjectGetterName e.g: { classObjectGetterName: 'profile' } which is the profile property inside the class

3. delimiter

   • As a delimiter your CSV file can contain [comma: ,] symbol if that is not the default for your CSV you can provide a different value defined inside.

4. mergeMapKeyValues

   • While setting a map to the key if you want to receive the same key values in an array make this value true

Example:

const fileName = "sample_profiles"; // file name inside /files/FILE_NAME.csv

 const options: CSVParserOptions = {
   classPath: '../entities/Profile', // Class name if you want to cast result into a class
   classObjectGetterName: 'profile', // Name of the property inside the Class which will be the value inside the Map
   mapKeyIndexes: [3] // Email as a key of the Map
 };

 // Initialize the parser with properties
 const csvParser = new CSVParser(
     fileName,
     options,
 );

 // Read the data inside the parser
 csvParser.Read().then((data) => {
   data.forEach((profile) => {
       console.log('>>> profile', profile);
   })
 });





const connection = dbConnection("profile", 100, 0).then((response) => {
   console.log('>>> response from db:', response);
}).catch((error) => {
   console.log('>>> error: ', error);
});


dbConnection = async (schema?: string, limit: number = 0, offset: number = 0) => {
   const DB = Sequelizer(schema);
   return DB.query(`SELECT uuid, brand_id FROM 
                profile.profiles
            WHERE brand_id IS NOT NULL
            ORDER BY uuid DESC LIMIT ${limit} OFFSET ${offset}`,
           {
              type: QueryTypes.SELECT,
              logging: false,
           });
}