🚀 Features

• 💾 CSV Source and Destination
• 👮 Validators
• 🤖 Transformers
• 🚨 Error Checking
• 🐘 Write directly to Postgres
• 👨‍⚕️ Intellisense

📦 Install

npm install --save-dev ffc-pay-etl-framework

🪄 Usage

// ESM
import {
  Etl,
  Loaders,
  Validators,
  Transformers,
  Destinations,
} from "ffc-pay-etl-framework";

// CJS
const {
  Etl,
  Loaders,
  Validators,
  Transformers,
  Destinations,
} = require("ffc-pay-etl-framework");

let csvFile = `${process.cwd()}/test/fixtures/SoilType.csv`;
const spinner = new pkg.Spinner().start("Running ETL Pipeline");

const etl = new Etl.Etl();

etl
  .loader(Loaders.CSVLoader({ path: csvFile, columns: columns }))
  .transform(
    Transformers.FakerTransformer({
      columns: [
        {
          name: "Dist Name",
          faker: "location.city",
        },
      ],
    })
  )
  .destination(
    Destinations.CSVFileDestination({
      fileName: "SoilType_Output.csv",
      headers: true,
      includeErrors: false,
      quotationMarks: true,
    })
  )
  .pump()
  .on("finish", () => {
    //Update spinner
    spinner.succeed("ETL Pipeline - succeeded");
  })
  .on("result", (data) => {
    console.log(data); // emits the last row with error information
  });

📢 Shout outs

Faker JS

Validator

Kaggle

✨ Contributing

Please make sure to read the Contributing Guide before making a pull request.

Contributors ✨

Thanks goes to these wonderful people (emoji key):

This project follows the all-contributors specification. Contributions of any kind welcome!