The Great SalesForce Importer

Importing and exporting your dreams since 2018

What It Does

This service grabs all the interesting things from SalesForce every hour or so, massages it into a structure we like, and dumps it into our system. This is our ETL system for SalesForce to the Volta Cloud.

This service also contains the charging-data exporter, which ships aggregate kWh deliver per station back to SalesForce. This baby is bi-directional!

Import

For all variables, please see the template.yaml file for Lambda function SSM Parameter to Environment Variable mappings.

Running Locally - .json option

Create 2 files from env.local.json.template:

1. env.local-integration.json
2. env.local-production.json

These two files correspond to the package scripts for importing integration (SalesForce "fullcopy sandbox") and production data. yarn import:int for integration and yarn import:prod for production. The difference between these two scripts is which SalesForce instance they pull from. Production is the main SalesForce instance, while integration will pull from the Fullcopy sandbox instance. The fullcopy sandbox instance gets refreshed with production data at minimum every 29 days, so data can drift between the two and the sandbox will fall behind production.

Now you can run:

1. yarn
2. yarn build
3. yarn import:prod or yarn import:int

NOTE: env.*.json file patterns are ignored by git.

Running Locally - shell env option

To run without using a env.local-*.json file, the following shell variables must be exported to your active shell.

DB_HOST
DB_NAME
DB_PASSWORD
DB_PORT
DB_USERNAME
ENV
SALESFORCE_LOGIN_URL
SALESFORCE_PASSWORD
SALESFORCE_SECURITY_TOKEN
SALESFORCE_USER

Now you can run:

1. yarn
2. yarn build
3. yarn import

Development Best Practices

Importing something new, are you? Here are a few best practices which we've learned the hard way:

1. Grab the Id property from your object in SalesForce and map it to a salesforce_id column in it's destination table. This relationship is the best way to ensure reliable upserts of your object in the future
2. Avoid anything with the phrase LIN in it. This deeply ingrained technical debt has caused numerous issues through our history, including the Potato Famine.
3. Add your object import to a new file, such as src/import/salesforce-mynewobject.ts. Separating SalesForce queries and mappings by file will make this mess of a repository far more sustainable going forward.