PI dashboard email NPM module

Email library for the PI Dashboard's mailer jobs and admin site. (Email Constructor, Email Sender, and Admin site)

Setup

The module expects the following environment variables to be defined:

• AWS_REGION
• EMAIL_BATCH_DYNAMODB (if Email#create or Email#getBatches are used)
• EMAIL_METADATA_DYNAMODB
• EMAIL_BODY_DYNAMODB
• For Email#send:
  • EMAIL_SENDER_LAMBDA
  • EMAIL_SENDER_QUEUE
  • EMAIL_SENDER_SERVICE
  • EMAIL_SENDER_CLUSTER
  • EMAIL_SENDER_TASKNAME
• For Email#construct:
  • EMAIL_CONSTRUCTOR_LAMBDA
  • EMAIL_CONSTRUCTOR_QUEUE
  • EMAIL_CONSTRUCTOR_SERVICE
  • EMAIL_CONSTRUCTOR_CLUSTER
  • EMAIL_CONSTRUCTOR_TASKNAME

Include this in your project by adding:

    "dependencies": {
        "pidash-email": "raisdev/pidash-email-npm-module#v1.0.8"
    }

Testing/linting

Run tests with npm test, and linting with npm run lint. Make sure that you have all the dependencies installed.

Architecture

When we want to create an email, we go through the following process:

Create emails on admin site -> (pushes to DynamoDB)
Click 'build' on admin site ->
Launches a lambda for each email, which:
    adds an item to an SQS queue
    creates (or updates) a AWS ECS service ->
The service starts up some tasks, each of which:
    Pulls an email off the queue(or quits the service if the queue is empty)
    Builds an email
    Updates its status
    Uploads the complete email to DynamoDB

The person waiting on the admin site can reload the page and
sees the email's status change (from 'generating' to 'generated')
they can review the email.

Click on 'send' on admin site ->
Launches (the same) lambda for each email, which:
    adds an item to an SQS queue
    creates (or updates) a AWS service ->
Each of the service's tasks:
    Pulls an email off the queue (or quits the service if the queue is empty)
    Sends the email (after some basic validating)
    Updates the email's status

One of the unexpected side effects is that, to change the ECS service, you need to delete the existing ECS services and then modify the Lambda to use the new service.

Amazon has put together [an example repository (Lambda ECS Worker Pattern)][ecs-worker-pattern] that has a somewhat similar architecture. [ecs-worker-pattern]: https://github.com/awslabs/lambda-ecs-worker-pattern

AWS resources

Lambda

• emailLambda (ECS & SQS permissions)

SQS

• emailSenderQueue
• emailConstructorQueue

ECS (each of these need to be in some cluster)

• some-admin-service
  • some-admin-task
    • admin container
• emailConstructors service
  • constructEmail task
    • emailConstructor container
• emailSenders service
  • sendEmail task
    • emailSender container

DynamoDB

• email_batch (batch, date)
• email_body (netid, date, batch, status)
• email_metadata (netid, date, body)
• pidash-admin-sessions (setup by admin)
• pidash-admin-users (setup by admin

SES

• fromEmailAddress@cornell.edu Email Address Identity

IAM

• ECS instance role
  • DynamoDB full access
  • SES full access
  • EC2 Container Service for EC2
  • SQS full access (reading from queue, removing from queue)
• Lambda Role
  • ECS full access (create, update, describe, etc...)
  • SQS full access (only uses adding to queue)