AuditBoard Development Environment

This project contains setup to configure a new AuditBoard contributor with a working local development environment.

If you run into any issues, please consult the FAQ section, and finally ask for help in the #engineering Slack channel.

Pre-Requisites

Depending on your operating system, install the necessary pre-requisites.

Debian / Ubuntu

• install curl, docker, git, postgresql, redis
• create your user as a superuser in postgresql

Check out some notes for Ubuntu

Mac

• Follow Setup, most of the dependencies will be added with the install-dependencies command in the steps below.

A Note on Git

Be sure to configure git to sign your commits

Setup

1. Generate an SSH key

ssh-keygen -t rsa

Confirm that it will be stored in ~/.ssh/id_rsa. You may want to leave your passphrase empty.

cat ~/.ssh/id_rsa.pub

2. Add the SSH key to github at https://github.com/settings/keys

   (copy the text starting with ssh-rsa)

3. Clone dev env repo

git clone [email protected]:soxhub/auditboard-dev-env.git && cd auditboard-dev-env/

4. Install dependencies
   • NOTE: This script is only compatible with macOS, Debian, and Ubuntu! If you are using a different OS, then please install all dependencies manually.

./bin/install-dependencies

This will use the Brewfile at the root of this project to install redis, postgres, volta and more. Let's pause and do a quick check to see if things are up and running.

brew services list

This should output at minimum postgres and redis. Also, redis-cli should work as well.

5. Create github token and log in to npm

Create a GitHub personal token (classic) with the read:packages box checked as shown below.

Generate the token and copy it to your clipboard.

Still in GitHub, go to: Settings >> Developr Settings >> Click Configure SSO on your newly generated token >> Authorize soxhub

6. Back in your terminal log into npm:

npm login --scope=@soxhub --registry=https://npm.pkg.github.com

Username: <GITHUB_USERNAME>
Password: <PASTE_GITHUB_TOKEN>
Email: <GITHUB_EMAIL_ADDRESS>

7. There are two possible layouts for auditboard-backend and auditboard-frontend:

   • This is the preferred folder structure, to keep auditboard-backend and auditboard-frontend as sibling directories to auditboard-dev-env. HOWEVER, you must explicitly opt in to this structure, by manually git cloneing the repos, before running the setup script.

   .
   └── parent-folder
       ├── auditboard-dev-env
       ├── auditboard-backend
       └── auditboard-frontend

   • By default, the setup script will clone the repos using the layout below:

   .
   └── parent-folder
       └── auditboard-dev-env
           ├── auditboard-backend
           └── auditboard-frontend

   Git clone commands:

   cd ..
   git clone [email protected]:soxhub/auditboard-backend.git
   git clone [email protected]:soxhub/auditboard-frontend.git
   cd auditboard-dev-env/

8. Mac M1

   Mac M1 users may encounter errors if you have not already installed Rosetta, so you're advised to run:

   softwareupdate --install-rosetta

9. Linux

   • Before proceeding, please ensure that the services docker, redis, and postgresql have been started. You can do so by running systemctl | grep ${SERVICE_NAME}
   • If a service hasn't been started yet, you can start it by running sudo systemctl start ${SERVICE_NAME}
   • To configure the service to auto-start when booting up your machine, run sudo systemctl enable ${SERVICE_NAME}
   • Docker:
     • To get access to the docker daemon, you may need to add your user to the docker group. See the post-install instructions

10. Build dev environment and add environmental variables

./bin/create-dev-environment

PLEASE READ BEFORE RUNNING THE SCRIPT:

• Watch the terminal while the script is running - it may ask for your help:
  • Manually installing a missing dependency
  • Provide your password for some sudo configurations
  • Re-run the script in a new terminal window
• If execution of the script halts due to the existing file: /Users/[email protected]/Library/LaunchAgents//homebrew.mxcl.postgresql.plist simply rm the file and re-run the script
• The script will git clone repos for various background services into the auditboard-dev-env directory. To start/stop these services, there are a few terminal commands you can use (more on this later).
• During the database setup, the script will ask if you would like to reset credentials for [email protected]. Please respond with y. This will set up a superuser account for you, with the email [email protected], and the password that you selected.
• Make sure to run ./bin/reset-testdb. Our integration tests run against a whole other db so run this to reset the tables there with the latest migrations.
• If execution fails with Error: Bindings not found it's possible we need to tell Volta we're using PNPM. In your shell execute the following:
  • | Shell | Command | | ----- | ------------------------------------------------------------------ | | bash | echo "export VOLTA_FEATURE_PNPM=1" >> ~/.bashrc | | zsh | echo "export VOLTA_FEATURE_PNPM=1" >> ~/.zshrc | | fish | echo "set -x VOLTA_FEATURE_PNPM 1" >> ~/.config/fish/config.fish |
  • Then rerun ./bin/create-dev-environment. More info here.

Usage

The following instructions will start all services inside docker containers. If you would rather just simply start the client and API, please refer to Simple Usage.

Start all background services, and the API, using docker compose and nodemon:

start-all

Note

The start-all script starts all of the backend, but does not start the client. To start the client, follow the client instructions below.

You can choose to start only background services by:

• Adding export START_COMPOSE_ONLY=true in your .envrc OR

• Running

   ./bin/start-background

Stop any services started by start-background:

stop-background

Start the client only:

cd ./auditboard-frontend/apps/client
pnpm install
pnpm run start # need to install dependencies separately from the start script

Note: For Apple Silicon (M1 etc.)

If you run into EMFILE errors, run the following:

brew install watchman

If you still face the same issue you may want to increase ulimit:

sudo launchctl limit maxfiles 16384 16384 && ulimit -n 16384

Source: https://github.com/facebook/watchman/issues/923#issuecomment-963669204

Start the API server only:

start-api

Reset the database. This drops all tables in the db, then restores using the .sql.zip or .sql file located in ./workspace.

reset-db

To skip user confirmations, use the -y flag:

reset-db -y

Background Processes

A variety of background processes all run under Docker. You can see what's running with docker ps.

API

The API runs via the https://nodemon.io/ process manager.

Client

We have two ways to access the Ember client. Please see comparison below:

| | Hot Reloading | Pusher Notifications | Realtime Updates | | --------------------- | ------------- | -------------------- | ---------------- | | http://localhost:9002 | ✅ | ❌ | ❌ | | https://localhost | ❌ | ✅ | ✅ |

NOTE that https://localhost uses the nginx proxy + https. The nginx proxy can be started via start-all or start-background.

Nginx notes

To test changes to Nginx (e.g. the Dockerfile or our config tmpl), See https://github.com/soxhub/soxhub-nginx and keep the following in mind:

• Make sure you have a "soxhub-nginx" working folder under auditboard-dev-env, and do your changes in there. That way docker compose will pick up your changes when you rebuild as you iterate.
• Each time you make changes to the nginx dockerfile or configs/etc, make sure to do docker compose -f <config-file> build nginx and then docker compose stop and up commands.
• You'll need to follow the instructions in previous sections of this README to run the background services (which include nginx), as well as your frontend and backend.
• Make sure your .envrc file has accurate settings for SOXHUB_API_DIR and SOXHUB_CLIENT_DIR, relative to auditboard-dev-env, such as:
  • export SOXHUB_API_DIR=./auditboard-backend
  • export SOXHUB_CLIENT_DIR=./auditboard-frontend/apps/client
• Open http://localhost (yes, port 80) in the browser. It is configured to proxy to your running frontend/backend.
• When you're ready, make a PR on soxhub-nginx, get it approved+merged, then cut a GitHub release on the repo, which will kick off triggers that build a new image.
• Finally, you will need to update the client Dockerfile to reference the new nginx tag.

Upgrading

If you have an existing .envrc, you can update it with:

CREATE_ENVRC=true ./bin/create-dev-environment

Upgrading is optional, but users who use develop in sibling directories (ie. ../auditboard-backend and ../auditboard-frontend) may prefer to upgrade to have better out-of-box support.

Your current .envrc file will be saved as .envrc.backup.

Managing pnpm versions

Volta introduced experimental pnpm support with version 1.1.1 so project specific versions of pnpm in package.json should be respected

The install-dependencies script should handle this, but to enable manually you need to verify you have volta 1.1.1 or higher and have the environment variable VOLTA_FEATURE_PNPM=1

Environment configuration (optional)

Silencing direnv

direnv is verbose. If you do not want to see its logging all the time, add export DIRENV_LOG_FORMAT= this to your .zshrc like so:

export DIRENV_LOG_FORMAT=
eval "$(direnv hook zsh)" # This line is automatically added in the created-dev-environment script.

(Source)

FAQ

accessKeyId errors when running the api service

Sample Error:

1|soxhub-worker  | /Users/username/dev/auditboard-backend/app/lib/sox-mailer.js:80
1|soxhub-worker  | 			throw new Error('accessKeyId cannot be null. Please set on instantiation.');
1|soxhub-worker  | Error: accessKeyId cannot be null. Please set on instantiation.

Your scripts did not set up the environment variables properly.

Fix:

Run the following command in your auditboard-dev-env folder.

direnv allow

Incompatible Architecture Errors (Mac M1)

Sample Error:

ERROR: dlopen(/Users/username/dev/auditboard-backend/node_modules/bcrypt/lib/binding/napi-v3/bcrypt_lib.node, 0x0001): tried: '/Users/username/dev/auditboard-backend/node_modules/bcrypt/lib/binding/napi-v3/bcrypt_lib.node' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))

Fix:

You may have to run.

softwareupdate --install-rosetta

Depending on when you ran the rosetta command, you might have to re-install dependencies for auditboard-backend. Re-run pnpm install in the auditboard-backend folder.

pnpm install --force

Postgresql failures

If you are seeing brew have trouble starting the postgresql service, then check the log file at /opt/homebrew/var/log/postgres.log.

If you see an error like:

2022-11-08 07:05:00.512 +08 [13255] FATAL:  lock file "postmaster.pid" already exists
2022-11-08 07:05:00.512 +08 [13255] HINT:  Is another postmaster (PID 848) running in data directory "/opt/homebrew/var/postgres"?

Then the solution is to delete the postmaster.pid file, i.e. run rm /opt/homebrew/var/postgres/postmaster.pid.

start-all command failure

If you see this error when attempting to run start-all:

sync-all: command not found

Then you may have an issue with your direnv configuration. Try re-installing direnv by running:

brew reinstall direnv

Then add the following to your .zshrc or .bashrc:

eval "$(direnv hook zsh)"

Setup current Sales and Engineering dataset (preferred over reset-db)

You may need to run git lfs pull to get this step to work.

./bin/reset-db-platform-dataset

Alternatively a more manual approach would be:

dropdb demo_data
createdb demo_data
pg_restore -Fc -d demo_data < ./workspace/platform-dataset.dump

cd auditboard-backend
pnpm shell
> const user = await models.user.findByPk(1)
> user.setPassword('password')
> await user.save({ validate: false });

pnpm install && pnpm run db:sync

Simple Usage

Before starting the API and client, lets check that node is installed. cd into auditboard-frontend and auditboard-backend repositories separately and run the following.

which node
> /Users/myusername/.volta/bin/node

Notice how Node's executable is referenced from .volta.

Next, the Node version should be the same version in the package.json as the "node": value.

node -v
> vXX.XX.X

Ok you are ready to start the client and API.

Client

This will start the client on port :9002

pnpm run start:client

API

First, clone and rename launch-example.json to launch.json. Then in VSCode, start it by clicking on the button as shown in the image below.

This will start the API on port :9001.

Now your client and API can talk to each other!

Process to recreate dump

Maintaining a well built dataset is important for engineering, sales and customer support. If you see problems with your dataset when running locally, follow the commands to generate a new dump file and create a PR.

1. Make sure you are starting from a clean dataset
2. Modify changes via the UI or in your postgres client of choice.
3. Dump database to a file using the -Fc (format custom). This allows you to use pg_restore.

pg_dump -U postgres -Fc demo_data > ./workspace/platform-dataset.dump

4. PR changes with notes of approximately what data updated.