polling-infinity
v1.6.1-polling-req-headers.0
Published
Periodically request changes anywhere!
Downloads
3
Readme
Polling^Infinity
Poll an EHR webservice for changes and send them through the RedoxEngine.
Design: https://docs.google.com/document/d/1POBcWWog8eOp_FiLVjWDEOA13gIQrpvLfmVLGyPIpWM/edit
Making Changes to the Code
If functionality changes are made, it is expected that they are documented in the web sequence diagrams in the diagrams
directory of this service. To view the diagrams and modify within a GUI, please install the VSCode Draw.io integration
extension by Henning Dieterichs.
RedoxLocal (rl-k8s) Commands
Polling^Infinity uses rl
to run local instances of the services that make up the entire polling platform. Refer to the RL setup documentation for instructions on getting your machine prepared to use RL.
The following manifest targets are available for this service:
rl run polling-infinity
- Provisions and starts the Web API service in Kubernetes, running the migrations job (see below) and launching apolling-scripterator
web API andrl-postgres-redoxservices
services viaredox.yml
dependency (if not already running).rl run polling-infinity:migrate
- Creates the migration job in Kubernetes, which applies migrations to thepollinginfinity
database inrl-postgres-redoxservices
.rl run polling-infinity:cleanup
- Provisions and starts the clean-up cron job service in Kubernetes, immediately running all jobs.rl run polling-infinity:workflow-sequential
- Provisions and starts the Workflow stage FQS Sequential worker service in Kubernetes.rl run polling-infinity:supplemental-sequential
- Provisions and starts the Supplemental stage FQS Sequential worker service in Kubernetes.rl run polling-infinity:workflowfinalize-sequential
- Provisions and starts the Workflow-Finalizer stage FQS Sequential worker service in Kubernetes.rl run polling-infinity:send-sequential
- Provisions and starts the Send-To-Engine stage FQS Sequential worker service in Kubernetes.
Note that the above run
commands use the latest built/released version of the Docker image for polling-infinity
, not your local code/changes. In order to create an instance with your local code, you need to use rl dev <target>
instead. Note that "automatically" run dependencies (like migrations and scripterator) don't "inherit" the dev
command — rl dev polling-infinity
will result in rl run polling-infinity:migrate
being run, which can be surprising. Remember to run those "upstream" dependencies manually with rl dev <target>
first to get local code in the containers.
If you need to reset the Docker image based on your local code (e.g. if rl dev <target>
seems to show old code), you have to possibly do some of the following:
rl remove <target>
to clear things out- Use Lens or similar commands to remove things like Jobs, ConfigMaps, or other things you want to rebuild "from scratch".
- Delete the docker image on your local machine with something like
docker image rm <image-ID>
.
Running Local Tests
Local tests can be run with cicd module test
, which uses the test-compose.yml
file to instantiate services for integration tests. There are commented out volumes
mount points and run-forever-do-nothing command
versions in that file. By un-commenting the volume mount and swapping the command
line for the commented out version, the local-test runner script will start up the container and do nothing, but apply your code edits to the container as you make them. That means you can docker exec -it
into the test container and run tests yourself with something like npm run test
and edit/re-run them without lift/stop delays.
Development Commands
npm run db:create
creates a dev and test postgres database.npm run db:setup
runs latest migrations for dev and test databases.npm run db:migrate:make
creates a new migration file in/src/db/migrations
.npm run db:migrate:latest
runs "up" migrations.npm run db:migrate:rollback
runs "down" migrations.npm run lint
runstslint
and displays errors.npm run lint:clean
runstslint
and fixes linting errors when possible.npm run test
compiles the typescript and then runs all tests in thelib/**/*.spec.ts
files. A code coverage report is also generated, and will display coverage errors if there are any.npm test:debug
runsnpm run test
in debug mode.npm start <worker type>
starts a worker and watches for any changes to the .ts files. When there are changes, it recompiles the typescript and restarts. Worker types include:scheduler
- Keep track of source build in the RedoxEngine and periodically place jobs on the polling queue table.workflow
- Grab jobs off the polling queue table and perform HTTP requests to generate a batch of changed records. Place the batch of changed records on FQS using the fqs-enqueuer service.supplemental concurrent
- Perform HTTP requests to generate extra information for changed records in a batch, then create a list of messages that are ready to be sent to the RedoxEngine.supplemental sequential
- Place individual messages from the batch on FQS using the fqs-enqueuer service.send concurrent
- No work to do in this step yet; messages are simply passed along to the the sequential worker.send sequential
- Attempt to send the message into the RedoxEngine.cleanup
- Every few minutes, drops the entries in error and fqs archive tables that are older than a day or two.