Model Job Wrapper

This library provides a wrapper around utilities to prepare a model process to run

Model overview

When a 'model' runs it needs to

1. Raise start events
2. copies the inputs from a file share on the azure batch node.
3. Produce a job.txt file
4. triggers the simulation model
5. Poll for updates to progress.txt
6. when the simulation process completes (successfully or not) icopies the outputs back to the file share after clearing the target directory
7. Raise stop or error events

Here is slightly more detailed overview of the activities that occur when this model is run.

The job-wrapper contains everything needed in a model repository to facilitate the above orchestration

Installation

The intention is that the job-wrapper is installed into a model project via additions to a Dockerfile

...
RUN mkdir /wrapper
RUN cd /wrapper && npm install [email protected]
RUN mv /wrapper/node_modules/capability-model-runner-job-wrapper/dockercmd.sh .
.....
CMD ./dockercmd.sh

Environment Parameters

The following environment parameters must be available to the wrapper as it runs.

These are properties exposed into the docker container use to connect to the job wrapper functionality. These should not be used by the simulation program as they are subject to change.

INPUT_PATHS=<The location where the inputs will be copied to on the batch VM>
OUTPUT_PATHS=<The location where the outputs will be copied to on the batch VM>

STAGE=<The identifier for the choreography stage that is running>

STORAGE_CONNECTION_STRING=<connection string> 
SHARE_NAME=<file share name>

EVENTHUB_CONNECTION_STRING=<connection string to the event hub namespace>
EVENTHUB_NAME=<name of the event hub into which events will be raised>

There are additional properties which

Properties related to the experiment

EXPERIMENT_ID=<The scenario or experiment configuration id>
EXPERIMENT_NAME=<The scenario or experiment configuration name>
REPICATION_NUMBER=<The Replication number>

File System

All model inputs will be download into INPUT_PATHS location The model outputs must be written into OUTPUT_PATHS directory

Events

This docker process raises a start event to the event hub before it starts the model. This docker process raises a stop event to the event hub after the model completes.

An additional process could be started to watch a file and raises more fine grained statuses if required.

Customizations for different model runtime

Besides copying the Docker and lib\job-wrapper.X.X.X.tgz to a model repository, 2 modifications are expected to be made to this project to 'hook in' different models.

The Dockerfile can be updated to install all dependencies required by the model.

FROM ubuntu:18.04

# Install extra components to be able to run the model here
....

Additionally the modelrun.sh command can be modified to start the process.

## Replace this content with whatever commands runs the model

Job.txt

A Job.txt file is produced into the INPUT_PATHS directory.

This file contains the following content.

key, value
seed, <seed number>
replicationNumber, <number>
runStart,  <date 2020-01-01>
runEnd,  <date 2020-01-01>
scenario, <scenaro id>

Simulation Processing Success or Failure Detection

The job wrapper expects the simulation job to return zero on a successful exit and non-zero on any error.