@johnlewisandpartners/composer-dag-trigger
v1.0.4
Published
Module that makes it easy to trigger a dag on GCP composer
Downloads
8
Readme
Composer Dag Trigger
This is a module that can be used to trigger a DAG on GCP composer. It handles all the authentication so that this can be easily added to a cloud function.
It is adapted from https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf
Setup
To authenticate to Cloud IAP, grant the Appspot Service Account (used by Cloud Functions) the Service Account Token Creator role on itself. To do this, execute the following command in the gcloud command-line tool or Cloud Shell:
gcloud iam service-accounts add-iam-policy-binding \
[email protected] \
--member=serviceAccount:[email protected] \
--role=roles/iam.serviceAccountTokenCreator
Usage
const composerTriggerDag = require('@johnlewisandpartners/composer-dag-trigger');
exports.triggerDag = function triggerDag(event, callback) {
const DAG_NAME = 'my_dag';
const CURRENT_DATE = new Date();
const runId = `my-trigger-run_${CURRENT_DATE.toISOString()}`;
const data = {
data_to_pass: 'test_data'
};
const AIRFLOW_WEB_URL = 'https://xxxxxxxxxxxxxxxxx-tp.appspot.com'
const PROJECT_ID = 'your-project-id'
const CLIENT_ID = 'xxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com'
const triggerDagParams = {
dagName: DAG_NAME,
runId,
data,
composerWebUrl: AIRFLOW_WEB_URL,
projectId: PROJECT_ID,
clientId: CLIENT_ID,
callback,
};
composerTriggerDag.triggerDag(triggerDagParams);
}
DAG_NAME should be the name of the dag you want to trigger.
runId is an optional param allowing you to specify the run id in airflow.
data is a JSON object containing data that you want to pass to the dag. This is available in the dag in the conf object.
AIRFLOW_WEB_URL should be the base URL of your composer instance.
PROJECT_ID should be the GCP project that airflow is in.
CLIENT_ID is the client id of the identity aware proxy running in front of composer. This can either be retrieved by navigating to composer and taking it from the sign in page, or by using this command
COMPOSER_ENVIRONMENT=mycomposerinstance
COMPOSER_LOCATION=europe-west-1
AIRFLOW_WEB_URL=`gcloud composer environments describe ${COMPOSER_ENVIRONMENT} --location ${COMPOSER_LOCATION} --format="get(config.airflow_uri)"`
# get the client id to be used in the trigger script so it can do proper oauth
CLIENT_ID=`curl ${AIRFLOW_WEB_URL} -I -s | grep -i "^location: "|grep -o "[?&]client_id=[^&]*"|cut -d= -f2`
echo $CLIENT_ID