CloudPirate

CloudPirate is a set of deployment tasks based on top of flightplan allowing deployments to a dynamic set of aws ec2 instances.

Whats new?

• 2.0.x

  • Using own bin script wrapping flightplan. See below on how to upgrade from 1.x.x.
  • The S3 bucket name can now be configured with the option bucket which will override the value within the aws configuration file.
  • A special role of single can now be used to identify the first instance without the need for defining a tag. A tag may still be defined and will take priority if exists.
  • Commands can now optionally use a space to separate task from target rather than a colon.
  • Options with local paths now expand tilde.
  • Improved handling of symlinking shared items, copy deployment contents if no current shared item exists and remove before symlink.

• 1.5.x

  • Ability to specify aws configuration file location with the option awsConfig. This enables the ability to configure credentials in a different file location and/or use different credentials per application/environment.

Updating configuration from version 1.x to 2.x

• Application deployment configuration now lives within the file cloudpirate.json within the current working directory. To migrate, move the configuration defined within flightplan.js (the second parameter to the CloudPirate constructor) and place it within its own json file.
• Aws configuration, by default, has now moved to the file cloudpirate-aws.json within the current working directory. This file location can still be overriden with the option awsConfig.
• If you were additionally configuring the flightplan instance within flightplan.js this can still be achieved within the new cloudpirate.js file, which exposes the flightplan instance, see below.

Looking for version 1.x readme?

This can be found here.

Installation

# Install the cloudpirate cli tool
$ npm install -g cloudpirate

Usage

The task and target may be separated either with a space or colon.

# View help
$ cloudpirate -h

# List the target servers for the myapp application on the prod environment
$ cloudpirate servers myapp.prod

# List the versions deployed for the myapp application across environments
$ cloudpirate version myapp

# Build myapp and place the asset in S3.
# using the branch configuration, or prompted if false
$ cloudpirate build myapp

# Build myapp and place the asset in S3 for the prod environment.
# Required, if configuration is overridden at the environment level.
$ cloudpirate build myapp.prod

# Deploy a previously built myapp version to the prod environment
# using the branch configuration, or prompted if false
$ cloudpirate deploy myapp.prod

# Run the task1 task on the scripts role of the prod environment
# for the myapp application
$ cloudpirate task1 myapp.prod.scripts

# Run the task3 task on the special single role of the prod environment
# for the myapp application.
# The single role uses the first instance and does not need a role tag.
$ cloudpirate task3 myapp.prod.single

Example cloudpirate.json

The cloudpirate.json file defines the application deployment configuration, and should exist within the current working directory. Alternatively, the location of this file can be specified with the -f cli option.

{
    "config": {
        "keep": 5,
        "sshKey": "/home/me/.ssh/key.pem"
    },
    "apps": {
        "myapp": {
            "default": {
                "awsConfig": false,
                "bucket": false,
                "envs": [ "stage", "prod" ],
                "git": "[email protected]:bedican/cloudpirate.git",
                "branch": false,
                "defaultBranch": false,
                "sshKey": "/home/me/.ssh/myapp-key.pem",
                "username": "ubuntu",
                "path": "/home/ubuntu/apps/cloudpirate",
                "shared": [ "cache", "log" ],
                "build": "echo 'GO BANANA !'",
                "buildRemote": "echo 'IM A UNITARD'",
                "postSymlink": "echo 'I BENT MY WOOKIE'",
                "roles": {
                    "single": {
                        "task3": "echo 'WHY DO PEOPLE RUN FROM ME?'"
                    },
                    "scripts": {
                        "task1": "echo 'ME FAIL ENGLISH? THAT UMPOSSIBLE'",
                        "task2": "echo 'MY CATS BREATH SMELLS LIKE CAT FOOD'"
                    }
                }
            },
            "preprod": {
                "sshKey": "/home/me/.ssh/myapp-preprod-key.pem"
            }
        }
    }
}

Options

Depending on the application and environment selected, configuration is determined by merging the default and the environment configuration under the selected application, along with the common config configuration.

Environments can be defined by either creating overriding configuration options at the same level as default under the application, or if there is no specific configuration needed, added to a envs configuration option. In the example cloudpirate.json above, the myapp application has three environments: stage, prod and preprod, where the preprod environment has an overridden sshKey option.

| Option | Description | | ------------- | --------------------------------------------------------------------------------------------------------------------------- | | git | The git repository of the application. The assumption is made that access is already configured to clone the repository. | | sshKey | The ssh key to use when connecting to ec2 instances. | | username | The username to use when connecting to ec2 instances. | | path | The remote path the project will reside under. | | keep | How many releases to keep remotely. Set to false not to cleanup old releases. | | shared | A list of files or directories that should remain constant across releases. e.g. a cache or log directory. | | branch | The branch to build or deploy. If false, you will be prompted. | | defaultBranch | If the branch option is false, you will be prompted either using this value as a default, or master if false. | | build | The build command to run locally relative to the project root used when running the build task. | | buildRemote | The build command to run remotely relative to the project root used when running the deploy task. This is executed before creating the current symlink. | | postSymlink | The command to run remotely relative to the project root used when running the deploy task. This is executed after creating the current symlink. | | envs | A list of environments if not defined with their own overriding configuration. | | roles | Used to define roles and tasks (commands) that can be run against them within the application. This allows targeted subset of ec2 instances using the cloudpirate.role tag. | | servers | To define a static list of servers if desired, see below. | | awsConfig | The filename of the aws configuration file, see below. Defaults to cloudpirate-aws.json within the current working directory. | | bucket | The S3 bucket name, if false or omitted, the value within cloudpirate-aws.json will be used. |

Example cloudpirate.js

The optional cloudpirate.js file provides an mechanism to access the cloudpirate and flightplan instances, and should exist within the current working directory. Alternatively, the location of this file can be specified with the -e cli option.

module.exports = function(cloudpirate, config) {
    var flightplan = cloudpirate.getPlan();
    // ...
};

Roles

Roles specify the purpose of an instance, and enable a subset of instances within an application for an environment to be defined using the optional cloudpirate.role tag.

Roles are defined using the roles configuration option, with a mapping of role and the tasks that can be invoked.

If using auto scaling groups, in order to tag ec2 instances differently, multiple auto scaling groups are required, each with a different cloudpirate.role tag value.

The single role

The single role is a special role that does not require a tag on any instances. Instead, a single instance will be selected to run the task against.

For example, in the case where a database migration script is required to be run.

It is still possible to define a single tag on one or more instances, which will take priority if defined.

Remote application layout

Based from the path configuration, each application will be laid out with the following.

| Path | Type | Description | | ---------- | --------- | ------------------------------------------------------------------------------------------------------ | | ./releases | Directory | Contains the deployed code, each release within its own subdirectory. | | ./shared | Directory | Each subdirectory is defined within the shared configuration and is symlinked out from each release. | | ./current | Symlink | The currently deployed release within the ./releases directory. | | ./previous | Symlink | The previously deployed release within the ./releases directory. |

Aws

Tagging

In order to use CloudPirate, ec2 instances need to be tagged with the following tags. These can be added within the aws console for each instance manually, within an auto scaling group, or defined within a cloud formation template. Tags can consist of multiple values as a comma separated list.

| Tag Name | Description | Required | | ---------------- | --------------------------------------------- | -------- | | cloudpirate.env | The environnent the instance resides in. | Yes | | cloudpirate.app | The application being deployed. | Yes | | cloudpirate.role | The role of the instance for the application. | No |

Configuration

Aws configuration, by default, resides within the file cloudpirate-aws.json in the current working directory. The location of this file can be overridden globally, per-application or per-environment by making use of the awsConfig option.

The IAM user should have privileges to upload and download to the specified bucket and be able to list ec2 instances.

{
    "aws": {
        "accessKeyId":"<access key>",
        "secretAccessKey":"<secret access key>",
        "region":"<region>"
    },
    "s3": {
        "bucket": "<bucket>"
    }
}

The aws credentials may be omitted from this file, in favour of the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. This will, however, remove the ability to set credentials on a per-application basis with the option awsConfig.

Usage with Launch Configuration

New ec2 instances can download the current deployed release as they are created, by making use of the UserData parameter within the Launch Configuration.

The following example demonstrates how a release is taken from S3, making use of the awscli tool.

#!/bin/bash -ex

BUCKET=mybucket
APPSDIR=/home/ubuntu/apps

function deploy {

    mkdir -p $APPSDIR/$2/init-deploy

    aws s3 cp s3://$BUCKET/$1/current-version-prod $APPSDIR/$2/init-deploy/current-version-prod
    VERSION=$(cat $APPSDIR/$2/init-deploy/current-version-prod)

    aws s3 cp s3://$BUCKET/$1/$VERSION $APPSDIR/$2/init-deploy/$VERSION
    tar -xz --strip-components=1 -C $APPSDIR/$2/init-deploy -f $APPSDIR/$2/init-deploy/$VERSION

    rm $APPSDIR/$2/init-deploy/current-version-prod $APPSDIR/$2/init-deploy/$VERSION

    # App project hook, to keep init scripts within the project
    if [ -f $APPSDIR/$2/init-deploy/bin/ec2-init ]; then
        $APPSDIR/$2/init-deploy/bin/ec2-init
    fi

    ln -s $APPSDIR/$2/init-deploy $APPSDIR/$2/current
}

deploy myapp myapp.co.uk

But I don't want to use dynamic instances and just need a static list of servers!

While this tool was written with aws in mind, with a dynamic pool of instances, it is possible to use with a static list of servers.

This can be done by defining a servers configuration option.

{
    "apps": {
        "myapp": {
            "default": {
                "servers": {
                    "default": [ "hostname1", "hostname2" ],
                    "role1": [ "hostname1" ]
                }
            }
        }
    }
}

License: MIT

Author: Ash Brown