@siteone/gitlab-ci-tools
v1.0.7
Published
**NOTE:** **This is a work in progress, please note that it's far from stable version. It's in usable state, but still in development. There are a good chance the configuration and API will change in the future. Also the docs are still work in progress.**
Downloads
413
Keywords
Readme
Deploy package
NOTE: This is a work in progress, please note that it's far from stable version. It's in usable state, but still in development. There are a good chance the configuration and API will change in the future. Also the docs are still work in progress.
Table of contents
// TODO
Introduction
This package main purpose is to provide basic tools to help you implement gitlab CI deployment process for projects. We use predefined (SiteOne standard) way to do this. If your project requires any changes to this standard process you can override or extend any part of the process.
It's created for deploying javascript frontend projects. In theory it can be used to deploy any project, but configuration is in javascript.
There are three main parts of this package:
- CI jobs task runner - Run any scripts in predefined CI stages
- Deploy tools - Provide tools to upload, start, stop and delete your build from server during this stages
- CLI - to help you create your gitlab-ci.yml config and create tags
No previous knowledge of .gitlab-ci.yml is required, but it really helps to understand the concept. For the deploy part we are using PM2 process manager.
NOTE: You will need gitlab project with CI/CD enabled and configured runners (it's beyond scope of this package).
SiteOne deploy process
SiteOne deploy process consists 2 stages (potentially 3 jobs)
- Build
- Deploy
- Stop (deploy stage but different job (stop action), when
auto_stop_in
is configured)
You can setup unlimited environments, with different configuration. Also there is a special isMultiEnv
option, which allows deploying multiple instances of the same environment. More on that later.
Deployment is initiated by creating new tag in gitlab matching certain regExp, don't worry there is a CLI command that helps you create thees tags.
Understanding "CI jobs task runner" part
This part helps with dealing with the CI part of your deployment process. It's just a tool to enable you to run any shell commands during your CI jobs. Now it supports only this predefined three jobs mentioned above. You can run unlimited scripts and processes during these jobs, so it should fit most of your needs.
We use three predefined jobs in two stages
1. Build
In this job build
(default) folder should be created. This build folder is saved as artifact to gitlab and used later in deploy stage. You can also run tests, and other code quality tools in this job, to make sure everything is ok and ready to deploy.
2. Deploy
This job should serve to upload previously created build artifact to your server start the build and make sure it's running. Potentially stop and delete old version of app if there is already one deployed.
3. Stop
This jobs is configured only for isMultiEnv=true
environments. It can be triggered manually from gitlab, or you can set auto_stop_in
option in you environment config. It's purpose is to stop and delete preview (review) of an app that is no longer needed.
NOTE: We are considering merging build and deploy job to one, to save time installing dependencies and setting up the runner.
Process flow chart
graph TD
subgraph create tag
tag_created(Tag created) --> environment{environment}
environment --> |"match"|build([build])
environment --> |no match|E(end)
build --> deployManual{deployManual}
deployManual --> |false| deploy([environnementName_deploy job])
deployManual --> |true| trigger_deploy([manual deploy job])
end
subgraph stop environment
stop_environment("stop environment triggered <br> (manual in gitlab or autostop)") --> stop_job([environnementName_stop job])
end
subgraph trigger deploy
trigger_deploy(Manually start <br> deploy production job in gitlab) --> manual_deploy([environnementName_deploy job])
end
deploy -.-> autostop
manual_deploy -.-> autostop
autostop{auto_stop_in}
autostop --> |false| E
autostop --> |true| stop_environment
stop_job --> E
Configuration (js)
1. ci.config.js
Configuration contains 3 main sections corresponding to gitlab CI jobs (build
, deploy
, stop
). Each job is divided to three phases: before
, main
, after
executed in this order. It really doesn't meter. You can execute all the script in main section and the result will be the same. It's just for easier organization of your tasks and it corresponds to gitlab CI job settings.
Each section corresponds to CiScripts type. The config it self is deep partial of the CiConfig. It's later merged with defaults and validated, so you don't need to fill all values.
type CiScripts {
before: string[], // Array of shell commands
main: string[], // Array of shell commands
after: string[], // Array of shell commands
}
type CiConfig {
build: CiScripts,
deploy: CiScripts,
stop: CiScripts,
}
So for example
const config: PartialCiConfig = {
build: {
main: [
"yarn test",
"yarn build",
"yarn start",
"yarn healthcheck:local"
],
},
deploy: {
main: ['yarn ci:deploy']
after: ['yarn notification:deploy:done']
}
stop: {
main: ['yarn ci:stop'],
}
};
NOTE: Implementing this npm scripts is developer responsibility. It's pretty much standard supported by all popular javascript frontend frameworks. This package will help you with the yarn ci:deploy
and yarn ci:stop
. Also yarn install
is executed at start of every job in Gitlab runner (this package wont work if it wasn't).
2. setup gitlab ci/cd env vars
Next you will need to setup CI/CD environment variable in gitlab. First create CI_TOOLS_IMAGE
variable. It's image gitlab runner should run your deploy scripts on. We are recommend alpine version and locking it to concrete version to ensure updates wont break your deploy process. It has to contain node and yarn installed. For example:
node:12.18.3-alpine
3. Gitlab .ci-config.yml
First you need to create .ci-config.yml
file in root of your repository (gitlab will recognize it and consume if CI/CI is enabled without any other setup). Than you can include your gitlab-ci config.
include: ./ci-tools-gl-config.yml
This ci-tools-gl-config.yml
config file is generated with our CLI script, based on your deploy.config.js
. More on this later.
Triggering deploy process
To trigger deploy process just create new tag in gitlab. This tag must match certain regex to start the process. It differs by your environment settings:
// WIP // TODO describe default env tag regexes.
The best way to create your tag is use this package CLI command createEnv
. More on this in CLI section.
Stop preview environment
If you configure auto_stop_in option your environment will auto stop in specified time. You can stop them manually in your gitlab repository environments (operations -> environments) by clicking the stop button. You can also prevent this environment from-auto stopping using button in gitlab environments page.
This is basically all you need to setup your deploy process with gitlab CI, at least the CI part.
Understanding "Deploy tools" part
This part provide some tools to:
- Create ssh connection to your severs and execute any command on this servers
- Upload and delete files to and from you servers
- Start, stop, delete processes from PM2
- Perform basic health check on just deployed app
You can use all this tools separately, override or extend them.
How it works
Deploy script is started using CLI command gitlab-ci-tools deploy
(more about this in CLI section) or you can create your custom script importing deploy
function from this package. Than add this script to your package json scripts.
There are two possible actions (--action parameter in CLI script on 3rd parameter on deploy function) default
and stop
Default Default action purpose is to copy your build folder to server, stop and remove old build (if present), start new build (in PM2) and make sure it's running. It's divided to 4 steps:
- Deploy before
- Deploy upload
- Deploy start
- Deploy after
Stop
Stop action is meant to stop and remove deployed build from server.
Deploy function flow chart
graph TD
cli_deploy[run deploy function] --> setup_dynamic_args["deployConfig.setupDynamicArgs || environmentConfig.setupDynamicArgs || nill"]
setup_dynamic_args --> action{action}
action --> |default| deploy_before["environmentConfig.deployBefore || deployConfig.deployBefore || nill"]
deploy_before --> deploy_upload["environmentConfig.deployUpload || deployConfig.deployUpload"]
deploy_upload --> deploy_start["environmentConfig.deployStart || deployConfig.deployStart"]
deploy_start --> deploy_after["environmentConfig.deployAfter || deployConfig.deployAfter || nill"]
action --> |stop| on_stop["environmentConfig.onStop || deployConfig.onStop"]
Configuration
Work in progress. Meanwhile please check DeployConfig in @siteone/gitlab-ci-tools/dist/configuration/interface
// TODO in depth configuration (ci, preview, env vars)
CLI
Work in progress. Meanwhile please check gitlab-ci-tools --help
(node_modules/.bin/gitlab-ci-tools) for more information.
// WIP
Api documentation
// TODO in depth api docs
Debugging
If you need more debug information you can set (in gitlab ci/cd settings) CI_TOOLS_DEBUG
environment variable for more verbose output.
Available levels are (using loglevel
library):
- TRACE
- DEBUG
- INFO
- WARN
- ERROR
- SILENT
Any other non falsy value will set loglevel to TRACE
.
Examples
Basic example (production deploy only)
We will start with minimum configuration example. Please note that this is simplest possible example, most of this can be changed or overridden.
Pre requisites
- node/js project
build
command (in package json scripts) that creates./build
folder and all externalities are bundled in, so your project don't neednode_modules
folder to run.- entry point (
build/index.js
) that starts your project's server
- gitlab with enabled ci/ci and configured runners.
- configured server(s) with node, pm2 installed where you wish to deploy your project.
Installation
yarn add @siteone/gitlab-ci-tools --dev
Configuration
package.json
In your package.json add ci
script:
"scripts": {
"build": "webpack --mode=production",
"ci": "gitlab-ci-tools",
"ci:createEnv": "gitlab-ci-tools createEnv",
"ci:deploy": "gitlab-ci-tools deploy",
"ci:stop": "gitlab-ci-tools deploy --action=stop",
"ci:createCiConfig": "gitlab-ci-tools createCiConfig"
}
deploy.config.js
Create file deploy.config.js and add following minimal configuration.
const KEY_MASTER = process.env.KEY_MASTER || "UNSET OUTSIDE RUNNER"
module.exports = {
environments: [
{
environmentName: "production",
environmentTagCreate: createTag,
environmentTagParseVersion: parseTag,
environmentTagIsSemver: true,
environmentTagIsVersioned: true,
defaultEnvironmentBranch: 'master',
isMultiEnv: false,
deployManual: true,
uploadConfig: {
files: ["ecosystem.production.config.js"],
folders: ["build"],
uploadTo: "www",
},
servers: [
{
getPort: () => 8080,
getAppName: () => "my-testapp",
sshConfig: {
host: "my.server.com",
username: "deploy.user",
privateKey: KEY_MASTER,
},
},
],
},
};
ecosyste.production.config.js
Create pm2 ecosystem config file:
// Extract arguments
var appName = process.argv[process.argv.length - 2];
var appPort = process.argv[process.argv.length - 1];
module.exports = {
apps: [
{
name: appName,
script: "./build/index.js",
max_restarts: 50,
instances: 6,
exec_mode: "cluster",
log_date_format: "YYYY-MM-DD HH:mm Z",
source_map_support: true,
// env related settings
env: {
NODE_ENV: "production",
APP_PORT: appPort,
},
},
],
};
ci.config.js
Create your ci.config.js. In this example default values is enough, but file still needs to exist. So just export empty object.
module.exports = {};
gitlab-ci.yml
Use CLI to generate your ci-tools-gl-config.yml.
Run ci:createCiConfig
to generate your ci-tools-gl-config.yml
file.
Create (or update if you already have one) gitlab-ci.yml with following content.
include: ./ci-tools-gl-config.yml
Environment variables
In your project add environment variables to your gitlab ci/cd settings. In this example we need KEY_MASTER
, variable containing private key which gitlab runner can use to connect to your server over ssh.
You also need to setup CI_TOOLS_IMAGE
which is image to create your gitlab CI runner from. Search docker hub to find image that suits your needs. Something like lts-alpine
would be a good choice. It should be close to the server your project will run on.
Development
This package is using microbundle to create production build.
yarn dev
starts microbundle watch
- complie files when they change
yarn build
runs microbundle
- create build (dist) from source
More in microbundle package documentation