TSFlow

The CI/CD solution nobody asked for.

• Freedom to manage your TypeScript projects the way you want.
• Run .ts files as scripts without building (using SWC).
• Flexible replacement for npm commands.
  • Account for platform and OS differences
• Alternative to awkward bash files in CI/CD
• Framework for CI/CD workflows written in TypeScript with autocompletion.
• Server that runs workflows in response to webhooks, provided in a Docker image, test locally
  • Share data between runners and projects

Scripting • Workflows • Runner

Usecases

Examples for most usecases can be seen within the repository itself. The source code is quite straightforward for the most part and will be well documented as time goes on. Once the interface is finalised I will work on some serious templates and guides but for now I have included some minimal examples. Although in early stages, I am currently using all features successfully for my own projects in production, but do not recommend this for anyone else yet.

Scripting

1. Install the core tools (only required if you want to use the TSFlow helpers):

npm i -D @tsflow/core

2. Write a script:

// ./test-script.ts
import { runProcess } from "@tsflow/core/process"

await runProcess("node --version")

3. Install the CLI:

npm i -g @tsflow/cli

4. Run your script:

tsf run test-script

Notes

• This is effectively a wrapper for @swc-node/register but also allows .env configs to be shared with other TSFlow functionality.
• As long as TSFlow can resolve your imports you can use code from anywhere in your project.
• This allows scripts to be run without any node_modules or build steps if the imports are entirely TypeScript or installed globally.
• This means that buildable libraries must either:
• Have path aliases in tsconfig.json that resolve to the source files (not dist).
• Be already built and resolvable by package.json.

Env

These are the defaults used if not defined in a root-level .env file. Both are optional, can be overriden with CLI arguments, and the target path is only used as a shortcut when no file is provided for tsf run/wf. I would recommend setting it to the path of your workflow entrypoint. TSC_CONFIG_PATH is also supplied to @swc-node/register.

# .env
TSF_TARGET_PATH     =./tsflow.ts
TSF_TSCONFIG_PATH   =./tsconfig.json

Example

This is a script used within this project to publish to Verdaccio and npm:

import { type Project, projectsDefs } from "./src/meta"
import { assert, wait } from "@tsflow/util/com"
import { getPackageStructure } from "./src/util"
import { runProcess } from "@tsflow/core/process"

main()

// Args can be accessed as usual (but
// currently can not start with a hyphen)

async function main() {
    await publish(process.argv[2] === "prod")
}

async function publish(prod = false) {
    if (!prod) {
        // You can use runProcess without await
        // to run background tasks or services

        runProcess([
            "verdaccio",
            "--listen", "30333",
        ], { spawnOptions: { detached: true } })

        // You can used a string or string[] to define commands,
        // they are just joined together after removing blank values
        // (making it easy to use ternaries without leaving blanks)

        // Detached opens the process in a new terminal
        // (see type definition for more options)

        await wait(3000)
    }

    const registry = prod
        ? "https://registry.npmjs.org/"
        : "http://localhost:30333/"

    const packages = await getPackageStructure()

    // I'm using a definition file here for
    // package-specific operations

    const publishable = packages.dirs
        .filter((d) => projectsDefs[d.name as Project].publishable)

    for (const dir of publishable) {
        // Here, the cwd is being set to the root of each package

        const publish = await runProcess([
            "pnpm", "publish",
            "--access", "public",
            "--tag", "latest",
            "--no-git-checks",
            "--registry", registry,
        ], { cwd: dir.path })
        assert(publish.succeeded, "Failed to publish package")
    }
}

Workflows

A workflow is a structured set of tasks that run within a clean copy of your repo. It's the TSFlow equivalent to yaml pipeline definitions but can be run locally or on a server using the tsflow/tsflow-runner image on Docker Hub.

Workflows operate like this:

1. A workflow is triggered by either:
   • The tsflow/tsflow-runner server Docker image in response to web requests (repo webhooks).
   • Manually with tsf wf.
2. A config is created from env variables, web request payloads, and environment, then saved to .tsflow/data/requests/ci_${id}.json.
3. The repo directory is wiped and freshly cloned to .tsflow/repo/, based on the url provided (server), or by running git remote get-url origin (local).
4. Your workflow is then executed.
5. When start() is called within the workflow, the following happens:
   • The config file is loaded and payloads are exposed to workflow.
   • If defined, State is loaded (non-persistent storage accessible through the workflow).
   • If triggered by web request, it is validated based on the logic provided within the TSFlow constructor.
   • If defined, Store is loaded (persistent storage accessible through the workflow and saved to .tsflow/data/ci.store whenever saveStore() is called).
6. At this point you can run pretty much anything. The class exposes it's own wrappers for logging and functions like runProcess and logLine to better record the workflow.
   • Many more utils are available separately in @tsflow/core and @tsflow/util like file system operations, but will soon be integrated into the framework itself too.
7. Once everything's finished, finish() is called:
   • A summary of tasks is shown
   • Logs are saved to .tsflow/data/logs/ci_${id}.log

Here is a minimal example:

1. Install the workflow framework:

pnpm i -D @tsflow/workflow

1. Write a workflow:

// tsflow.ts
import { TSFlow } from "@tsflow/workflow"

const ci = new TSFlow()

await ci.start()
ci.logLine().log("WORKFLOW", "Listing files...")
await ci.run(["ls", "-lha"])
await ci.finish()

3. Run the workflow locally (requires @tsflow/cli):

tsf wf

Notes

• If testing locally, it is your responsibility to ensure the workflow is not unexpectedly relying on anything outside of the repo as this will not be reflected in production.
• If @tsflow/cli is installed globally, or using npx tsf wf, it is possible to run these workflows without depending on any TSFlow package???
• Unliked tasks, TSFlow must be able to resolve any imports on a freshly cloned repo.
  • If necessary, you can install globally, or build things outside of the repo, and it will persist until the container is destroyed.
• See here for a list of differences between TSFlow and a traditional CI solution.

Env

If running locally, the same variables apply as for tsf run.

Runner

Although you can run workflows locally, this doesn't allow you respond to changes in your repository. This is where the runner comes into play, available on Docker Hub (tsflow/tsflow-runner:latest).

Docker setup

I won't be going into the details of getting a home server set up but the process is quite straightforward and there are many tutorials online. The extent to which you secure yourself is entirely up to you. Alternatively you can use a cloud provider, or even host your own git repository and keep everything local.

Requirements

• Required:
  1. Docker Desktop, Azure Container App etc.
  2. External access to the container (port forwarding, allowing inbound).
  3. Webhooks from your git repository.
• Recommended:
  1. SSL certificate.
  2. Private domain.
  3. DDNS.

Here is a basic guide to run the image on Docker Desktop:

1. Install Docker Desktop.
2. Create .tsflow/ and .keys/ in the local copy of your repo and add them to your .gitignore.
3. Place your SSL certificates and keys in .keys/.
4. Update .env in the root of your local repo:
   • The key, cert, and ca should all match the names of the files within .keys/.
   • Add your passphrase too if required.
   • Alternatively, just set TSF_DISABLE_HTTPS=1 (not recommended, requires unencrypted access to your server).
   • Set the target path to the location of your workflow file.
   • Make sure your tsconfig.json path is correct.

   TSF_DISABLE_HTTPS   =
   TSF_SSL_KEY         =local-server.key
   TSF_SSL_CERT        =local-server.crt
   TSF_SSL_CA          =
   TSF_SSL_PASSPHRASE  =abcdefghijklmnopqrstuvwx
   TSF_TARGET_PATH     =bin/ci
   TSF_CLONE_URL       =https://<token>@github.com/c-jaye/tsflow.git
   TSF_TSCONFIG_PATH   =tsconfig.json

5. Start the container:

   docker run
       --sig-proxy=false
       --restart always
       --name tsflow-runner
       --publish 50555:60666
       --env-file C:/test-app/.env
       --volume C:/test-app/.keys/:/etc/ssl/certs/tsflow/
       --volume C:/test-app/.tsflow/data/:/tsflow/data/
       tsflow/tsflow-runner:latest

6. Setup webhooks for your repository to call your private domain and check the container logs to ensure the server is accepting requests.
7. You now have your own private CI/CD server.

Env

In addition to the main variables used for scripts and local workflows, these are also used for the runner:

# Must be set to "1" to disable https (not recommended until SSH is supported)
TSF_DISABLE_HTTPS   =
# These settings are the paths to the SSL files relative to the /etc/ssl/tsflow/ mount
TSF_SSL_KEY         =key.crt
TSF_SSL_CERT        =cert.crt
# Key and cert are required, but CA and passphrase are optional, refer to Node https settings
TSF_SSL_CA          =
TSF_SSL_PASSPHRASE  =
# This is required for TSFlow to know how to clone your repository. Currently, the best way to achieve this is with a read-only PAT
TSF_CLONE_URL:      =https://<token>@github.com/user/repo.git

Comparisons

|TSFlow|Gitlab CI etc.| |:-|:-| |Only node:lts-alpine|Any docker image| |Only one container|Multiple containers and images| |One project per runner|Multiple projects per runner| |Full access to runner while running|No access outside of cloned repo| |State can be shared between workflows|Artifacts| |State can be shared between projects|Artifacts| |Can be tested and ran locally|No testing| |Ability to import repo code directly|Shell and bash| |Autocompletion for headers, webhook payloads, etc.|Manual testing and validation| |Helper functions for affected apps*, validating GitHub tokens*, file system operations etc.|Integrations, env variables| |Not opinionated (outside of the environment)|Restrictive and opinionated, but highly compatible| |Easy dynamic workflows|Triggered child pipelines| |Easy to modularise, share configurable blocks and create plugins|Integrations?|