schedic - restic on steroids

schedic makes it easy to backup multiple hosts in your network using the fantastic restic tool.

Everything is put into one YAML configuration file on your backup server. There is no schedic configuration on the hosts you like to backup. They require just an installation of restic. The backup server executes restic remotely using SSH and passes all required information in a secure fashion.

restic is an ultra fast backup tool which performs super efficient deduplication and encryption of your backup data. It stores data locally or uploads to multiple different storage backends.

Contents

• Installation
  • From npm registry
  • Building from source
    • Automatic TypeScript transpilation
• Usage
  • Options
  • Commands
    • run
    • backup
    • forget
    • check
    • update
    • repo
• Backup configuration
• Cron configuration
• Example output of summaries
• Security considerations
  • Storing restic password
  • SSH access to all hosts
• CI pipeline
• License
• TODO / IDEAS

Installation

Prerequisites

schedic requires Node.js (minimum v8.x). It's recommended to use Yarn for installation, but npm is supported as well.

As well bash is required on the schedic host and on the target hosts. That's no problem with Unix operating systems, but additionals steps may be necessary to use schedic with Windows and probably Mac (bash is default shell since Mac OS 10.3).

schedic uses ssh (and assumes OpenSSH) to execute restic commands on the target hosts. By default it sets -o 'BatchMode Yes' to prevent interactive password prompts which causes the process to hang or wait infinitely. If you don't use OpenSSH and require different options to accomplish this you can specify them in the host-defaults configuration section (see below);

From npm registry

# Using Yarn
yarn global add schedic

# Or npm (but without package locking)
npm -g install schedic

Building from source

This will transpile the source from TypeScript to JavaScript and then links the schedic script globally.

# Using yarn
yarn && yarn build
yarn link

# Using npm (but without package locking)
npm install && npm run build
npm link

Automatic TypeScript transpilation

During development you should open a shell running a nodemon watcher for automatic transpilation:

# Using yarn
yarn watch

# Using npm
npm run watch

Usage

schedic command [options]

Options

• --config|c file.yml: specify configuration file, Default: ./schedic.yml, Environment: SCHEDIC_CONFIG
• --dry-run: just print commands, which would be executed
• --repo|r name: perform operations just for this repo, Default: all repos in configuration file
• --host|h name: perform operations just for this host, Default: all hosts in configuration file
• --path|p path: perform operation just for this backup path, Default: all paths in configuration file
• --sync-output: print log immediately, even if stdout is redirected

Commands

run

Performs a combined execution of the backup and forget commands. Refer to their descriptions below.

Example:

# Perform backup, forget & prune of everything in ./schedic.yml
schedic run

backup

Performs backups for all selected hosts and paths in the configuration file.

Examples:

# Backup all paths of all hosts listed in ./schedic.yml
schedic backup

# Backup all paths of host "homesrv"
schedic backup --host homesrv

# Backup /home of host "homesrv"
schedic backup --host homesrv --path /home

# Backup /home of host "homesrv" to repo "mycloud" only
schedic backup --host homesrv --path /home --repo mycloud

forget

Removes backups for all selected hosts and paths in the configuration file according to the configured forget policies. Afterwards restic prune is executed on the affected repos.

Examples:

# Remove all backups according to the configured "forget" policies
schedic forget

# Remove all backups of host "homesrv" according to the "forget" policies
schedic forget --host homesrv

# Remove all "/home" backups of host "homesrv" according to the "forget" policies
schedic forget --host homesrv --path /home

check

This command checks if all selected hosts are accessable using SSH and if the restic command is available (by executing restic version).

Examples:

# Check all configured hosts
schedic check

# Check host "homesrv"
schedic check --host homesrv

update

Runs restic self-update on all selected hosts.

Examples:

# Update restic on all configured hosts
schedic update

# Update restic on host "homesrv"
schedic update --host homesrv

repo

Prints shell code to set your environment to a specific restic repo.

Example:

# Select repository "mycloud"
eval `schedic repo mycloud`

# List all snapshots in selected repo
restic snapshots

Backup configuration

All backups for all hosts are configured in a single YAML file. schedic validates the schema and prints specific error messages, if the configuration file is invalid.

This is an annotated example configuration:

# Version of the configuration
version: "2.0"

# Repo definition
# ---------------
# The following properties are known for a repository:
# - name: an arbitrary name
# - url: repository URL
# - password-file: path to a file containing the repo password
# - password: or the repo password itself (not recommended)
# - restic: an object of additional restic options
#   use boolean values for switches, and numeric/string values
#   for options with parameters
# - env: required environment variables (e.g. auth keys)
#   keys may be prefixed with "<<" to express that the
#   value is a filename where the env var is read from
#   (to keep secrets out of this file; eol removed).
#   env variables can be overridden by host-defaults and
#   host sections beyond.
repos:
  - name: mycloud
    url: "rest:https://mycloud.example.com"
    password-file: "test/assets/.restic-mycloud.pw"
    restic:
      limit-upload: 900

  - name: local
    url: "/storage/backup/restic"
    password: "super secret local restic password"

  - name: s3
    url: "s3:s3.amazonaws.com/myresticbucket"
    password: "mys3awsresticpassword"
    env:
      AWS_ACCESS_KEY_ID: myawsaccesskey
      <<AWS_SECRET_ACCESS_KEY: test/assets/.restic.myawssecretaccesskey.pw

# Default Host Options
# --------------------
# Specify default options for all backup hosts. Can be
# overridden by each host.
# - ssh: the ssh command to use (e.g. for a non-default port)
#   or false, if command should be executed locally;
#   the "BatchMode" option is important to prevent hanging
#   of the process in interactive password prompts
# - arch: the CPU architecture (yet unused)
# - restic: see "repos[].restic"
# - env: see "repos[].env"
host-defaults:
  ssh: "ssh -o 'BatchMode Yes'"
  arch: linux_amd64
  restic:
    limit-upload: 500

# Default Backup Options
# ----------------------
# Specify default options for all backups. Can be overridden
# by each backup.
# - repos: list of repos to push this backup to
# - exclude: list of paths resp. globs to exclude from the backup
# - keep: your "forget" policy. All restic --keep-* options are supported
# - restic: see "repos[].restic"
backup-defaults:
  repos:
    - local
  exclude:
    - .gvfs
    - .thumbnails/
    - .cache/
  keep:
    daily: 3
    weekly: 4
    monthly: 6
  restic:
    one-file-system: true

# Host & Backup definition
# ------------------------
# Specify all hosts and paths to be backed up.
# - name: network hostname
# - pre: a list of commands to execute prior backup
# - ssh: the ssh command to use (e.g. for a non-default port)
#   or false, if command should be executed locally
# - arch: the CPU architecture (yet unused)
# - disabled: set to true, to disable a host (temporarily)
# - restic: see "repos[].restic"
# - env: see "repos[].env"
hosts:
  - name: homesrv
    ssh: false
    pre:
      - "cd /home/gitlab && ./mkbackup && mv $(ls -1atr /home/gitlab/data/gitlab/backups/* | tail -1) /home/backup/gitlab-latest.tar"
    env:
      PATH: /opt/restic/bin

    # Backup definition
    # -----------------
    # Specify all backup paths for this host. All settings here
    # are merged with the "backup-defaults" specified above.
    # - path: the local directory path
    # - tags: single tag or list of tags
    # - repos: list of repos to push this path to
    # - exclude: list of paths resp. globs to exclude from the backup
    # - restic: see "repos.restic"
    backups:
      - path: /home
        tags: important
        repos:
          - mycloud
          - local
          - s3
        exclude:
          - /home/gitlab
      - path: /etc
        tags: etc
      - path: /root
        tags: root-home
      - path: /var
        tags: var
        exclude:
          - /var/cache
          - /var/lib/docker
          - /var/log
      - path: /
        tags: system
        exclude:
          - /root
          - /var
      - path: /mnt/music
        tags: music
        keep:
          weekly: 2

  - name: raspberry1
    arch: linux_arm
    backups:
      - path: /etc
        tags: etc
      - path: /var
        tags: var
        exclude:
          - /var/cache
          - /var/log
      - path: /
        tags: system
        exclude:
          - /var

  - name: raspberry2
    arch: linux_arm
    disabled: true
    backups:
      - path: /etc
        tags: etc
      - path: /var
        tags: var
        exclude:
          - /var/cache
          - /var/log
      - path: /
        tags: system
        exclude:
          - /var

Cron configuration

This is an example cron configuration for regular operation:

# Daily backup and removal
11 3 * * * root schedic --config /usr/local/etc/schedic.yml run 2>&1 | mail -s "Daily schedic report" [email protected]

# Weekly restic self-update
1 3 * * 0 root schedic --config /usr/local/etc/schedic.yml update 2>&1 | mail -s "Weekly schedic update report" [email protected]

Example output of summaries

schedic prints summaries of all executed tasks. If the output is no terminal, all log messages are buffered and the summaries are printed first. This is very useful when you send the output with e-mail, so everything important is in front.

This is an example output of schedic run --dry-run with the shipped example schedic.yml (log messages are removed):

> Backup summary
> OK: All tasks successfully completed
╔════════════╤═════════╤═════════╤═══════════════════════════╤════╤══════╤══════════╤═════════╗
║ Host       │ Type    │ Repo    │ Details                   │ ID │ Size │ Duration │ Result  ║
╟────────────┼─────────┼─────────┼───────────────────────────┼────┼──────┼──────────┼─────────╢
║ homesrv    │ Command │         │ cd /home/gitlab && ./m... │    │      │    -     │ Dry run ║
║ homesrv    │ Backup  │ mycloud │ /home                     │ -  │ -    │    -     │ Dry run ║
║ homesrv    │ Backup  │ local   │ /home                     │ -  │ -    │    -     │ Dry run ║
║ homesrv    │ Backup  │ local   │ /etc                      │ -  │ -    │    -     │ Dry run ║
║ homesrv    │ Backup  │ local   │ /root                     │ -  │ -    │    -     │ Dry run ║
║ homesrv    │ Backup  │ local   │ /var                      │ -  │ -    │    -     │ Dry run ║
║ homesrv    │ Backup  │ local   │ /                         │ -  │ -    │    -     │ Dry run ║
║ homesrv    │ Backup  │ local   │ /mnt/music                │ -  │ -    │    -     │ Dry run ║
╟────────────┼─────────┼─────────┼───────────────────────────┼────┼──────┼──────────┼─────────╢
║ raspberry1 │ Backup  │ local   │ /etc                      │ -  │ -    │    -     │ Dry run ║
║ raspberry1 │ Backup  │ local   │ /var                      │ -  │ -    │    -     │ Dry run ║
║ raspberry1 │ Backup  │ local   │ /                         │ -  │ -    │    -     │ Dry run ║
╚════════════╧═════════╧═════════╧═══════════════════════════╧════╧══════╧══════════╧═════════╝

> Forget summary
> OK: All tasks successfully completed
╔════════════╤═════════╤═══════════════════════════╤══════╤═════════╤═════════╗
║ Host       │ Repo    │ Path                      │ Kept │ Removed │ Result  ║
╟────────────┼─────────┼───────────────────────────┼──────┼─────────┼─────────╢
║ homesrv    │ mycloud │ /home                     │ -    │ -       │ Dry run ║
║ homesrv    │ local   │ /home                     │ -    │ -       │ Dry run ║
║ homesrv    │ local   │ /etc                      │ -    │ -       │ Dry run ║
║ homesrv    │ local   │ /root                     │ -    │ -       │ Dry run ║
║ homesrv    │ local   │ /var                      │ -    │ -       │ Dry run ║
║ homesrv    │ local   │ /                         │ -    │ -       │ Dry run ║
║ homesrv    │ local   │ /mnt/music                │ -    │ -       │ Dry run ║
╟────────────┼─────────┼───────────────────────────┼──────┼─────────┼─────────╢
║ raspberry1 │ local   │ /etc                      │ -    │ -       │ Dry run ║
║ raspberry1 │ local   │ /var                      │ -    │ -       │ Dry run ║
║ raspberry1 │ local   │ /                         │ -    │ -       │ Dry run ║
╚════════════╧═════════╧═══════════════════════════╧══════╧═════════╧═════════╝

> Prune summary
> OK: All tasks successfully completed
╔═════════╤══════════╤═════════╗
║ Repo    │ Duration │ Result  ║
╟─────────┼──────────┼─────────╢
║ mycloud │ -        │ Dry run ║
║ local   │ -        │ Dry run ║
╚═════════╧══════════╧═════════╝

Security considerations

A few words about security.

Storing restic password

restic requires a password to encrypt all backup data. You have two options to store this passsord with schedic:

1. in a separate text file
2. directly in the schedic.yml file

It's recommended to use option 1) and important to mention, that only the schedic server has to know the password. There is no need to store this password on the hosts you back up. schedic will pass the password through SSH using a pipe, so it's even not visible in the processes environment.

You should use restricted permissions on the password file.

The advantage to not having the password in the schedic.yml is that it's easier to not disclosure it by accident - e.g. by showing someone else your schedic configuration or posting questions to stackoverflow ;)

SSH access to all hosts

In terms of security it's not optimal that the schedic server requires passwordless (key based) root access to all your hosts you want to back up. A person who gains access to your schedic server and the account you're running schedic with has also root access to all your hosts.

So you really have to take care to protect your schedic server.

On the other hand you can still make use of schedic if this open SSH configuration is no option for you. You could install schedic on all hosts and run it locally (besides restic you have to install anyway on every host). At least you have the benefit of schedic's nice configuration file and reporting features.

CI pipeline

This project uses a simple GitLab CI pipeline to ensure code quality (linting and unit testing) and that the code is buildable. Deployment is a manual step I perform from my workstation.

The pipeline can be simply executed locally just by launching ci/local.sh from within the project directory. A temporary Docker container is created to execute all steps (yarn install, yarn lint, yarn cover). A few tests depend on the restic binary, which is downloaded (and cached) automatically from GitHub.

Publishing a new release

# add entry to CHANGELOG.md and
# add all files to be commited along with release tag
git add CHANGELOG.md README.md

# set new version (commits package.json and all added files)
yarn version [--patch]

# publish to npm registry
yarn publish

License

Copyright (c) 2019 Jörn Reder

MIT License

See LICENSE file in this directory.

TODO / IDEAS

• ping log messages for long running commands

• configuration

  • new key host.repos
  • overrides backup-defaults.repos if present

• put some repo stats at top of mail report

• commands

  • schedic install -- install restic on target hosts
  • schedic snapshots -- list snapshots for selected backups
  • schedic setup -- simple wizard to setup a basic schedic.yml