@xen-orchestra/immutable-backups

Install

Installation of the npm package:

npm install --save @xen-orchestra/immutable-backups

Usage

Immutability

The goal is to make a remote that XO can write to, but not modify during the immutability duration set per remote. That way, it is not possible for XO to delete or encrypt any backup during this period. It protects your backups against ransomware, at least as long as the attacker does not have root access to the remote server.

We target governance type of immutability: the local root account of the remote server will be able to lift immutability.

We use Linux filesystem capabilities (chattr +i). They are tested on process start.

It is compatible with encryption at rest made by XO.

Prerequisites

• The service must run as root, or as a user with the CAP_LINUX_IMMUTABLE capability.
• chattr and lsattr must be installed on the system.
• The underlying filesystem must support the immutable attribute (ext2/ext3/ext4, btrfs, etc.).

Configuring

This package uses app-conf to load its configuration. The application name is xo-immutable-backups. A sample config file is provided in this package.

The default config path is /etc/xo-immutable-backups/config.toml. Example:

liftEvery = "1h"

[remotes.remote1]
root = "/mnt/ssd/vhdblock/"
immutabilityDuration = "7d"

Parameters

• liftEvery (required): How often the service checks for files whose immutability duration has expired and lifts their protection (e.g. 1h).
• Per remote:
  • root (required): Absolute path to the root of the backup repository.
  • immutabilityDuration (required, minimum 1d): How long files remain protected after the backup datetime encoded in their filename.
  • delayBetweenSizeCheck (optional, default 100ms): Polling interval used when waiting for a file write to complete before locking it.

CLI commands

• xo-immutable-remote: Start the watching daemon. Must be kept running to protect new backups reliably. It lifts expired immutability on startup, then every liftEvery.

How protection works

On startup, xo-immutable-remote:

1. Verifies the filesystem supports immutability by creating, locking, and deleting a temporary test file.
2. Writes an immutability.json file to the remote root (containing since, duration, and immutable: true) and locks it with chattr +i.
3. Sets up fs.watch listeners on the backup directory tree (see below).

Protection is applied to a backup only after the file write is complete. The service polls the file size until it stabilises before locking. This means "real-time" here means "as soon as possible after each backup file is fully written".

VM backups (xo-vm-backups/<vmUUID>/)

The watcher triggers on <YYYYMMDD>T<HHmmss>[Z].json files appearing in a VM directory. The .json metadata file is always written last by XO, so its stable size guarantees all other files for that run are already fully written.

Once stable, the following paths are locked together with a single chattr +i -R call:

<vmDir>/<datetime>.json
<vmDir>/<datetime>.xva
<vmDir>/<datetime>.xva.checksum
<vmDir>/vdis/<jobId>/<vdiId>/<datetime>.vhd        (flat VHD or alias)
<vmDir>/vdis/<jobId>/<vdiId>/<datetime>.alias.vhd
<vmDir>/vdis/<jobId>/<vdiId>/data/<datetime>.vhd   (VHD directory — locked recursively)

Missing optional paths (e.g. .xva for delta backups, .alias.vhd for full backups) are silently skipped by chattr.

Config backups (xo-config-backups/<scheduleId>/<datetime>/)

Pool metadata backups (xo-pool-metadata-backups/<scheduleId>/<poolUUID>/<datetime>/)

The watcher triggers on metadata.json appearing in the date directory. Once its write is complete, the entire date directory is locked recursively with chattr +i -R, covering data, data.json, metadata.json, and any other files inside.

How lifting works

On each liftEvery tick (and once immediately on startup), the service walks the backup directory tree directly:

• It parses the datetime encoded in each backup's filename or directory name.
• It compares that datetime against the threshold now - immutabilityDuration.
• Files older than the threshold have their immutability lifted with chattr -i -R.

The expiry reference is the datetime in the filename, not the file's mtime. XO periodically rewrites metadata .json files (cache refresh, reconciliation), which would otherwise reset mtime and defer expiry indefinitely.

On the first lift run after startup, all backup files are scanned unconditionally (full scan). This catches orphaned immutable files left by a previous partial or interrupted lock. Subsequent runs use a fast-path: only backups whose .json sentinel is currently immutable are processed.

The first run also walks the disk directories (xo-vm-backups/<vmUUID>/vdis/<jobId>/<vdiId>/) directly and lifts any expired disk found there, still using the datetime in its own filename. Both the locking and the regular lifting name a backup's disks from its <datetime>.json, so a disk stops being reachable that way once XO's retention has deleted that json — or once a merge has renamed the disk, since the surviving data/<datetime>.vhd then carries the datetime of the older backup its blocks came from. Such a disk would otherwise stay immutable forever, and an immutable disk prevents XO from ever merging or deleting the backups of that VDI.

Troubleshooting

Some files are still immutable after the duration expired

Restart xo-immutable-remote. The first lift run after startup always performs a full scan, which will find and release any expired files that were missed.

Make a remote fully mutable again

1. Stop the xo-immutable-remote service.
2. As root on the file server, run:

   chattr -i -R /path/to/remote/on/fileserver/

Make one VM temporarily mutable again

If one or a few VM had an issue and need manual cleanup, you can manually lift the immutability for one VM.

1. Stop the xo-immutable-remote service.
2. As root on the file server, run:

   chattr -i -R /path/to/remote/on/fileserver/xo-vm-backups/<vm uuid>

3. Run the backup job, it will fix what is repairable and purge the non recoverable backups
4. As root on the file server, run:

   chattr +i /path/to/remote/on/fileserver/xo-vm-backups/<vm uuid>/*.json
   chattr +i -R /path/to/remote/on/fileserver/xo-vm-backups/<vm uuid>/vdis/*/*/*.vhd
   chattr +i -R /path/to/remote/on/fileserver/xo-vm-backups/<vm uuid>/vdis/*/*/data/*.vhd

   Lock the disks, not the directories that hold them: chattr +i -R …/vdis/ also locks vdis, the job and VDI directories and data, and an immutable directory rejects the files of the next backup, so every later run of that VM would fail. Depending on whether the remote uses VHD directories, one of the two commands above may report that it matched nothing.

Note that this VM will be mutable between step 2 and 4, it's up to you to document and test the backup after.

Increasing the immutability duration

Change the setting. The next lift cycle will use the new duration; files already past the old duration but within the new one will remain protected.

Reducing the immutability duration

Change the setting, then restart xo-immutable-remote to apply immediately, or wait for the next scheduled liftEvery cycle.

Why are my incremental backups not marked as protected in XO?

For incremental backups to be marked as protected in XO, the entire backup chain must be within the immutability window. To guarantee at least 7 days of protected backups, set both the immutability duration and the retention to 14 days, with a full backup interval of 7 days.

This ensures that when the last chain is complete, it and the previous chain are both fully under protection:

Kd Kdddddd Kdddddd K #  8 backups protected, 2 chains
K Kdddddd Kdddddd Kd #  9 backups protected, 2 chains
 Kdddddd Kdddddd Kdd # 10 backups protected, 2 chains
 Kddddd Kdddddd Kddd # 11 backups protected, 2 chains
 Kdddd Kdddddd Kdddd # 12 backups protected, 2 chains
 Kddd Kdddddd Kddddd # 13 backups protected, 2 chains
 Kdd Kdddddd Kdddddd #  7 backups protected, 1 chain (previous full is now mutable)
Kd Kdddddd Kdddddd K #  8 backups protected, 2 chains

(K = full backup, d = delta)

Why doesn't the service start?

• It must run as root or as a user with the CAP_LINUX_IMMUTABLE capability.
• The underlying filesystem must support the immutable attribute (chattr/lsattr must work).
• Check logs with journalctl.

Contributions

Contributions are very welcomed, either on the documentation or on the code.

You may:

• report any issue you've encountered;
• fork and create a pull request.

License

AGPL-3.0-or-later © Vates SAS