VMSnap

VMSnap is a production-ready Node.js tool designed to simplify the process of creating and managing snapshots of KVM domains. Backups generated by VMSnap are incremental, if possible. VMSnap is also capable of performing intelligent backup rotation with configurable retention policies.

This README provides an overview of the project, installation instructions, usage guidelines, and contribution information.

Features

• 🔍 Query KVM domains for comprehensive backup status with integrity checks
• 💾 Create incremental snapshots of virtual machines with checkpoint management
• 🧹 Delete unnecessary bitmaps and checkpoints with selective or bulk cleanup
• 📅 Intelligent backup rotation with configurable periods (monthly, quarterly, bi-annual, yearly)
• 🛡️ Enterprise-grade reliability with 92% test coverage and comprehensive error handling
• 📊 Multiple output formats including JSON, YAML, and human-readable reports
• ⚡ Performance optimized with concurrent operations and efficient resource usage

Requirements

You must have the following on your host OS:

• virsh
• qemu-img
• virtnbdbackup

Getting these installed is out of scope for this doc.

The app will let you know if you are missing any required programs when you start running commands with it.

Installation

To install VMSnap, follow these steps:

npm install -g vmsnap

Local

You may also choose to install VMSnap by checking the code out and running it locally. To run localy, do the following:

1. Clone the repository:

   git clone [email protected]:slackdaystudio/vmsnap.git

2. Navigate to the project directory:

   cd vmsnap

3. Install the required dependencies:

   npm install

Usage

This usage guide assumes you have installed VMSnap via the npm install -g vmsnap command. Doing so will install VMSnap which includes a vmsnap bin.

Tip: You may execute the same commands from a local checkout by swapping out the name of the bin for npm run vmsnap --. For example, to run a status check from a local version you first go to your code checkout and then run npm run vmsnap -- --domains=vm1,vm2 --status

Command Line Switches

The following CLI switches are available when invoking VMSnap.

| Switch | Status | Backup | Scrub | Type | Examples/Notes | |----------------|--------|--------|--------|---------|------------------------------------------------------------------------------| | domains | ✅ | ✅ | ✅ | string | "vm1" or "vm1,vm2,etc" or "" | | status | ✅ | - | - | boolean | Querys the domain(s) | | backup | - | ✅ | - | boolean | Does an incremental backup (if possible) | | scrub | - | - | ✅ | boolean | Cleans checkpoints and bitmaps off of the domain | | output | ✅ | ✅ | - | string | A full path to a directory where backups are placed | | verbose | ✅ | - | - | boolean | Prints out extra information when running a status check | | machine | ✅ | - | - | boolean | Removes some output from the status command | | json | ✅ | - | - | boolean | Outputs the status command is JSON | | yaml | ✅ | - | - | boolean | Output YAML from the status command (aliased to --yml) | | raw | - | ✅ | - | boolean | Enables raw disk handling | | groupBy | ✅ | ✅ | - | string | Defines how backups are grouped on disk (month, quarter, bi-annual or year) | | prune | - | ✅ | - | boolean | Rotates backups by deleting last periods backup | | pretty | ✅ | - | - | boolean | Pretty prints disk sizes (42.6 GB, 120 GB, etc) | | checkpointName | - | - | ✅ | string | The name of the checkpoint to delete (no effect when scrubType=*) | | scrubType | - | - | ✅ | string | The type of item to scrub (checkpoint, bitmap, both, or * for ALL) |

*This happens on or after the the middle of the current period (15 days monthly, 45 days quarterly, 90 days bi-annually or 180 yearly)

Status

The default action for VMSnap is to display a status report for VMs supplied.

vmsnap --domains=vm1 --status

Tip: The --domains flag also accepts a comma seperated list of domains. You may also pass in "*" to select all found VMs. This is applicable to backing up, scrubbing, or querying VMs.

Tip: The --status flag may be omited. Leaving it in is useful when constructing backup and scrub commands because you may test the command by querying the status of the domain. If that query works you then swap the --status flag for the --backup or --scrub flag, as appropriate.

This could return the following information if ran, as an example.

Status for vm1:
  Overall status: OK
  Checkpoints found for vm1:
    virtnbdbackup.0
    virtnbdbackup.1
    virtnbdbackup.2
  Eligible disks found for vm1:
    vda
      Virtual size: 107374182400
      Actual size: 14286573568
      Bitmaps found for vda:
          virtnbdbackup.0
          virtnbdbackup.1
          virtnbdbackup.2

Tip: Pass in an output=/PATH/TO/BACKUP_ROOT flag to see statistics about the backups already saved to disk. VMSnap will perform additional integrity checks using the information it collects.

Machine parsable output is possible with the --json and --yaml flags in combination with the --machine flag.

For example, running the following command...

vmsnap --domains=vm1 --machine --json

..will produce something like the following.

{"vm1":{"checkpoints":["virtnbdbackup.0","virtnbdbackup.1","virtnbdbackup.2"],"disks":[{"disk":"vda","virtualSize":107374182400,"actualSize":14293934080,"bitmaps":["virtnbdbackup.0","virtnbdbackup.1","virtnbdbackup.2"]}],"overallStatus":0}}

Backup

Backups are always incremental unless VMSnap is cutting a new periods first backup. Subsequent backups will be incremental meaning only the changes from the VM will be captured.

Create a snapshot for vm1 and output it to the tmp direcory:

vmsnap --domains=vm1 --output=/tmp --backup

The above command will create a the backup for the domain. This creates a checkpoint and dirty bitmap on the VM file and deposits the backup to the /tmp directory.

Tip: Make sure you can read and write to the target directory in --output

You may also specify the --groupBy flag to tell VMSnap how to group your files on disk. Look at the table below for more information.

| groupBy Flag | Middle Mark | Sample Folder Name | |--------------|-------------|-----------------------------------| | month | 15d | vmsnap-backup-monthly-2024-11 | | quarter | 45d | vmsnap-backup-quarterly-2024-Q4 | | bi-annual | 90d | vmsnap-backup-bi-annually-2024-p2 | | year | 180d | vmsnap-backup-yearly-2024 |

Tip: If you do not set the groupBy flag the default period is assumed to be "month."

Backup Pruning (Caution)

Note: Pruning is destructive. Be careful when using it and check your backups frequently!

Pruning backups may be done by setting --prune on the backup command. This flag will automatically delete last periods backup once the middle of the current backup period comes up.

Pruning provides a sliding window for the given period of +/-50% depending upon where you are in the backup cycle. For example, setting the groupBy flag to "month" would mean you would have 2-6 weeks of backups on hand at any given time.

Raw Disk Handling

You can turn on raw disk handling by setting the --raw flag.

Scrubbing

Note: These commands are inherently destructive, be careful!

It is occasionally useful to be able to scrub one or more checkpoints or bitmaps from your domain. Doing so is fairly straight forward with VMSnap but please do be cautious.

Use this command to scrub a single bitmap from your backup disks. Keep in mind that bitmaps are stored on a per disk basis. VMSnap will scrub each disk of the bitmap if it find it.

vmsnap --domains=vm1 --scrub --scrubType=bitmap --checkpointName=virtnbdbackup.17

To scrub a domain of ALL checkpoints and bitmaps

vmsnap --domains=vm1 --scrub --scrubType=*

Testing & Quality Assurance

VMSnap maintains enterprise-grade quality standards with comprehensive testing:

Test Coverage

• 188 unit tests across all modules
• 50 integration tests against real KVM virtual machines
• 92.07% line coverage (target: 90%+)
• 85.71% function coverage (target: 95%+)
• Complete CI/CD integration with GitHub Actions

Running Tests

# Run all tests
npm test

# Run tests with coverage report
npm run test:coverage

# Run tests in watch mode during development
npm run test:watch

# Run only unit tests
npm run test:unit

Integration Tests

VMSnap includes a comprehensive integration test suite that tests real backup operations against actual KVM virtual machines. These tests require a KVM-enabled environment.

Prerequisites for Integration Tests

• KVM/QEMU with hardware virtualization support
• libvirt daemon running (libvirtd)
• virtnbdbackup installed and configured
• qemu-img and virsh command-line tools
• User must have permissions to create/manage VMs (typically libvirt and kvm groups)

Running Integration Tests Locally

# Verify KVM is available
ls -la /dev/kvm

# Verify libvirt connection
virsh version

# Run integration tests
npm run test:integration

# Optional: Setup test VMs manually
npm run test:integration:setup

# Optional: Cleanup test environment
npm run test:integration:cleanup

Integration Test Categories

| Test Suite | Tests | Description | |------------|-------|-------------| | backup-operations.test.js | 6 | Single/multiple VM backups, wildcards, selective domains | | incremental-backup.test.js | 7 | Full and incremental backup chains, disk change handling | | rotation-pruning.test.js | 7 | Monthly/quarterly/bi-annual/yearly grouping and pruning | | scrubbing-operations.test.js | 6 | Checkpoint and bitmap cleanup (handles offline VMs) | | status-commands.test.js | 11 | Status output in text/JSON/YAML formats, multiple VMs | | error-scenarios.test.js | 13 | Error handling, invalid inputs, concurrent execution |

Running on a Self-Hosted GitHub Actions Runner

Integration tests can run in CI using a self-hosted runner with KVM support:

1. Set up a self-hosted runner with nested virtualization enabled:

   # On the runner host, verify KVM support
   egrep -c '(vmx|svm)' /proc/cpuinfo  # Should return > 0
   
   # Install required packages (Ubuntu/Debian)
   sudo apt-get install -y qemu-kvm libvirt-daemon-system libvirt-clients virtinst qemu-utils
   
   # Install virtnbdbackup
   pip3 install virtnbdbackup
   
   # Add runner user to required groups
   sudo usermod -aG kvm,libvirt $USER
   
   # Start libvirt
   sudo systemctl enable --now libvirtd

2. Configure the runner following GitHub's self-hosted runner documentation

3. Update the workflow to use your self-hosted runner:

   # In .github/workflows/integration-tests.yml
   jobs:
     integration-tests:
       runs-on: self-hosted  # Change from ubuntu-latest

Why Integration Tests Skip on GitHub-Hosted Runners

Standard GitHub-hosted runners (ubuntu-latest) don't expose /dev/kvm because nested virtualization is disabled. The integration test workflow automatically detects this and:

• Skips integration tests with a warning
• Still runs all unit tests successfully

This ensures CI doesn't fail while allowing full integration testing on capable environments.

Code Quality

# Lint code
npm run lint

# Format code
npm run format

# Check formatting
npm run check-format

# Build project
npm run build

Security

VMSnap prioritizes security and maintains:

• 0 known vulnerabilities (regularly audited)
• Latest secure dependencies with automated updates
• Comprehensive error handling to prevent data corruption
• Input validation on all user-provided parameters
• Safe backup operations with integrity checks

Run security audit:

npm audit

Development

Prerequisites

• Node.js 18+ (specified in package.json engines)
• npm or yarn package manager

Setup Development Environment

1. Clone and setup:

   git clone [email protected]:slackdaystudio/vmsnap.git
   cd vmsnap
   npm install

2. Run in development mode:

   # Watch mode for automatic rebuilds
   npm run watch
      
   # Run locally without building
   npm run vmsnap -- --domains=vm1 --status

3. Before committing:

   npm run lint          # Check code style
   npm run test          # Run all tests
   npm run build         # Verify build works

Project Structure

vmsnap/
├── libs/                 # Core modules
│   ├── general.js       # Utility functions, dependency checking, error handling
│   ├── libnbdbackup.js  # Main backup orchestration with virtnbdbackup
│   ├── print.js         # Output formatting (text, JSON, YAML)
│   ├── qemu-img.js      # QEMU image operations & bitmap management
│   ├── serialization.js # Status collection & integrity analysis
│   └── virsh.js         # KVM domain & checkpoint management
├── test/
│   ├── unit/            # 188 unit tests across all modules
│   │   └── libs/        # Module-specific unit tests
│   └── integration/     # 50 integration tests with real KVM VMs
│       ├── helpers/     # VM lifecycle management & test assertions
│       │   ├── vm-manager.js      # Create/destroy test VMs
│       │   ├── test-assertions.js # Backup verification helpers
│       │   └── cleanup-helpers.js # Environment cleanup
│       ├── setup/       # Shell scripts for test environment
│       └── tests/       # 6 integration test suites
├── dist/                # Built output (generated)
└── vmsnap.js           # Main CLI entry point with argument parsing

Architecture

VMSnap follows a modular architecture with clear separation of concerns:

• CLI Layer: Argument parsing and user interaction (vmsnap.js)
• Service Layer: Core backup logic (libnbdbackup.js)
• Utility Layer: System integration (virsh.js, qemu-img.js)
• Data Layer: Status collection and serialization

Performance & Reliability

VMSnap is designed for production environments with emphasis on reliability:

Reliability Features

• Comprehensive error handling with specific exit codes
• Input validation prevents invalid operations
• Dependency checking ensures required tools are available
• Atomic operations with proper rollback on failures
• Lock file management prevents concurrent execution conflicts

Performance Optimizations

• Concurrent operations where safe (status checks, validation)
• Efficient resource usage with streaming and memory management
• Incremental backups minimize storage and time requirements
• Intelligent pruning maintains optimal backup retention
• Fast status reporting with optimized disk scanning

Monitoring & Logging

• Structured logging with configurable verbosity levels
• Progress reporting for long-running operations
• Exit codes for integration with automation tools
• JSON/YAML output for monitoring system integration

Contributing

We welcome contributions! VMSnap maintains high standards for code quality, testing, and security.

Contribution Guidelines

1. Fork and setup:

   git clone [email protected]:YOUR_USERNAME/vmsnap.git
   cd vmsnap
   npm install

2. Create a feature branch:

   git checkout -b feature/your-feature-name

3. Follow development standards:

   • Write tests for new functionality (maintain >90% coverage)
   • Follow existing code style (ESLint + Prettier)
   • Update documentation as needed
   • Ensure all checks pass:

     npm run lint          # Code style
     npm run test          # All tests pass
     npm run build         # Successful build
     npm audit            # No security issues

4. Commit your changes:

   git commit -m "feat: add new backup validation feature"

   Follow Conventional Commits format.

5. Push and create PR:

   git push origin feature/your-feature-name

   Then create a pull request with:

   • Clear description of changes
   • Test results and coverage impact
   • Screenshots/examples if applicable

Testing Requirements

All contributions must maintain our testing standards:

• Unit tests for new functions/modules
• Integration tests for complex workflows
• Edge case coverage for error scenarios
• Documentation updates for new features

Run the full test suite before submitting:

npm run test:coverage

License

This project is licensed under the MIT License. See the LICENSE file for details.

Release Notes

v1.1.0-beta (Latest)

🎉 Major Quality & Security Release

New Features:

• ✨ Comprehensive test suite with 188 unit tests and 50 integration tests
• 📊 92% test coverage across all modules
• 🛡️ Zero security vulnerabilities (fixed 7 issues)
• 🚀 Enhanced performance with edge case handling
• 📈 Improved error reporting and validation with proper exit codes

Integration Test Suite:

• 🧪 Full end-to-end testing against real KVM virtual machines
• 💾 Tests for backup operations (single VM, multiple VMs, wildcards)
• 🔄 Incremental backup chain validation
• 📅 Backup rotation testing (monthly, quarterly, bi-annual, yearly)
• 🧹 Checkpoint and bitmap scrubbing verification
• 📊 Status command output validation (text, JSON, YAML)
• ⚠️ Comprehensive error scenario coverage
• 🔌 Automatic libvirt connection handling (system/session)
• ⏸️ Graceful handling of offline VMs (copy mode vs checkpoints)

Security & Dependencies:

• 🔒 Updated all dependencies to latest secure versions
• 🛠️ Removed unnecessary React/Babel dependencies
• ⚡ Updated build tools (esbuild 0.24→0.27)
• 🔍 Added automated security auditing

Developer Experience:

• 🧪 Full test infrastructure with Vitest
• 📋 Enhanced documentation and architecture guides
• 🔧 Improved development workflow and standards
• 📝 Updated contribution guidelines with testing requirements

Bug Fixes:

• 🐛 Fixed error code propagation (errors now return proper exit codes)
• 🐛 Fixed virtnbdbackup command arguments
• 🐛 Improved empty domain list handling with clear error messages

Infrastructure:

• ✅ GitHub Actions CI/CD integration
• 📦 Automated testing and security checks
• 🏗️ Optimized build process and distribution
• 🔄 Sequential test execution for VM operations

Contact

For any questions or feedback, please open an issue on GitHub.

VMSnap v1.1.0-beta - Production-ready KVM backup solution with 238 tests (188 unit + 50 integration) and enterprise-grade reliability.