npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

ceph-sync

v0.11.1

Published

Sync tool between LOCAL file system and REMOTE object storage.

Downloads

18

Readme

ceph-sync

Tool to sync contents between local file system and remote object storage.

total downloads of ceph-sync ceph-sync's License latest version of ceph-sync

This tool can achieve synchronizations as:

  • directory A → container B
  • container B → directory A
  • container B → container C

Here, directory is located in local file system and made up of files and sub directories, while container (also called bucket according to AWS S3) is vessel in remote CEPH storage where objects saved.

Table of Contents

Links

Get Started

In CLI:

# Command "ceph-sync", "cehsync" and "ossync" will be generated.
npm install -g ceph-sync

# Show help info.
ceph-sync -h

# Run sync task.
ceph-sync --source /path/of/container --target /path/of/conn.json

As API:

const fs2ceph = require('ceph-sync/fs2ceph');

const progress = fs2ceph(
	/* source */ '/path/of/container',
	/* target */ connConfig );

progress.on('error', (err) => {
	// ...
});

progress.on('end', (meta) => {
	// Sychronization successfully finished.
});

Connection Config

The connection configuration is a JSON object required by the dependent package ceph. To describe an accessible (readable and writable) CEPH container, following properties are required:

  • endPoint
  • subuser
  • key
  • container

Here is a dummy example:

{
	"endPoint"   : "http://storage.example.com/",
	"subuser"    : "userName:subUserName",
	"key"        : "380289ba59473a368c593c1f1de6efb0380289ba5", 
	"container"  : "containerName"
}

For CLI usage, CEPH connection config should be stored in a JSON file.

CLI

When installed globally, ceph-sync will create a homonymous global command. Run ceph-sync -h in terminal to print the man page.

ceph-sync will occupy a hidden directory named .ceph-sync in home directory of current user.

API

ceph-sync offers three functions to achieve different tasks:

  • jinang/Progress ceph2ceph(object sourceConn, object targetConn, object options)
  • jinang/Progress ceph2fs(object sourceConn, string targetDir, object options)
  • jinang/Progress fs2ceph(string sourceDir, object targetConn, object options)
  1. Here "2" is a homophone of "to".
  2. sourceConn and targetConn may be an object containing CEPH storage connection configuration, or an instance of swift Connection.
  3. The functions accept similar options argument, see section Parameter options for details.
  4. The functions are all asynchronous and will return an instance of jinang/Progress. Via the returned value, we may learn about and control the sync progress. See section Get Into Sync Progress for details.

Each function may be required solely:

const cephSync  = require('ceph-sync');

const ceph2ceph = require('ceph-sync/ceph2ceph');
const ceph2fs   = require('ceph-sync/ceph2fs');
const fs2ceph   = require('ceph-sync/fs2ceph');

// E.g., next two functions are equivalent.
cephSync.ceph2ceph
ceph2ceph

Parameter options

  • string[] options.names
    Object names to be synchronised.

  • Function options.mapper
    Object name mapper.

  • Function options.filter
    Object name filter.

  • Function options.dualMetaFilter
    Filter with paramenter (stat, meta).
    Only acceptable in fs2ceph().

  • boolean options.ifNoneMatch
    Check etag firstly. If target object / file already exists and has same etag with source object / file, keep it instead of doing replacement.
    This option is only effective in ceph2ceph().

  • number options.maxCreated
    Maximum creation allowed (then the progress will be terminated).

  • number options.maxCreating
    Maximum cocurrent creating operation allowed.

  • number options.maxErrors
    Maximum exceptions allowed (then the progress will be terminated).

  • number options.retry
    Maximum retry times on exception for each object or object list.

Get Into Sync Progress

Via the returned instance of jinang/Progress, we may learn about what happened and then control the sync progress.

  • progress.on(string eventName, Function listener)
    See section Events During Sync Progress for aviable events and their accompanied arguments.

  • progress.abort()
    Terminate the progress as soon as possible.

  • progress.quit()
    Quit the progress gracefully.

Events During Sync Progress

Event: 'created'

  • Object meta

Event: 'moveon'

  • string mark

Event: 'ignored'

  • Object meta

Event: 'skipped'

  • Object meta

Event: 'warning'

  • Error error

Event: 'error'

  • Error error

Event: 'end'

  • Object meta
    {
    	errors /* number */,
    	created /* number */, 
    	ignored /* number */,
    }