npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

aws-s3-multipart-copy

v1.1.0

Published

This node js package provides a simple usage for multipart copying between buckets in aws-s3 via the official aws-sdk. It manages the 3 steps required for multipart copy including byte-range offsetting and parts order.

Downloads

6,590

Readme

NPM Version Build Status Test Coverage NPM Downloads NSP Status

aws-s3-multipart-copy

Wraps aws-sdk with a multipart-copy manager, in order to provide an easy way to copy large objects from one bucket to another in aws-s3. The module manages the copy parts order and bytes range according to the size of the object and the desired copy part size. It speeds up the multipart copying process by sending multiple copy-part requests simultaneously.

** The package supports aws-sdk version '2006-03-01' and above.

** The package supports node 8 version and above.

NPM

Installing

npm install aws-s3-multipart-copy

init

aws-s3-multipart-copy is based on the aws-sdk and therefore requires an initialized AWS.S3 instance.

Also, it requires a logger instance which supports 'info' and 'error' level of logging (meaning logger.info and logger.error are functions).

Example

let bunyan = require('bunyan'),
    AWS = require('aws-sdk'),
    s3Module = require('aws-s3-multipart-copy');

let logger = bunyan.createLogger({
        name: 'copy-object-multipart',
        level: 'info',
        version: 1.0.0,
        logType: 'copy-object-multipart-log',
        serializers: { err: bunyan.stdSerializers.err }
    });

let s3 = new AWS.S3();

s3Module.init(s3, logger);

copyObjectMultipart

After module is initialized, the copyObjectMultipart functionality is ready for usage. copyObjectMultipart returns a promise and can only copy (and not upload) objects from bucket to bucket.

** Objects size for multipart copy must be at least 5MB.

The method receives two parameters: options and request_context

Request parameters

  • options: Object (mandatory) - keys inside this object must be as specified below
    • source_bucket: String (mandatory) - The bucket that holds the object you wish to copy
    • object_key: String (mandatory) - The full path (including the name of the object) to the object you wish to copy
    • destination_bucket: String (mandatory) - The bucket that you wish to copy to
    • copied_object_name: String (mandatory) - The full path (including the name of the object) for the copied object in the destination bucket
    • object_size: Integer (mandatory) - A number indicating the size of the object you wish to copy in bytes
    • copy_part_size_bytes: Integer (optional) - A number indicating the size of each copy part in the process, if not passed it will be set to a default of 50MB. This value must be between 5MB and 5GB - 5MB. ** if object size does not divide exactly with the part size desired, last part will be smaller or larger (depending on remainder size)
    • copied_object_permissions: String (optional) - The permissions to be given for the copied object as specified in aws s3 ACL docs, if not passed it will be set to a default of 'private'
    • expiration_period: Integer/Date (optional) - A number (milliseconds) or Date indicating the time the copied object will remain in the destination before it will be deleted, if not passed there will be no expiration period for the object
    • content_type: String (optional) A standard MIME type describing the format of the object data
    • metadata: Object (optional) - A map of metadata to store with the object in S3
    • cache_control: String (optional) - Specifies caching behavior along the request/reply chain
    • storage_class: String (optional) - Specifies the storage class for the copied object. The valid values are specified in the aws s3 docs. When unset, the class will be 'STANDARD'
  • request_context: String (optional) - this parameter will be logged in every log message, if not passed it will remain undefined.

Response

  • A successful result might hold any of the following keys as specified in aws s3 completeMultipartUpload docs

    • Location — (String)
    • Bucket — (String)
    • Key — (String)
    • Expiration — (String) If the object expiration is configured, this will contain the expiration date (expiry-date) and rule ID (rule-id). The value of rule-id is URL encoded.
    • ETag — (String) Entity tag of the object.
    • ServerSideEncryption — (String) The Server-side encryption algorithm used when storing this object in S3 (e.g., AES256, aws:kms). Possible values include:
      • "AES256"
      • "aws:kms"
    • VersionId — (String) Version of the object.
    • SSEKMSKeyId — (String) If present, specifies the ID of the AWS Key Management Service (KMS) master encryption key that was used for the object.
    • RequestCharged — (String) If present, indicates that the requester was successfully charged for the request. Possible values include:
      • "requester"
  • In case multipart copy fails, three scenarios are possible:

    • The copy will be aborted and copy parts will be deleted from s3 - copyObjectMultipart will reject
    • The abort procedure passed but the copy parts were not deleted from s3 - copyObjectMultipart will reject
    • The abort procedure fails and the copy parts will remain in s3 - copyObjectMultipart will reject

Example

Positive

let request_context = 'request_context';
let options = {
        source_bucket: 'source_bucket',
        object_key: 'object_key',
        destination_bucket: 'destination_bucket',
        copied_object_name: 'someLogicFolder/copied_object_name',
        object_size: 70000000,
        copy_part_size_bytes: 50000000,
        copied_object_permissions: 'bucket-owner-full-control',
        expiration_period: 100000,
        storage_class: 'STANDARD'
    };

    return s3Module.copyObjectMultipart(options, request_context)
        .then((result) => {
            console.log(result);    
        })
        .catch((err) => {
            // handle error
        })

        /* Response:
            result = {
                Bucket: "acexamplebucket", 
                ETag: "\"4d9031c7644d8081c2829f4ea23c55f7-2\"", 
                Expiration: 100000,
                Key: "bigobject", 
                Location: "https://examplebucket.s3.amazonaws.com/bigobject"
            }
        */

Negative 1 - abort action passed but copy parts were not removed

let request_context = 'request_context';
let options = {
        source_bucket: 'source_bucket',
        object_key: 'object_key',
        destination_bucket: 'destination_bucket',
        copied_object_name: 'someLogicFolder/copied_object_name',
        object_size: 70000000,
        copy_part_size_bytes: 50000000,
        copied_object_permissions: 'bucket-owner-full-control',
        expiration_period: 100000,
        storage_class: 'STANDARD'
    };

    return s3Module.copyObjectMultipart(options, request_context)
        .then((result) => {
            // handle result 
        })
        .catch((err) => {
            console.log(err);
        })

        /*
            err = {
                message: 'Abort procedure passed but copy parts were not removed'
                details: {
                    Parts: ['part 1', 'part 2']
                    }
                }
        */

Negative 2 - abort action succeded

let request_context = 'request_context';
let options = {
        source_bucket: 'source_bucket',
        object_key: 'object_key',
        destination_bucket: 'destination_bucket',
        copied_object_name: 'someLogicFolder/copied_object_name',
        object_size: 70000000,
        copy_part_size_bytes: 50000000,
        copied_object_permissions: 'bucket-owner-full-control',
        expiration_period: 100000,
        storage_class: 'STANDARD'
    };

    return s3Module.copyObjectMultipart(options, request_context)
        .then((result) => {
            // handle result 
        })
        .catch((err) => {
            console.log(err);
        })

        /*
            err = {
                    message: 'multipart copy aborted',
                    details: {
                        Bucket: destination_bucket,
                        Key: copied_object_name,
                        UploadId: upload_id
                    }
                }
        */