@cumulus/aws-client

v21.2.0

Published

a month ago

Utilities for working with AWS

0High
0Medium
0Low

@cumulus/aws-client

Utilities for working with AWS. These utilities can be used for interacting with live AWS services or Localstack. For ease of setup, testing, and credential management, code interacting with AWS services should use the helpers in this module.

⚠️ The documented API of this package will not change without a deprecation warning being provided in earlier releases. Code in this package that is not documented in this README may change without warning, and is not considered part of the package's public API.

Usage

npm install @cumulus/aws-client

Interacting with Localstack

To use these utilities with Localstack, make sure you have a running instance of Localstack and set this environment variable:

NODE_ENV=test

API

Modules

Classes

Functions

CloudFormation

CloudFormation
- ~describeCfStack(StackName) ⇒ Promise.<CloudFormation.Stack>
- ~describeCfStackResources(StackName) ⇒ Promise.<CloudFormation.StackResources>
- ~getCfStackParameterValues(stackName, parameterKeys) ⇒ Promise.<Object>

CloudFormation~describeCfStack(StackName) ⇒ Promise.<CloudFormation.Stack>

Describes a given CloudFormation stack

See CloudFormation.Stack

Kind: inner method of CloudFormation
Returns: Promise.<CloudFormation.Stack> - The resources belonging to the stack

| Param | Type | Description | | --- | --- | --- | | StackName | string | The name of the CloudFormation stack to query |

CloudFormation~describeCfStackResources(StackName) ⇒ Promise.<CloudFormation.StackResources>

Describes the resources belonging to a given CloudFormation stack

See CloudFormation.StackResources

Kind: inner method of CloudFormation
Returns: Promise.<CloudFormation.StackResources> - The resources belonging to the stack

| Param | Type | Description | | --- | --- | --- | | StackName | string | The name of the CloudFormation stack to query |

CloudFormation~getCfStackParameterValues(stackName, parameterKeys) ⇒ Promise.<Object>

Get parameter values for the given CloudFormation stack

Kind: inner method of CloudFormation
Returns: Promise.<Object> - Object keyed by parameter names

| Param | Type | Description | | --- | --- | --- | | stackName | string | The name of the CloudFormation stack to query | | parameterKeys | Array.<string> | Key names for the stack parameters that you want to return |

CloudwatchEvents

CloudwatchEvents~putEvent(name, schedule, state, [description], [role]) ⇒ Promise.<CloudWatchEvents.PutRuleResponse>

Create a CloudWatch Events rule

Kind: inner method of CloudwatchEvents

| Param | Type | Description | | --- | --- | --- | | name | string | the rule name | | schedule | string | a ScheduleExpression | | state | string | the state of the rule | | [description] | string | | | [role] | string | a Role ARN |

DynamoDb

DynamoDb
- static
  - .scan ⇒ Promise.<Object>
  - .createAndWaitForDynamoDbTable(params) ⇒ Promise.<Object>
  - .deleteAndWaitForDynamoDbTableNotExists(params) ⇒ Promise
- inner
  - ~get(params) ⇒ Promise.<Object>
  - ~parallelScan(params) ⇒ Promise

DynamoDb.scan ⇒ Promise.<Object>

Call DynamoDb client scan

See DocumentClient.scan() for descriptions of params and the return data.

Kind: static property of DynamoDb

| Param | Type | | --- | --- | | params | Object |

DynamoDb.createAndWaitForDynamoDbTable(params) ⇒ Promise.<Object>

Create a DynamoDB table and then wait for the table to exist

Kind: static method of DynamoDb
Returns: Promise.<Object> - the output of the createTable call

| Param | Type | Description | | --- | --- | --- | | params | Object | the same params that you would pass to AWS.createTable See https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb/classes/dynamodb.html#createtable |

DynamoDb.deleteAndWaitForDynamoDbTableNotExists(params) ⇒ Promise

Delete a DynamoDB table and then wait for the table to not exist

Kind: static method of DynamoDb

| Param | Type | Description | | --- | --- | --- | | params | Object | the same params that you would pass to AWS.deleteTable See https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-dynamodb/classes/dynamodb.html#deletetable |

DynamoDb~get(params) ⇒ Promise.<Object>

Call DynamoDb client get

See DocumentClient.get() for descriptions of params and the return data.

Kind: inner method of DynamoDb
Throws:

RecordDoesNotExist if a record cannot be found

| Param | Type | Description | | --- | --- | --- | | params | Object | | | params.tableName | string | Table name to read | | params.item | GetCommandInput.Key | Key identifying object to get | | params.client | DynamoDBDocument | Instance of a DynamoDb DocumentClient | | params.getParams | Object | Additional parameters for DocumentClient.get() |

DynamoDb~parallelScan(params) ⇒ Promise

Do a parallel scan of DynamoDB table using a document client.

See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html#Scan.ParallelScan. See DocumentClient.scan().

Kind: inner method of DynamoDb

| Param | Type | Description | | --- | --- | --- | | params | Object | | | params.totalSegments | number | Total number of segments to divide table into for parallel scanning | | params.scanParams | ScanInput | Params for the DynamoDB client scan operation See https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html | | params.processItemsFunc | function | Function used to process returned items by scan | | [params.dynamoDbClient] | DynamoDBDocument | Instance of Dynamo DB document client | | [params.retryOptions] | pRetry.Options | Retry options for scan operations |

Kinesis

Kinesis~describeStream(params, retryOptions) ⇒ Promise.<Object>

Describe a Kinesis stream.

Kind: inner method of Kinesis
Returns: Promise.<Object> - The stream description response

| Param | Type | Description | | --- | --- | --- | | params | Object | | | params.StreamName | string | A Kinesis stream name | | retryOptions | Object | Options passed to p-retry module |

Lambda

Lambda
- ~isEventBridgeEvent()
- ~invoke()

Lambda~isEventBridgeEvent()

Bare check for EventBridge shape

Kind: inner method of Lambda

Lambda~invoke()

Invoke a Lambda function

Kind: inner method of Lambda

S3

S3
- static
  - ~~.getS3Object~~
  - .recursivelyDeleteS3Bucket ⇒
  - .listS3ObjectsV2(params) ⇒ Promise.<Array>
  - .listS3ObjectsV2Batch(params)
- inner
  - ~s3Join(...args) ⇒ string
  - ~parseS3Uri(uri) ⇒ Object
  - ~buildS3Uri(bucket, key) ⇒ string
  - ~s3TagSetToQueryString(tagset) ⇒ string
  - ~deleteS3Object(bucket, key) ⇒ Promise
  - ~headObject(Bucket, Key, retryOptions) ⇒
  - ~s3ObjectExists(params) ⇒ Promise.<boolean>
  - ~waitForObjectToExist(params) ⇒
  - ~s3PutObject(params)
  - ~putFile()
  - ~s3CopyObject()
  - ~promiseS3Upload()
  - ~streamS3Upload()
  - ~getObjectReadStream()
  - ~downloadS3File()
  - ~getObjectSize()
  - ~s3GetObjectTagging()
  - ~s3PutObjectTagging()
  - ~getObject()
  - ~waitForObject()
  - ~getObjectStreamContents()
  - ~getTextObject()
  - ~getJsonS3Object()
  - ~fileExists()
  - ~deleteS3Files(s3Objs)
  - ~uploadS3FileStream(fileStream, bucket, key, s3opts) ⇒ Promise
  - ~listS3Objects()
  - ~deleteS3Buckets(buckets) ⇒ Promise
  - ~calculateObjectHash(params)
  - ~validateS3ObjectChecksum(params) ⇒ Promise.<boolean>
  - ~getFileBucketAndKey(pathParams) ⇒ Array.<string>
  - ~createBucket(Bucket) ⇒ Promise
  - ~createS3Buckets(buckets) ⇒ Promise
  - ~multipartCopyObject(params) ⇒ Promise.<{etag: string}>
  - ~copyObject()
  - ~moveObject() ⇒ Promise.<undefined>

S3.getS3Object

Deprecated

Gets an object from S3.

Kind: static property of S3

S3.recursivelyDeleteS3Bucket ⇒

Delete a bucket and all of its objects from S3

Kind: static property of S3
Returns: the promised result of S3.deleteBucket

| Param | Description | | --- | --- | | bucket | name of the bucket |

S3.listS3ObjectsV2(params) ⇒ Promise.<Array>

Fetch complete list of S3 objects

listObjectsV2 is limited to 1,000 results per call. This function continues listing objects until there are no more to be fetched.

The passed params must be compatible with the listObjectsV2 call.

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjectsV2-property

Kind: static method of S3
Returns: Promise.<Array> - resolves to an array of objects corresponding to the Contents property of the listObjectsV2 response

| Param | Type | Description | | --- | --- | --- | | params | Object | params for the s3.listObjectsV2 call |

S3.listS3ObjectsV2Batch(params)

Fetch lazy list of S3 objects

listObjectsV2 is limited to 1,000 results per call. This function continues listing objects until there are no more to be fetched.

The passed params must be compatible with the listObjectsV2 call.

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#listObjectsV2-property

Kind: static method of S3

| Param | Description | | --- | --- | | params | params for the s3.listObjectsV2 call |

S3~s3Join(...args) ⇒ string

Join strings into an S3 key without a leading slash

Kind: inner method of S3
Returns: string - the full S3 key

| Param | Type | Description | | --- | --- | --- | | ...args | string | Array.<string> | the strings to join |

S3~parseS3Uri(uri) ⇒ Object

parse an s3 uri to get the bucket and key

Kind: inner method of S3
Returns: Object - Returns an object with Bucket and Key properties

| Param | Type | Description | | --- | --- | --- | | uri | string | must be a uri with the s3:// protocol |

S3~buildS3Uri(bucket, key) ⇒ string

Given a bucket and key, return an S3 URI

Kind: inner method of S3
Returns: string - an S3 URI

| Param | Type | Description | | --- | --- | --- | | bucket | string | an S3 bucket name | | key | string | an S3 key |

S3~s3TagSetToQueryString(tagset) ⇒ string

Convert S3 TagSet Object to query string e.g. [{ Key: 'tag', Value: 'value }] to 'tag=value'

Kind: inner method of S3
Returns: string - tags query string

| Param | Type | Description | | --- | --- | --- | | tagset | Array.<Object> | S3 TagSet array |

S3~deleteS3Object(bucket, key) ⇒ Promise

Delete an object from S3

Kind: inner method of S3
Returns: Promise - promise of the object being deleted

| Param | Type | Description | | --- | --- | --- | | bucket | string | bucket where the object exists | | key | string | key of the object to be deleted |

S3~headObject(Bucket, Key, retryOptions) ⇒

Get an object header from S3

Kind: inner method of S3
Returns: returns response from S3.headObject as a promise

| Param | Description | | --- | --- | | Bucket | name of bucket | | Key | key for object (filepath + filename) | | retryOptions | options to control retry behavior when an object does not exist. See https://github.com/tim-kos/node-retry#retryoperationoptions By default, retries will not be performed |

S3~s3ObjectExists(params) ⇒ Promise.<boolean>

Test if an object exists in S3

Kind: inner method of S3
Returns: Promise.<boolean> - a Promise that will resolve to a boolean indicating if the object exists

| Param | Description | | --- | --- | | params | same params as https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#headObject-property |

S3~waitForObjectToExist(params) ⇒

Asynchronously waits for an S3 object to exist at a specified location.

This function uses p-wait-for to repeatedly check for the object's existence until it's found or a timeout is reached. It provides configurable interval between checks and a total timeout duration.

Kind: inner method of S3
Returns: A Promise that resolves when the S3 object is found.
Throws:

TimeoutError If the object does not exist within the specified timeout period.
Error If an unexpected error occurs during the S3 existence check.

| Param | Description | | --- | --- | | params | The parameters for waiting for the object. | | params.bucket | The name of the S3 bucket where the object is expected. | | params.key | The key (path) of the S3 object within the bucket. | | params.interval | The time in milliseconds to wait between checks. Defaults to 1000ms (1 second). | | params.timeout | The maximum time in milliseconds to wait before giving up and throwing a TimeoutError. Defaults to 30000ms (30 seconds). |

S3~s3PutObject(params)

Put an object on S3

Kind: inner method of S3

| Param | Description | | --- | --- | | params | same params as https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObject-property |

S3~putFile()

Upload a file to S3

Kind: inner method of S3

S3~s3CopyObject()

Copy an object from one location on S3 to another

Kind: inner method of S3

S3~promiseS3Upload()

Upload data to S3

see https://github.com/aws/aws-sdk-js-v3/tree/main/lib/lib-storage

Kind: inner method of S3

S3~streamS3Upload()

Upload data to S3 using a stream

Kind: inner method of S3

S3~getObjectReadStream()

Get a readable stream for an S3 object

Kind: inner method of S3

S3~downloadS3File()

Downloads the given s3Obj to the given filename in a streaming manner

Kind: inner method of S3

S3~getObjectSize()

Get the size of an S3 object

Kind: inner method of S3

S3~s3GetObjectTagging()

Get object Tagging from S3

Kind: inner method of S3

S3~s3PutObjectTagging()

Puts object Tagging in S3 https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#putObjectTagging-property

Kind: inner method of S3

S3~getObject()

Gets an object from S3.

Kind: inner method of S3

S3~waitForObject()

Get an object from S3, waiting for it to exist and, if specified, have the correct ETag.

Kind: inner method of S3

S3~getObjectStreamContents()

Transform streaming response from S3 object to text content

Kind: inner method of S3

S3~getTextObject()

Fetch the contents of an S3 object

Kind: inner method of S3

S3~getJsonS3Object()

Fetch JSON stored in an S3 object

Kind: inner method of S3

S3~fileExists()

Check if a file exists in an S3 object

Kind: inner method of S3

S3~deleteS3Files(s3Objs)

Delete files from S3

Kind: inner method of S3

| Param | Description | | --- | --- | | s3Objs | An array of objects containing keys 'Bucket' and 'Key' |

S3~uploadS3FileStream(fileStream, bucket, key, s3opts) ⇒ Promise

Upload the file associated with the given stream to an S3 bucket

Kind: inner method of S3
Returns: Promise - A promise

| Param | Type | Description | | --- | --- | --- | | fileStream | ReadableStream | The stream for the file's contents | | bucket | string | The S3 bucket to which the file is to be uploaded | | key | string | The key to the file in the bucket | | s3opts | Object | Options to pass to the AWS sdk call (defaults to {}) |

S3~listS3Objects()

List the objects in an S3 bucket

Kind: inner method of S3

S3~deleteS3Buckets(buckets) ⇒ Promise

Delete a list of buckets and all of their objects from S3

Kind: inner method of S3
Returns: Promise - the promised result of S3.deleteBucket

| Param | Type | Description | | --- | --- | --- | | buckets | Array | list of bucket names |

S3~calculateObjectHash(params)

Calculate the cryptographic hash of an S3 object

Kind: inner method of S3

| Param | Type | Description | | --- | --- | --- | | params | Object | | | params.s3 | S3 | an S3 instance | | params.algorithm | string | cksum, or an algorithm listed in openssl list -digest-algorithms | | params.bucket | string | | | params.key | string | |

S3~validateS3ObjectChecksum(params) ⇒ Promise.<boolean>

Validate S3 object checksum against expected sum

Kind: inner method of S3
Returns: Promise.<boolean> - returns true for success
Throws:

InvalidChecksum - Throws error if validation fails

| Param | Type | Description | | --- | --- | --- | | params | Object | params | | params.algorithm | string | checksum algorithm | | params.bucket | string | S3 bucket | | params.key | string | S3 key | | params.expectedSum | number | string | expected checksum | | [params.options] | Object | crypto.createHash options |

S3~getFileBucketAndKey(pathParams) ⇒ Array.<string>

Extract the S3 bucket and key from the URL path parameters

Kind: inner method of S3
Returns: Array.<string> - [Bucket, Key]

| Param | Type | Description | | --- | --- | --- | | pathParams | string | path parameters from the URL bucket/key in the form of |

S3~createBucket(Bucket) ⇒ Promise

Create an S3 bucket

Kind: inner method of S3

| Param | Type | Description | | --- | --- | --- | | Bucket | string | the name of the S3 bucket to create |

S3~createS3Buckets(buckets) ⇒ Promise

Create multiple S3 buckets

Kind: inner method of S3

| Param | Type | Description | | --- | --- | --- | | buckets | Array.<string> | the names of the S3 buckets to create |

S3~multipartCopyObject(params) ⇒ Promise.<{etag: string}>

Copy an S3 object to another location in S3 using a multipart copy

Kind: inner method of S3
Returns: Promise.<{etag: string}> - object containing the ETag of the destination object

note: this method may error if used with zero byte files. see CUMULUS-2557 and https://github.com/nasa/cumulus/pull/2117.

| Param | Type | Default | Description | | --- | --- | --- | --- | | params | Object | | | | params.sourceBucket | string | | | | params.sourceKey | string | | | | params.destinationBucket | string | | | | params.destinationKey | string | | | | [params.sourceObject] | S3.HeadObjectOutput | | Output from https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#headObject-property | | [params.ACL] | string | | an S3 Canned ACL | | [params.copyTags] | boolean | false | | | [params.chunkSize] | number | | chunk size of the S3 multipart uploads |

S3~copyObject()

Copy an S3 object to another location in S3

Kind: inner method of S3

| Param | Default | Description | | --- | --- | --- | | params.ACL | | an S3 Canned ACL | | params.copyTags | false | | | params.chunkSize | | chunk size of the S3 multipart uploads |

S3~moveObject() ⇒ Promise.<undefined>

Move an S3 object to another location in S3

Kind: inner method of S3

| Param | Type | Default | Description | | --- | --- | --- | --- | | [params.ACL] | string | | an S3 Canned ACL | | [params.copyTags] | boolean | false | | | [params.chunkSize] | number | | chunk size of the S3 multipart uploads |

SecretsManager

SNS

SNS
- ~publishSnsMessageWithRetry(snsTopicArn, message, retryOptions) ⇒ Promise.<undefined>
- ~createSnsTopic(snsTopicName) ⇒

SNS~publishSnsMessageWithRetry(snsTopicArn, message, retryOptions) ⇒ Promise.<undefined>

Publish a message to an SNS topic. Does not catch errors, to allow more specific handling by the caller.

Kind: inner method of SNS

| Param | Description | | --- | --- | | snsTopicArn | SNS topic ARN | | message | Message object | | retryOptions | options to control retry behavior when publishing a message fails. See https://github.com/tim-kos/node-retry#retryoperationoptions |

SNS~createSnsTopic(snsTopicName) ⇒

Create an SNS topic with a given name.

Kind: inner method of SNS
Returns: - ARN of the created SNS topic

| Param | Description | | --- | --- | | snsTopicName | Name of the SNS topic |

SQS

SQS~createQueue()

Create an SQS Queue. Properly handles localstack queue URLs

Kind: inner method of SQS

SQS~sendSQSMessage()

Send a message to AWS SQS

Kind: inner method of SQS

SQS~receiveSQSMessages()

Receives SQS messages from a given queue. The number of messages received can be set and the timeout is also adjustable.

Kind: inner method of SQS

SQS~isSQSRecordLike()

Bare check for SQS message Shape

Kind: inner method of SQS

SQS~parseSQSMessageBody()

Extract SQS message body

Kind: inner method of SQS

SQS~deleteSQSMessage()

Delete a given SQS message from a given queue.

Kind: inner method of SQS

SQS~sqsQueueExists()

Test if an SQS queue exists

Kind: inner method of SQS

StepFunctions

StepFunctions
- static
  - .describeExecution(params) ⇒ Promise.<DescribeExecutionOutput>
  - .describeStateMachine(params) ⇒ Promise.<DescribeStateMachineOutput>
  - .getExecutionHistory(params) ⇒ Promise.<GetExecutionHistoryOutput>
  - .listExecutions(params) ⇒ Promise.<Object>
- inner
  - ~executionExists(executionArn) ⇒ Promise.<boolean>