@itentialopensource/s3-crud-operations

v2.0.1-2022.1.1

Published

8 months ago

Default artifact description

Downloads

0High
0Medium
0Low

Itential IAP artifacts

S3 CRUD Operations

S3 CRUD Operations

Overview

The S3 CRUD Operations pre-built consists of an Operations Manager automation to showcase the basic operations that can be performed on AWS S3. This was created to automate some of the manual tasks that were performed directly on the AWS S3 using the tasks exposed by the Itential open source adapter: adapter-aws_s3. A description for each operation is listed below:

Create: Create a new object and insert it into the desired bucket in AWS S3.
Read: Perform a search operation on the bucket to display the list of objects.
Update: Search a desired object from a bucket and perform an update operation on the result.
Delete: Search a desired object from a bucket and perform a delete operation on the result.

The S3_CRUD_Workflow serves as an entry point for Operations Manager automation to run the use cases described above. The running path of the workflow depends on the operation selected by the user. The user is provided with a JSON form to input data.

Supported IAP Versions

Itential pre-builts are built and tested on particular versions of IAP. In addition, pre-builts that work with devices are often dependent on certain orchestration systems (e.g. NSO and IAG). As such, these pre-builts will have dependencies on these other systems. This version of the S3 CRUD Operations pre-built has been tested with:

IAP 2022.1.x

Getting Started

These instructions will help you get a copy of the pre-built in your IAP instance for testing in your environment. Reading this section is also helpful for deployments as it provides you with pertinent information on prerequisites and capabilities.

Prerequisites

Users must satisfy the following prerequisites to install and run this pre-built:

Itential Automation Platform
- ^2022.1.x

Capabilities

Each operation is designed as a stand-alone application (child workflows) which can be re-used in different workflows and applications.
Performs a create operation of an object in the bucket through JSON forms.
Performs a list objects operation on the bucket.
Performs an update operation by choosing to update an existing object or by adding a new object to the bucket.
Performs a delete operation of an object in the bucket through JSON form.
Select either verbose or non-verbose options in the input JSON Form for viewing the details.

How to Install

To install the Pre-Built:

Verify you are running a supported version of the Itential Automation Platform (IAP) as listed above in the Prerequisites section in order to install the Pre-Built.
The Pre-Built can be installed from within App-Admin_Essential. Simply search for the name of your desired Pre-Built and click the install button (as shown below).

Testing

While Itential tests this pre-built and its capabilities, it is often the case the customer environments offer their own unique circumstances. Therefore, it is our recommendation that you deploy this pre-built into a development/testing environment in which you can test the pre-built.

Using this Pre-Built

The main requirements to use this Pre-built are:

Running instance of the Itential OpenSource adapter-aws_s3, which can be found in adapter-aws_s3
Creating and setting up an AWS S3 account:
- Create an AWS S3 account
- Create a bucket. The bucket name must be globally unique.
- Go to AWS IAM and create a user group. For eg. "aws_admin". Create a User name in the user group.
- Select the user name created and go to Security Credentials.
- Create access key to setup IAP connection to the created S3 account.
- Use the access key in the AWS S3 adapter service config.
- sampleServiceConfig.json is available in /documentation to refer to for the adapter service config.

The pre-built can be run using Operations Manager automation S3 CRUD Automation. Use the following steps to complete the form:

Input the name of the S3 bucket that is needed to perform the CRUD operations. Use the name of the bucket created while setting up the S3 account.
Select the appropriate adapter-aws_s3 instance from the dropdown.
Input the name of the S3 object in the bucket to perform the CRUD operation on.
Select the required CRUD operation to be performed from the dropdown.
Select either verbose or non-verbose option for the performed operation details.

Note: All fields are mandatory except for the object name.

CRUD Operations Cases:

Create: If nothing is submitted in the object field in the initial JSON form when trying to create an object the pre-built will create an object called "default.json" with the file details as specified by the user in the object JSON form.
Read: The operation lists the objects in the bucket.
Update: If the object that was inputted to be updated does not exist in the bucket the pre-built will create an object of the specified name. If nothing is entered in the object field in the initial form(invalid entry) when trying to update an object the pre-built will create an object called "default.json" (if it already don't exist) with the file details as specified by the user in the object JSON form.
Delete: If a non existing object is submitted to be deleted or nothing is submitted in the initial form it is an invalid delete request and will throw an error.
Invalid Bucket: If a non existing bucket is submitted for an operation the pre-built will throw an error.

Input Schema

Example input formData:

{
    "formData":{
      "bucket": "s3-bucket-iap",
      "object": "default.json",
      "options": {
        "verbose": true
      },
      "adapterId": "Adapter-aws_s3",
      "selectACrudOperation": "Read"
    }
}

The following table details the property keys of the input object.

| key | type | required | description | |-------------------------------|---------|----------|--------------------| | formData.bucket | string | yes | bucket name | | formData.object | string | no | object name | | formData.adapterId | string | yes | adapter id | | formData.selectACrudOperation | string | yes | crud operation | | formData.options | object | yes | additional options | | formData.options.verbose | boolean | yes | verbose |

Output Schema

The ReturnStatus job variable returned from the run automation S3 CRUD Automation or the run childJob task S3_Read reports the success or failure of a CRUD operation performed on a S3 bucket.

Example output of successful read operation:

    "ReturnStatus": {
      "response": {
        "read_details": {
          "icode": "AD.200",
          "response": ""
          "headers": {
            "x-amz-id-2": "",
            "x-amz-request-id": "",
            "x-amz-bucket-region": "us-east-1",
            "content-type": "application/xml",
            ...
          },
          "metrics": {
            "code": 200,
            "timeouts": 0,
            ...
          }
        }
      },
      "status": "SUCCESS",
      "message": "Read operation successful",
      "errors": []
    }

The following table details the property keys of the output object.

| key | type | description | |------------------------------------|--------|---------------------------| | ReturnStatus | object | job variable | | ReturnStatus.response | object | response | | ReturnStatus.response.read_details | object | operation details | | ReturnStatus.status | string | status | | ReturnStatus.object | string | object name | | ReturnStatus.message | string | operation message | | ReturnStatus.errors | array | error details and message | | ReturnStatus.errors.error_details | object | error details | | ReturnStatus.errors.message | string | error message |

Success Example - Successful Create Operation

Example input for a successful create operation:

{
    "formData":{
      "bucket": "s3-bucket-iap",
      "object": "new_file.json",
      "options": {
        "verbose": true
      },
      "adapterId": "Adapter-aws_s3",
      "selectACrudOperation": "Create"
    }
}

Example output for successful create operation:

    "ReturnStatus": {
      "response": {
        "create_details": {
          "icode": "AD.200",
          "response": "",
          "headers": {
            "x-amz-id-2": "",
            "x-amz-request-id": "",
            "date": "Mon, 20 Mar 2023 03:11:50 GMT",
            "x-amz-server-side-encryption": "AES256",
            "etag": "",
            "server": "AmazonS3",
            "content-length": "0",
            "connection": "close"
          },
          "metrics": {
            "code": 200,
            "timeouts": 0,
            "redirects": 0,
            "retries": 0,
            "tripTime": 522,
            "isThrottling": false,
            "capabilityTime": "644ms"
          }
        }
      },
      "status": "SUCCESS",
      "object": "new_file.json",
      "message": "Object successfully created",
      "errors": []
    }

Failure Example 1 - Deleting an object that doesn't exist

This example has the key object value as file.json that does not exist in the bucket which causes the delete operation to throw an error.

Input:

{
    "formData":{
      "bucket": "s3-bucket-iap",
      "object": "file.json",
      "options": {
        "verbose": true
      },
      "adapterId": "Adapter-aws_s3",
      "selectACrudOperation": "Delete"
    }
}

Output:

    "ReturnStatus": {
      "response": {
        "delete_details": "The object does not exist in the bucket or, the input to the object field of the form is invalid. Retry using a valid object name. "
      },
      "status": "FAILED",
      "object": "null",
      "message": "Object delete unsuccessful",
      "errors": [
        {
          "error_details": {
            "icode": "AD.500",
            "IAPerror": {
              "origin": "Adapter-aws_s3-connectorRest-handleEndResponse",
              "displayString": "Error 404 received on request",
              "recommendation": "Verify the request is accurate via debug logs and postman",
              "code": 404,
              "raw_response": {
                "status": "success",
                "code": 404,
                "headers": {
                  ...
                }
              }
            },
            "metrics": {
              "code": 404,
              ...
            },
            "response": ""
          },
          "message": "Object delete unsuccessful"
        }
      ]
    }

Failure Example 2 - Running the delete operation without specifying the object name in the initial JSON form

This example is missing the key object which causes the delete operation to throw an error.

Input:

{
    "formData":{
      "bucket": "s3-bucket-iap",
      "options": {
        "verbose": false
      },
      "adapterId": "Adapter-aws_s3",
      "selectACrudOperation": "Delete"
    }
}

Output:

    "ReturnStatus": {
      "response": {
        "delete_details": "The object does not exist in the bucket or, the input to the object field of the form is invalid. Retry using a valid object name. "
      },
      "status": "FAILED",
      "object": "null",
      "message": "Object delete unsuccessful",
      "errors": [
        {
          "error_details": {
            "icode": "AD.300",
            "IAPerror": {
              "origin": "Adapter-aws_s3-adapter-headObjectSTSRole",
              "displayString": "key is required",
              "recommendation": "Please provide the required data"
            }
          },
          "message": "Object delete unsuccessful"
        }
      ]
    }

The following table details the property keys of the ReturnStatus object.

Additional Information

Please use your Itential Customer Success account if you need support when using this pre-built.

Helpful Links:

AWS S3: https://en.wikipedia.org/wiki/Amazon_S3
AWS S3 Adapter: https://gitlab.com/itentialopensource/adapters/cloud/adapter-aws_s3
AWS S3 Homepage: https://aws.amazon.com/s3/
AWS IAM: https://aws.amazon.com/iam/
AWS S3 API Calls: https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html