@pellegrims/nx-remotecache-s3

A task runner for @nrwl/nx that uses s3 storage as a remote cache. This enables all team members and CI servers to share a single cache. The concept and benefits of computation caching are explained in the NX documentation.

This package was built with nx-remotecache-custom 🙌

Setup

npm install --save-dev @pellegrims/nx-remotecache-s3

| Parameter | Description | Environment Variable / .env | nx.json | | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | ---------------- | | Endpoint | Optional. The fully qualified endpoint of the webservice. This is only required when using a custom (non-AWS) endpoint. | NXCACHE_S3_ENDPOINT | endpoint | | Bucket | Optional. Specify which bucket should be used for storing the cache. | NXCACHE_S3_BUCKET | bucket | | Prefix | Optional. Specify prefix path of target object key. | NXCACHE_S3_PREFIX | prefix | | Region | Optional. The AWS region to which this client will send requests. | NXCACHE_S3_REGION | region | | Profile | Optional. The AWS profile to use to authenticate. | NXCACHE_S3_PROFILE | profile | | Force Path Style | Optional. Whether to force path style URLs for S3 objects (e.g., https://s3.amazonaws.com/<bucket>/ instead of https://<bucket>.s3.amazonaws.com/ | NXCACHE_S3_FORCE_PATH_STYLE | forcePathStyle | | Read Only | Optional. Disable writing cache to the S3 bucket. This may be useful if you only want to write to the cache from a CI but not localhost. | NXCACHE_S3_READ_ONLY | readOnly | | Http Proxy | Optional. Useful when your environment must connect to a proxy to access the internet. | http_proxy || HTTP_PROXY | | | Https Proxy | Optional. Useful when your environment must connect to a proxy to access the internet. | https_proxy || HTTPS_PROXY | | | No Proxy | Optional. Useful when your environment must connect to a proxy to access the internet, but is not needed for connecting to aws. | no_proxy || NO_PROXY | |

{
  "tasksRunnerOptions": {
    "default": {
      "runner": "@pellegrims/nx-remotecache-s3",
      "options": {
        "cacheableOperations": ["build", "test", "lint", "e2e"],
        "endpoint": "https://some-endpoint.com",
        "bucket": "name-of-bucket",
        "prefix": "prefix/",
        "region": "us-west-000",
        "profile": "name-of-aws-profile",
        "forcePathStyle": true,
        "readOnly": false
      }
    }
  }
}

Authentication is handled by @aws-sdk/credential-provider-node, so credentials will be attempted to be found from the following sources (listed in order of precedence):

• Environment variables exposed via process.env (example: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
• SSO credentials from token cache
• Web identity token credentials
• Shared credentials and config ini files
• The EC2/ECS Instance Metadata Service

Run it 🚀

Running tasks should now show the storage or retrieval from the remote cache:

------------------------------------------------------------------------
Built Angular Package
 - from: /Users/name/example-workspace/libs/example-lib
 - to:   /Users/name/example-workspace/dist/libs/example-lib
------------------------------------------------------------------------
------------------------------------------------------------------------
Stored output to remote cache: s3 storage
Hash: d3d2bea71ea0f3004304c5cc88cf91be50b02bb636ebbdfcc927626fd8edf1ae
------------------------------------------------------------------------

Advanced Configuration

See nx-remotecache-custom.

Proxy Configuration

To use a proxy to connect to the S3 bucket, set any of the http_proxy https_proxy HTTP_PROXY or HTTPS_PROXY environment variables, with the lowercase version taking precedence. If needed, the proxy can be bypassed by adding the S3 endpoint to the no_proxy or NO_PROXY environment variables.

According to AWS, the endpoint and region parameters are meant to be mutually exclusive (though that isn't the case 100% of the time). Due to the complexity of creating a specific and valid s3 endpoint, and that there is no API provided by the aws-sdk libraries to generate one, this project handles checking the no_proxy for the s3 endpoint in the following manner.

If the s3 endpoint has been configured, it will be checked against the no_proxy and if found, s3 requests will bypass the proxy.

If the s3 endpoint has not been configured, it is assumed that the global s3 endpoint s3.amazonaws.com is being used, and the no_proxy will be checked for that url. Because of this, you could see the following behavior:

• If the s3 client does use the global s3 endpoint to make requests and if that url has been added to the no_proxy, then those requests will correctly bypass the proxy.
• If the s3 client does not use the global s3 endpoint (which could happen if a region other than us-east-1 is specified), and if that non global s3 endpoint has not been added to the no_proxy, then requests that should bypass the proxy will in fact NOT bypass the proxy. In this case, it is up to the user to determine which endpoint is being used and to add it to the no_proxy.

TIP: no_proxy is a global env variable. To limit the blast radius of no_proxy, include the specific service subdomain in the no_proxy entry; e.g. s3.amazonaws.com vs amazonaws.com.

More Custom Runners

See nx-remotecache-custom.