@koopjs/provider-s3-select
v1.1.4
Published
Koop provider for accessing S3 data with S3 Select
Downloads
1
Keywords
Readme
@koopjs/provider-s3-select
Koop provider for fetching data stored S3 with S3 Select. Files can be JSON, JSON lines, or CSVs. Note that CSV source files currently result in features with empty geometries.
Usage
Install
npm install @koopjs/provider-s3-select
Register
// initialize Koop
const Koop = require('koop')
const koop = new Koop()
// register
const s3SelectProvider = require('@koopjs/provider-s3-select')
koop.register(s3SelectProvider)
Configuration
The S3 Select provider uses the config module to store configuration settings in an object scoped to the koopProviderS3Select
key. The following JSON provides an example configuration:
{
"koopProviderS3Select": {
"s3Path": "my-default-bucket", // default path
"ttl": 300, // The TTL option is measured in seconds, it will be used to set the cache expiration entry
"stores": {
"my-store-key": {
"s3Path": "my-other-bucket/a-folder", // specific path for this store
"serialization": {
"JSON": {
"Type": "LINES"
}
}
}
}
}
}
| key | type | description | required |
| -- | -- | -- | -- |
| s3Path
| String | Default path-prefix that will be prepended to all requested files. Begins with the S3 bucket name and may include any sub-directory path. | No |
| ttl
| Integer | Number of seconds to cache the data | No |
| stores
| Object |key-value lookup used by the provider to fetch specific configuration information for a request. Each key in stores
defines a valid value of the :host
route parameter | Yes |
| stores[<key>].s3Path
| String | Path-prefix that will be prepended to all file requests when :host
is equal to <key>
| No |
| stores[<key>].serialization
| Object | S3 input serialization object. Informs S3 how files requested with a give :host
value should be deserialized. See S3 documentation.| No |
As noted above, s3Path
is always optional, but if set,it should begin with the S3 bucket name and include any sub-directory path along which you want to restrict requests for fetching files. If s3Path
is not set, the full S3 path to the file, including the bucket name, needs to be included in the request's :id
parameter.
AWS Credentials
In order to use the provider (and its S3 sdk), you will need to set your AWS credentials as environment variables. See the AWS docs for details.
Request Parameters
Once the provider is registered with Koop, it will be available for all output services and will include the :host
and :id
route parameters.
| parameter | type | desription |
| -- | -- | -- |
| :host
| String | Points the provider to an entry in the koopProviderS3Select.stores
config object. |
| :id
| String | The S3 path to a S3 file, with adjustments for any path prefix set in configuration with s3Path
. Any forward slashes in the path should be replaced with ::
to avoid routing errors. |
The Koop output service provides an example of the usage of these parameters. The pattern for the query service is:
/s3/:host/:id/FeatureServer/:layer/:method
An example request using this pattern is below:
/s3/json-lines/path::to::json::file.json/FeatureServer/0/query
In the above request, json-lines
is the :host
parameter and refers to a key in the configurations stores
object. path::to::json::file.json
is the :id
parameter and is a partial path to an S3 file with all /
replaced with ::
. Using the s3Path
from the json-lines
entry of the configuration object, the noted request would fetch the file with the following path /my-other-bucket/a-folder/path/to/json/file.json
.