stors
v1.0.3
Published
File upload and management service with support for thumbnailing, S3 storage, and JWT-based user authentication
Downloads
2
Readme
STORS
File upload and management service with support for thumbnailing, S3 storage, and JWT-based user authentication
Stors is a straightforward ExpressJS app for receiving and storing user uploads. There are many modules like this
available (e.g. multer-s3
) but STORS was designed to address a few use cases that some applications have that are
not provided by other solutions:
Upload requests can be validated by checking for a JWT auth token. The JWT can be issued by another server (e.g. the back-end API the client application is talking to). This is much simpler and easier to use than S3 Signed Requests or other auth techniques.
Metadata such as token contents, UUIDs, and date elements may be used to format clean, final URLs for uploaded assets.
S3 is currently supported as a storage backend, but the storage layer is modular. New destinations may be easily added by implementing a simple module interface (
init()
andstore()
).After upload requests have been completed, a Webhook may be called with the information about the upload and the authentication block if enabled. This can allow back-ends to update database references to files without having to "trust" the user. The back-end could also kick of validation or post-processing routines.
Installation and Configuration
Installation is simple. Just clone this repository into a suitable working directory, then copy config.json.sample
to config.json
and customize to suit your needs. Run the service via npm start
, although a NodeJS daemon manager
such as ForeverJS
or PM2
is recommended.
Basic Options
{
"port": 3000,
"uploadUri": "/upload",
...
}
port: The port to listen on. May be overriden with the PORT
environment variable, e.g. PORT=80 npm start
.
Defaults to 3000.
uploadUri: The path to listen on for uploads, e.g. "/upload".
Notifications
The notifications block is optional and may be omitted to disable this feature. If included, each upload request will trigger a POST request to the specified url after the files have been processed (and before the final response is made to the client).
Requests are not retried - if high reliability is required for this step, it is recommended that you use a reliable, scalable service for making this hook, such as an AWS Lambda + Dead Letter Queue endpoint. Failures to post to the notification endpoint DO trigger errors to the client requesting the upload, so a (less ideal, but simpler) alternative may be to offer the end-user a "retry" option.
Note that one request is triggered per upload request, not per uploaded file. If a request contains multiple files, they will be included together in a single notification.
{
...
"notifications": {
"url": "https://requestb.in/1as1q1s1",
"fields": {
"secretKey": "1234"
}
},
...
}
url: The URL to trigger a POST request to for each upload.
fields: Optional collection of additional fields to merge into the POST request. May be useful for including routing data and/or a secret key to authenticate the notification.
The notification will include the following fields:
{
source: 'stors',
event: 'upload',
session: {
token: /* Optional. Included if JWT authentication is enabled. Will contain the decoded token fields. */ }
},
uploads: [{
fieldName: 'thumbnail',
destinationPath: 'path/to/file.jpg',
mimetype: 'image/jpeg',
size: 6236,
url: 'https://mybucket.s3.amazonaws.com/path/to/file.jpg'
}, {... additional files ...}],
secretKey: '1234'
}
Storage
The storage system was designed to be modular, although today only an S3 engine is provided. To create a new storage
engine simply copy storage-s3.js
to a new file, adjust the logic (there are only two methods), and load it in
storage.js
.
{
...
"storage": {
"destinationPath": "{date.y}/{date.m}/{token.userId}/{uuid}{ext}",
...
},
...
}
destinationPath: Tokenized string indicating where the file should be stored in the back-end. Relative to the root of the configured destination. Tokens must be placed in curly brackets. Some standard token replacements are available:
{date.\[y/m/d/h/i/s/ms]}
- Date fields (year, month, day, hour, minute, second, millisecond). Note: dates are generated at the time the upload request is processed, not per file. Do not use dates as unique identifiers!{token.\[fieldname]}
- Field from an authentication token. Note: currently these are not validated and if not present, will be silently ignored! It is up to the application developer to provide valid tokens!{uuid}
- A UUIDv4 unique value for each file.{ext}
- The original extension for the file.
S3 Engine
{
...
"storage": {
"s3": {
"region": "us-east-1",
"params": {
"Bucket": "my-bucket"
}
}
},
...
}
region: S3 region the bucket is in.
params.Bucket: Bucket name to upload to.
These parameters are passed to the AWS.S3 SDK during configuration when the application is started. This means you can include any additional parameters as required. For instance, an ideal configuration would use EC2 Instance Roles to not require authentication here. However, if this is not possible (server is hosted elsewhere, or to support local dev work) you can easily include an access key and secret here:
{
...
"storage": {
"s3": {
"accessKeyId": "ACCESSKEY",
"secretAccessKey": "SECRETKEY",
"region": "us-east-1",
"params": {
"Bucket": "my-bucket"
}
}
},
...
}
For a full list of configuration options available, see Constructing a S3 object
Form Processing
Form processing is handled by Formidable. For a full list of options please see that project's documentation, but reasonable defaults might be:
{
...
"formidable": {
"encoding": "utf-8",
"keepExtensions": true,
"multiples": true
},
...
}
Authentication
User access to the service may be validated using JWT. Auth settings are passed directly to the JWT library - for a full list of options, please see that project's documentation.
{
...
"auth": {
"secret": "CHANGEME",
"algorithm": "HS256"
},
...
}
This is a very simple yet powerful mechanism because it can address two use-cases:
If you are already assigning JWTs to your users as part of other authentication mechanisms, you can reuse the same token here. Just make sure the client sends the token with each upload request, and configure the secret to match your other servers. As a bonus, any data (e.g.
userId
) that you encode into the token is made available through the request handling (and Notification postback) in Stors. This makes it easy to match up files in your back-end to the users uploading them.If you prefer a more secure approach, you can have your back-end pre-flight upload requests. Configure your application to make an API call to your back-end before an upload is transmitted. The back-end can validate the request and issue an application-specific JWT with a very short expiration (e.g. 60 seconds). For additional security you can even generate unique field names to make sure a token cannot be used more than once.