@modbox/s3-uploads-server

Note: This is the server-side package for the s3-upload module, see @modbox/s3-upload-client for the client package.

s3-upload approach

This lib abstracts away the complexity of presigned requests uploads to an S3-compatible object storage service.
Based on two (backend and frontend) simple configurations that define onse or several upload types, for each of which your can :

• Choose between simple or multipart upload
• Whitelist a set of allowed mimetypes and sizes
• Allow or forbid file replacement
• Define a naming strategy and key conflict handler
• Define a post-upload renaming strategy (e.g. your file should be ${postId}/${filename} and the postId is not generated until the upload is successful and the resource created in database)

Written in typescript with DX in mind.

Server-side usage

Setup and config

utils/upload.ts

import path from 'path'
import {
  setupUploadModule,
  FileToUpload,
  UploadMode,
  MimeTypesConfig,
  UploadedFile,
} from '@modbox/s3-uploads-server'

// Define your upload types, you can use an enum
export enum UploadType {
  Avatar = 'avatar',
  CommunityPost = 'community-post',
  ResourceFiles = 'resource-files',
}

const MULTIPART_CHUNK_SIZE = 10485760 // 10Mb = 10485760 bytes
const IMAGE_TYPE_CONDITION = {
  // Regex for all images
  types: [/^image\//],
  // Max size of 15MiB in bytes
  maxSize: 15728640,
}
const TEXT_TYPE_CONDITION = {
  // Strings for strict equality checks
  types: ['text/plain', 'another_mimetype', /some_regex/i],
  // Max size of 5MiB in bytes
  maxSize: 5242880,
}
const PDF_TYPE_CONDITION = {
  types: ['application/pdf'],
  // Max size of 50MiB in bytes
  maxSize: 52428800,
}

export const uploadModule = setupUploadModule({
  s3Config: {
    credentials: {
      accessKeyId: '<Your access key>',
      secretAccessKey: '<Your secret key>',
    },
    // Some S3-compatible object storage provider
    domain: '<Your cloud provider domain>',
    region: '<Your region>',
  },
  // Setting the multipart chunk size (defaults to 5MiB)
  multipartChunkSize: MULTIPART_CHUNK_SIZE,
  // Allowing this type for all uploads
  mimeTypesConfig: [IMAGE_TYPE_CONDITION],
  // Here define a condition where deletion is allowed to avoid data loss for example
  canDelete: env.NODE_ENV === 'production',
  uploads: {
    [UploadType.Avatar]: {
      bucket: env.SCW_AVATAR_BUCKET,
      mode: UploadMode.Single,
      buildKey: (
        file: FileToUpload,
        ctx: { userId: number; username: string },
      ) => {
        return `${ctx.userId}-${ctx.username}${path.extname(file.filename)}`
      },
    },
    [UploadType.CommunityPost]: {
      bucket: env.SCW_COMMUNITY_THREAD_BUCKET,
      mode: UploadMode.Multipart,
      // This allows you to build the key where the file will be stored based on current upload
      buildKey: (file: FileToUpload) => {
        return `tmp-${file.filename}`
      },
      /* When buildFinalKey is provided, you file once uploaded will be moved at this new key.
      This allows for example to create a resource in database only when upload is a success and then
      move the file around based on the new data (a generated id for example) */
      buildFinalKey: (
        file: UploadedFile,
        ctx: { postId: number; discussionId: number },
      ) => {
        return `${ctx.discussionId}/${ctx.postId}/${file.filename}`
      },
      /* This tells the lib how to rename the uploaded file when key is already taken
      If not provided and such case happens, the lib will throw (except if allowReplace is set to true) */
      onKeyConflict: (key: string, tryCount: number) => {
        const ext = path.extname(key)
        return `${key.substring(0, key.length - ext.length)}-${tryCount}${ext}`
      },
      // This adds PDF for this kind of uploads
      mimeTypesConfig: [PDF_TYPE_CONDITION],
    },
    [UploadType.ResourceFiles]: {
      // Upload the object even if it overrides an existing file
      allowReplace: true,
      bucket: env.SCW_RESOURCE_FILE_BUCKET,
      mode: UploadMode.Multipart,
      buildKey: (file: FileToUpload) => {
        return `tmp-${file.filename}`
      },
      buildFinalKey: (file: UploadedFile, ctx: { resourceId: number }) => {
        return `${ctx.resourceId}/${file.filename}`
      },
      // This adds TEXT files for this kind of uploads
      mimeTypesConfig: [TEXT_TYPE_CONDITION],
    },
  },
})

/* You can re-export some utils that are already setup with your provider credentials to use elsewhere in your app */
export const {
  buildBucketHostname,
  buildObjectUrl,
  deleteObject,
  doesObjectExist,
  downloadObject,
  getCleanFilename,
  moveObject,
} = uploadModule.utils

routes/users.ts

import express from 'express'
import { uploadModule, UploadTypes } from 'utils/upload.ts'

const app = express()
app.get('/avatar-upload-request', async (req, res) => {
  // Get user and file to upload from request
  const fileToUpload = req.body
  const user = req.user

  // Initiate upload
  const initiatedUpload = await uploadModule[UploadType.Avatar].initiateOne(
    fileToUpload,
    { userId: user.id, username: user.username },
  )
  res.json(initiatedUpload)
})

app.post('/users', async (req, res) => {
  // Get user and file to upload from request
  const uploadedFile = req.body
  const user = req.user

  // Complete upload
  const s3Object = await uploadModule[UploadType.Avatar].completeOne(
    uploadedFile,
  )
  const updatedUser = updateUserInDatabase({ ...user, avatar: s3Object })
  res.json(updatedUser)
})

app.listen(3000)

Configuration object

This is the definition of the config object passed to setupUploadModule.

| Property | Type | Required | Default | Description | | ------------------------- | --------------------------------------------------------------------------------------------------- | -------- | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | s3Config | { region: string; domain: string; credentials: { accessKeyId: string; secretAccessKey: string } } | true | | Pass you provider credentials and domain to setup the library (For some use cases, this accept the same config as aws-sdk, typechecking will help you) | | canDelete | boolean \| ((file: S3Location) => boolean \| Promise<boolean>) | false | true | This prevent object deletion to happen if passed false or a function that returns false | | multipartChunkSize | number | false | 5242880 (5MiB) | This number represent the chunk size for multipart uploads in bytes. | | mimeTypesConfig | Array<{ types: (string \| RegExp) \| (string \| RegExp)[]; maxSize: number }> | false | | Defines the list of allowed mime types and their corresponding max size. If not provided, all is accepted. | | uploads | See below for full breakdown of this object | true | | This object defines the different upload types as keys and their specific config as value. | | uploads.allowReplace | boolean | false | false | Allow or disallow object replacement (i.e. when you're uploading an object with the same key as an existing one). The lib will throw if false and uploads.onKeyConflict is not provided. | | uploads.bucket | string | true | | The bucket where these uploads should be stored | | uploads.buildKey | (fileToUpload: FileToUpload, ctx: any) => string | true | | Handler to build the object key based on file to upload | | uploads.buildFinalKey | (uploadedFile: UploadedFile, ctx: any) => string | false | | Handler that allow to rename the object once uploaded, useful in cases where you need some extra data to build the final key like an id not generated yet at the time of the initial upload. | | uploads.mimeTypesConfig | Same as root mimeTypesConfig | false | | Same as root mimeTypesConfig but this only applies to the upload type it's declared for (root mimeTypesConfig is applied to all specific upload types, no need to repeat it.) | | uploads.mode | UploadMode (enum exported by the lib) | true | | The upload to use for this upload type (can be UploadMode.Multipart \| UploadMode.Single) | | uploads.onKeyConflict | (key: string, tryCount: number, ctx: any) => string \| Promise<string> | false | | Handler for cases where an object already exist at the specified key. The lib will throw if not provided and uploads.allowReplace is false. |

Notes

This lib is still in development and will be considered production-ready when it reach version 1.0.0. Use at your own risk. Breaking changes might occur before 1.0.0, so you might want to use strict references in your package.json dependencies. After 1.0.0, semver will be rigorously respected.