@carimus/node-uploads
v1.1.0
Published
A library for accepting uploads, storing them in abstract disks, and referencing them in your database.
Downloads
19
Keywords
Readme
Node Uploads
A library for accepting uploads, storing them in abstract disks, and referencing them in your database.
Getting Started
Install the package in your project:
yarn add @carimus/node-uploads
Or if you're using npm
:
npm install --save @carimus/node-uploads
Usage
Basic Example
This example (written in TypeScript) shows how to use the Uploads
service with a traditional database model and a Koa
web server to upload uploads.
Note: the database model MyDBUploadModel
implementation isn't covered here but it can be thought of as a traditional database
model with async CRUD methods for interacting with the database.
./uploads.ts
import {
Uploads,
UploadRepository,
UploadedFile,
UploadMeta,
} from '@carimus/node-uploads';
import { DiskDriver } from '@carimus/node-disks';
import { MyDBUploadModel } from './database';
// We create the uploads service, specifying our database model as our `Upload` generic type.
export const uploads = new Uploads<MyDBUploadModel>({
// We specify a single disk `foo` using the Local driver and alias it as the default
disks: {
default: 'foo',
foo: {
driver: DiskDriver.Local,
root: '/tmp',
},
},
// We specify to store all new uploads in the `uploads` directory in the root of our disk (`/tmp` on the local FS in this case).
pathPrefix: 'uploads',
// We specify our repository using a simple object literal but you can also use an ES6 class instance here too.
repository: {
async create(
uploadedFile: UploadedFile,
meta?: UploadMeta,
): Promise<MyDBUploadModel> {
return MyDBUploadModel.create({
disk: uploadedFile.disk,
path: uploadedFile.path,
uploadedAs: uploadedFile.uploadedAs,
context: meta.context || '',
});
},
async update(
upload: MyDBUploadModel,
newUploadedFile: UploadedFile,
newMeta?: UploadMeta,
): Promise<MyDBUploadModel> {
await upload.update({
disk: newUploadedFile.disk,
path: newUploadedFile.path,
uploadedAs: newUploadedFile.uploadedAs,
context: newMeta.context || '',
});
return upload;
},
async delete(upload: MyDBUploadModel): Promise<void> {
await upload.delete();
},
async getUploadedFileInfo(
upload: MyDBUploadModel,
): Promise<UploadedFile> {
const { disk, path, uploadedAs } = upload;
return { disk, path, uploadedAs };
},
async getMeta(upload: MyDBUploadModel): Promise<UploadMeta> {
const { meta } = upload;
return { meta };
},
} as UploadRepository<MyDBUploadModel>,
});
./server.ts
import * as fs from 'fs';
import * as Koa from 'koa';
import * as Router from 'koa-router';
import * as koaBody from 'koa-body';
import { uploads } from './uploads';
import { MyDBUploadModel } from './database';
const app = new Koa();
const router = new Router();
router.get('/uploads', async (ctx) => {
ctx.body = await MyDBUploadModel.all();
});
router.get('/uploads/:uploadId', async (ctx) => {
const uploadId = parseInt(ctx.params.uploadId);
ctx.body = await MyDBUploadModel.find(uploadId);
});
router.delete('/uploads/:uploadId', async (ctx) => {
const uploadId = parseInt(ctx.params.uploadId);
await uploads.delete(uploadId);
ctx.body = { deleted: true };
});
router.post('/uploads', koaBody({ multipart: true }), async (ctx) => {
const { path, name } = ctx.request.files.file;
ctx.body = await uploads.upload(fs.createReadStream(path), name, {
context: '*',
});
});
router.put('/uploads/:uploadId', koaBody({ multipart: true }), async (ctx) => {
const uploadId = parseInt(ctx.params.uploadId);
const upload = await MyDBUploadModel.find(uploadId);
const { path, name } = ctx.request.files.file;
ctx.body = await uploads.update(upload, fs.createReadStream(path), name, {
context: '*',
});
});
app.use(router.routes());
app.use(router.allowedMethods());
app.listen(process.env.PORT || 3000, () => {
console.log(`Server running on port ${process.env.PORT || 3000}`);
});
API
Uploads
Class
Methods
constructor(config: UploadsConfig<Upload>)
: Instantiate the uploads service. See available config described below.upload(fileData: Buffer | Readable, uploadedAs: string, meta: UploadMeta | null = null, diskName?: string): Promise<Upload>
: Upload a file, persisting it to the Uploads service repository and returning the persisted Upload.fileData
: A buffer or a readable stream containing the uploaded file datauploadedAs
: The un-sanitized filename the upload as uploaded withmeta
: An optional object containing meta data about the upload to pass through to the repositorydiskName
: The disk to upload to, defaulting to thedefaultDisk
config option, which defaults to'default'
update(upload: Upload, fileData: Buffer | Readable, uploadedAs: string, meta: UploadMeta | null = null, diskName?: string): Promise<Upload>
: Update an existing upload with a new file, deleting its old file and returning the updated upload.upload
: The existing upload to update- See
upload(...)
above for a description of the remaining 4 arguments.
duplicate(original: Upload, meta: UploadMeta | null = null, newDiskName?: string, regeneratePath: boolean = true): Promise<Upload>
: Similar toupdate(...)
but duplicates an existing upload as if it were a new upload, creating a new upload with the repository and returning that upload.original
: The existing upload to duplicatemeta
: The metadata for the new upload, defaults to the existingoriginal
upload's metadata.newDiskName
: Seeupload(...)
'sdiskName
argument for a description of this argumentregeneratePath
: Regenerate the duplicated uploads path. Be careful when setting this to false as it will cause an error if you're duplicating the file on the same disk since the file will already exist there.
transfer(upload: Upload, newDiskName?: string, newMeta: UploadMeta | null = null, regeneratePath: boolean = false): Promise<Upload | false>
: Similar toduplicate(...)
but moves file instead of copying it and updates the existing upload instead of creating a new one in the repository. This is typically helpful when migrating uploads from one disk to another. If the file already exists on the target disk, this will not throw and instead just returnfalse
.upload
: The upload to transfernewDiskName
: Seeupload(...)
'sdiskName
argument for a description of this argumentnewMeta
: Seeduplicate(...)
'smeta
argument for a description of this argumentregeneratePath
: Seeduplicate(...)
for a description of this argument, noting that this defaults tofalse
instead meaining the default behaviour is to not regenerate the path and simply move the file with the same unmodified path to the new disk.
delete(upload: Upload, onlyFile: boolean = false): Promise<void>
: Delete an upload by deleting it from the disk and also from the repository (by default).upload
: The upload to delete.onlyFile
: Set totrue
to delete only the file from the disk and not from the repository.
read(upload: Upload): Promise<Buffer>
: Read an upload's file data into memory into aBuffer
upload
: The upload whose file should be read into memory and returned.
createReadStream(upload: Upload): Promise<Readable>
: Open a readable stream direct to the file on the disk.upload
: The upload whose file we should open up a readable stream to.
withTempFile(upload: Upload, execute: ((path: string) => Promise<void> | void) | null = null): Promise<string>
: Download or copy an upload's file to a temporary location on the local filesystem. Useful in rare cases where it's easier to work with a file on the disk as opposed to in memory (seeread(...)
andcreateReadStream(...)
).upload
: The upload whose file we should download to a temporary location.execute
: A callback to execute which will do something with the temporary file, whose path it's given as its sole argument. When the callback is done or resolves (if it's async), the file will be automatically deleted. Ifexecute
is not provided, this method will resolve to the string file path which the developer can use but must delete themselves. Cleanup can not be handled automatically if not using a callback, as such, the callback method is the recommended approach.
UploadsConfig
Interface
Fields
To instantiate a new Uploads
service, the following UploadsConfig
config values is required:
disks
: Either aDiskManager
instance or aDiskManagerConfig
object that aDiskManager
instance can be created from. See@carimus/node-disks
for more information.repository
: An object or instance that conforms to theUploadRepository
interface and is used to fetch and persist information about the uploads in whatever way is useful/necessary for the application. Read more below under "UploadRepository
Interface".
The following config values can also be provided but are optional:
defaultDisk
: The name of the disk in thedisks
to use as the default/active disk for storing uploads. Defaults to'default'
.sanitizeFilename
: A function that generates a disk-friendly filename from the client-provided filename that the upload was uploaded as. The sanitized version of the filename is what's persisted as theuploadedAs
and in the defaultgeneratePath
logic. The default logic replaces all characters that don't pass a certain whitelist of characters with__
(double underscores).generatePath
: A function that generates a storage path on the disk (not including or considering thepathPrefix
described below) based on the already sanitizeduploadedAs
client-provided filename. SeedefaultGeneratePath
in the source to understand the default implementation. IMPORTANT: This function should generate unique filenames in the same tick given the sameuploadedAs
value. The default implementation usesprocess.hrtime()
to do this.pathPrefix
: The root storage path for uploads on the selected storage disk. This prefix is included in the path stored in the database so this value can be changed and will only affect new uploads. If all of the existing uploads are moved into a subdirectory of the current storage disk, change the root of the storage disk instead. More complex scenarios will likely require creating a new disk to store old uploads vs. new uploads. This option is particularly useful to isolate / namespace your uploads from another developer's uploads if you're using the same remote s3 bucket for example.
UploadRepository
Interface
Methods
In order for the Uploads
service to be of any real use, you need to provide it with an object that conforms to the UploadRepository
interface which it can use to persist data about where an upload is stored and the metadata surrounding the upload.
See the documentation in the source for more details but in general, the methods that an UploadRepository
must
implement are:
create(uploadedFile: UploadedFile, meta?: UploadMeta): Promise<Upload> | Upload
update(upload: Upload, newUploadedFile: UploadedFile, newMeta?: UploadMeta): Promise<Upload> | Upload
delete(upload: Upload): Promise<void> | void
getUploadedFileInfo(upload: Upload): Promise<UploadedFile> | UploadedFile
getMeta(upload: Upload): Promise<UploadMeta> | UploadMeta
Upload
Generic Type
Upload
is a generic type argument on:
Upload
and can be anything used to uniquely identify the upload in your repository.
If you're using a database it's typically easiest and most performant to use the full record as the Upload
because all of the
operations that return an Upload
will return the full record as opposed to just an ID or something which would require another
database query to get the full info in order to i.e. return the data to the client in a server response. That's simply a recommendation
though; the developer can use whatever they like as an Upload
.
UploadedFile
Interface
Fields
disk
: The name of the disk the upload was stored onpath
: The path on the disk the upload was stored atuploadedAs
: The sanitized filename the upload was uploaded with
UploadMeta
Interface
An object with any or no string keys with any value. It's designed to store and pass thru whatever information is helpful to pass around
and through the uploads service pertaining to a specific upload. A string context
key is recommended for identifying the context in
which the upload was uploaded or is used (i.e. '*'
or 'user_profile'
).
TODO
- [ ] Document errors
Development
This project is based on the carimus-node-ts-package-template
. Check out the
README and docs there
for more up to date information on the development process and tools available.