express-fileupload-personalized

Personalized version of express-fileupload package

Changes in Fork

1). Clean all files if error occurs for any of files being uploaded

One thing important to keep in mind is that busboy parses request body sequentially, and as a result this library also sequentially parses files present in the request body

Original Behaviour

When multiple files are being uploaded either via single input field or via multiple input fields, default behaviour if error occurs for any of the files, is to stop parsing request furthermore, clean the file which gave the error, and call next(error) to propagate error to next error handler.

This behaviour is true for files which exceed configured size limits.

Problem Statement

There is a serious drawback of original approach when using with useTempFiles option. Let's consider our form has 6 file fields and if error occurs for 4th file being parsed, what will be the behaviour of original library? We have to acknowledge that at the time of error, 3 files have already been parsed and saved to tempFileDir. The behaviour will be to stop parsing request furthermore, clean the 4th file being parsed from tempFileDir, call next(error) to propagate error to next error handler which will most probably do some logging and end the response with details of error. It will not parse remaining 2 files and it will also not clean 3 files which have already been parsed.

One might say that rationale behind not cleaning already parsed files from tempFileDir is that library user may need to know which files the library was able to parse and save to tempFileDir without any errors so that he/she may somehow process those files or log those files. But this seems very illogical because we have already skipped remaining 2 files which would have alse been parsed without errors if we would have continued parsing request after encountring error in one file.

Fork Behaviour

If error occurs for any of the files being uploaded, stop parsing request furthermore, clean the file which gave the error, clean files which have already been parsed for current request, and call errorHandler if passed by the user or next(error) so that error may be handled appropriately by the user.

If user passed errorHandler, it is then responsibility of the user to end the response. If files are optional fotr next middleware, User may either decide to call next() to propagate request to next middleware in chain. User may call next(error) to propagate error to next errorHandlerMiddleware in chain. User may call res.send, res.end, or res.json to end the response.

2). Remove abortOnLimit and truncated support

In Original, if limit is defined in options, default behaviour is to stop receiving more bytes for that file and set file on req object with truncated flag set to true on file, and call the next middleware without any errors. If abortOnLimit is also set to true, it first calls limitHandler and then tries to close connection by calling res.end with 413 status code. There is no cleaning of already parsed file as desctibed above.

In Fork, if limit is defined, default behaviour is to stop receiving more bytes for that file, clean that file, clean files which have already been parsed for current request, and call the errorHandler with FileUploadLimitError if errorHandler is passed by user or next(FileUploadLimitError) so that error may be handled appropriately by the user.

We are doing this change because, desired behaviour in most of cases, is to abort and do not process truncated files. In original version, we could not call res.end in limitHandler if abortOnLimit is used as it will result in errors because library has already sent response in closeConnection method. If we do not use abortOnLimit, closeConnection is not called and library will continue to parse other files, if any, because req.unpipe(busboy) is called in closeConnection.

3). Make req.files an array in case of single file upload also

In Original, req.files is array of file objects only if user sends multiple files per key by setting multiple prop of html's input element set to true, otherwise req.files is a file object if user only sends one file per key.

In Fork, req.files is always array of file objects irrespective of whether user send multiple or one file per key.

We are doing this change because of better typing and validation.

4). Fix Bug. next middleware is called even after uploadTimeout.

In Original, uploadTimeout sends an error event to file that does cleanup and calls the next middleware without passing error. So express thinks that uploadFile middleware was successful but in reality it was not successful.

In Fork, call next(err) instead of next in file.on('err', ...) so that uploadFile middleware is not considered successful and error is propagated to appropriate error handler by express.

TODO

• Re add testing

Install

# With NPM
npm i express-fileupload-personalized

# With Yarn
yarn add express-fileupload-personalized

Usage

When you upload a file, the file will be accessible from req.files.

Example:

• You're uploading 2 files called car.jpg and bike.jpg respectively
• Your input's name field is foo: <input name="foo" type="file" multiple/>
• In your express server request, you can access your uploaded files from req.files.foo:

app.post('/upload', function (req, res) {
  console.log(req.files.foo.length); // 2
  console.log(req.files.foo[0]); // the uploaded file object for car.jpg
  console.log(req.files.foo[1]); // the uploaded file object for bike.jpg
});

The req.files.foo[index] object will contain the following:

• req.files.foo.name: "car.jpg"
• req.files.foo.mv: A function to move the file elsewhere on your server. Can take a callback or return a promise.
• req.files.foo.mimetype: The mimetype of your file
• req.files.foo.data: A buffer representation of your file, returns empty buffer in case useTempFiles option was set to true.
• req.files.foo.tempFilePath: A path to the temporary file in case useTempFiles option was set to true.
• req.files.foo.size: Uploaded size in bytes
• req.files.foo.md5: MD5 checksum of the uploaded file

Notes about breaking changes with MD5 handling:

• Before 1.0.0, md5 is an MD5 checksum of the uploaded file.
• From 1.0.0 until 1.1.1, md5 is a function to compute an MD5 hash (Read about it here.).
• From 1.1.1 onward, md5 is reverted back to MD5 checksum value and also added full MD5 support in case you are using temporary files.

Examples

• Example Project
• Basic File Upload
• Multi-File Upload

Using Busboy Options

Pass in Busboy options directly to the express-fileupload-personalized middleware. Check out the Busboy documentation here.

app.use(
  createFileUploaderMiddleware({
    limits: { fileSize: 50 * 1024 * 1024 },
  })
);

Limiting size of file that can be uploaded

Let's say we have an endpoint called /upload to which we can post csv file(s). Here is frontend code:

<form action="/upload" method="post" enctype="multipart/form-data">
  <p>
    <label for="csv">CSV File</label>
    <input id="csv" type="file" name="csv" accept=".csv" multiple />
  </p>
  <p><button type="submit">Submit</button></p>
</form>

Here is backend code:

app = express();

app.post('/upload', createFileUploaderMiddleware(), (req, res) => {
  let csvFiles = req.files.csv;
  if (csvFiles === undefined || csvFiles === null) {
    return res.send('No files were uploaded!');
  }

  const response = [];
  for (const csvFile of csvFiles) {
    response.push({
      name: csvFile.name,
      size: csvFile.size,
      md5: csvFile.md5,
    });
  }
  res.json(response);
});

If we want to limit the size of individual file that can be uploaded to 2MB, you can configure limits on middleware like:

// If we do not set `errorHandler` option, default behaviour of file uploader muddleware is to call next(err)
createFileUploaderMiddleware({
  limits: {
    fileSize: 2 * 1024 * 1024, // 2 MB
  },
});

// global error handler
app.use((err, req, res, next) => {
  // handle errors here, including any limit exceeded errors
  res.status(500).json({
    error: true,
    code: 500,
    message: 'Internal Server Error',
  });
});

If you want to distinguish errors coming from file uploader middlewares from other errors, you can configure an errorHandler like:

createFileUploaderMiddleware({
  limits: {
    fileSize: 2 * 1024 * 1024, // 2 MB
  },
  errorHandler: (err, req, res, next) => {
    // handle all errors from file uploader middleware here
    if (err instanceof FileUploadLimitError) {
      return res.status(413 /* request entity too large code */).json({
        error: true,
        code: 413,
        message: err.message,
      });
    }

    // pass errors other than limit exceeded errors to global error handler
    next(err);
  },
});

// global error handler
app.use((err, req, res, next) => {
  res.status(500).json({
    error: true,
    code: 500,
    message: 'Internal Server Error',
  });
});

NOT RECOMMENDED, but if you want to ignore all errors including limit exceeded errors, and continue to next middleware without files (req.files will be undefined), you can do something like:

createFileUploaderMiddleware({
  limits: {
    fileSize: 2 * 1024 * 1024, // 2 MB
  },
  errorHandler: (err, req, res, next) => {
    // ignore all errors and continue to next middleware (req.files will be undefined)
    next();
  },
});

Using useTempFile Options

Use temp files instead of memory for managing the upload process.

// Note that this option available for versions 1.0.0 and newer.
app.use(
  createFileUploaderMiddleware({
    useTempFiles: true,
    tempFileDir: '/tmp/',
  })
);

Using debug option

You can set debug option to true to see some logging about upload process. In this case middleware uses console.log and adds express-fileupload-personalized prefix for outputs. You can set a custom logger having .log() method to the logger option.

It will show you whether the request is invalid and also common events triggered during upload. That can be really useful for troubleshooting and we recommend attaching debug output to each issue on Github.

Output example:

express-fileupload-personalized: Temporary file path is /node/express-fileupload/test/temp/tmp-16-1570084843942
express-fileupload-personalized: New upload started testFile->car.png, bytes:0
express-fileupload-personalized: Uploading testFile->car.png, bytes:21232...
express-fileupload-personalized: Uploading testFile->car.png, bytes:86768...
express-fileupload-personalized: Upload timeout testFile->car.png, bytes:86768
express-fileupload-personalized: Cleaning up temporary file /node/express-fileupload/test/temp/tmp-16-1570084843942...

Description:

• Temporary file path is... says that useTempfiles was set to true and also shows you temp file name and path.
• New upload started testFile->car.png says that new upload started with field testFile and file name car.png.
• Uploading testFile->car.png, bytes:21232... shows current progress for each new data chunk.
• Upload timeout means that no data came during uploadTimeout.
• Cleaning up temporary file Here finaly we see cleaning up of the temporary file because of upload timeout reached.

Available Options

Pass in non-Busboy options directly to the middleware. These are express-fileupload-personalized specific options.

| Option | Acceptable Values | Details | | ------------------ | --------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | createParentPath | false (default)true | Automatically creates the directory path specified in .mv(filePathName) | | uriDecodeFileNames | false (default)true | Applies uri decoding to file names if set true. | | safeFileNames | false (default)trueregex | Strips characters from the upload's filename. You can use custom regex to determine what to strip. If set to true, non-alphanumeric characters except dashes and underscores will be stripped. This option is off by default.Example #1 (strip slashes from file names): app.use(createFileUploaderMiddleware({ safeFileNames: /\\/g }))Example #2: app.use(createFileUploaderMiddleware({ safeFileNames: true })) | | preserveExtension | false (default)trueNumber | Preserves filename extension when using safeFileNames option. If set to true, will default to an extension length of 3. If set to Number, this will be the max allowable extension length. If an extension is smaller than the extension length, it remains untouched. If the extension is longer, it is shifted.Example #1 (true):app.use(createFileUploaderMiddleware({ safeFileNames: true, preserveExtension: true }));myFileName.ext --> myFileName.extExample #2 (max extension length 2, extension shifted):app.use(createFileUploaderMiddleware({ safeFileNames: true, preserveExtension: 2 }));myFileName.ext --> myFileNamee.xt | | useTempFiles | false (default)true | By default this module uploads files into RAM. Setting this option to True turns on using temporary files instead of utilising RAM. This avoids memory overflow issues when uploading large files or in case of uploading lots of files at same time. | | tempFileDir | String (path) | Path to store temporary files.Used along with the useTempFiles option. By default this module uses 'tmp' folder in the current working directory.You can use trailing slash, but it is not necessary. | | parseNested | false (default)true | By default, req.body and req.files are flattened like this: {'name': 'John', 'hobbies[0]': 'Cinema', 'hobbies[1]': 'Bike'}When this option is enabled they are parsed in order to be nested like this: {'name': 'John', 'hobbies': ['Cinema', 'Bike']} | | debug | false (default)true | Turn on/off upload process logging. Can be useful for troubleshooting. | | logger | console (default){log: function(msg: string)} | Customizable logger to write debug messages to. Console is default. | | uploadTimeout | 60000 (default)Integer | This defines how long to wait for data before aborting. Set to 0 if you want to turn off timeout checks. | | errorHandler | false (default)function(err, req, res, next) | User defined error handler which will be invoked if there was any error, including exceeding limit error, while parsing files. |

Thanks & Credit

Brian White for his stellar work on the Busboy Package and the connect-busboy Package