huge uploader nodejs

huge-uploader-nodejs is a Node.js promise-based module made to receive chunked & resumable file uploads. It's made to work with its frontend counterpart huge-uploader.

From huge-uploader:

HTTP and especially HTTP servers have limits and were not designed to transfer large files. In addition, network connexion can be unreliable. No one wants an upload to fail after hours… Sometimes we even need to pause the upload, and HTTP doesn't allow that.

The best way to circumvent these issues is to chunk the file and send it in small pieces. If a chunk fails, no worries, it's small and fast to re-send it. Wanna pause? Ok, just start where you left off when ready.

The frontend module chunks and sends, this backend module receives and assembles all the pieces back together at the end of the process.

Installation & usage

1npm install huge-uploader-nodejs --save

As an exemple, I'll give something in pure Node.js, without any framework. But it is obviously compatible with any framework out there.

1const http = require('http');
2const uploader = require('huge-uploader-nodejs');
3
4// you must specify a temp upload dir and a max filesize for the chunks
5const tmpDir = './tmp';
6const maxFileSize = 10;
7
8http.createServer((req, res) => {
9    if (req.url === '/upload' && req.method === 'POST') {
10        // we feed the function with node's request object (here req),
11        // the temp directory path and the max size for the chunks
12        uploader(req, tmpDir, maxFileSize, maxChunkSize)
13        .then((assembleChunks) => {
14            // chunk written to disk
15            res.writeHead(204, 'No Content');
16            res.end();
17
18            // on last chunk, assembleChunks function is returned
19            // the response is already sent to the browser because it can take some time if the file is huge
20            if (assembleChunks) {
21                // so you call the promise, it assembles all the pieces together and cleans the temporary files
22                assembleChunks()
23                // when it's done, it returns an object with the path to the file and additional post parameters if any
24                .then(data => console.log(data)) // { filePath: 'tmp/1528932277257', postParams: { email: 'upload@corp.com', name: 'Mr Smith' } }
25                // errors if any are triggered by the file system (disk is full…)
26                .catch(err => console.log(err));
27            }
28        })
29        .catch((err) => {
30            if (err.message === 'Missing header(s)') {
31                res.writeHead(400, 'Bad Request', { 'Content-Type': 'text/plain' });
32                res.end('Missing uploader-* header');
33                return;
34            }
35
36            if (err.message === 'Missing Content-Type') {
37                res.writeHead(400, 'Bad Request', { 'Content-Type': 'text/plain' });
38                res.end('Missing Content-Type');
39                return;
40            }
41
42            if (err.message.includes('Unsupported content type')) {
43                res.writeHead(400, 'Bad Request', { 'Content-Type': 'text/plain' });
44                res.end('Unsupported content type');
45                return;
46            }
47
48            if (err.message === 'Chunk is out of range') {
49                res.writeHead(400, 'Bad Request', { 'Content-Type': 'text/plain' });
50                res.end('Chunk number must be between 0 and total chunks - 1 (0 indexed)');
51                return;
52            }
53
54            if (err.message === 'File is above size limit') {
55                res.writeHead(413, 'Payload Too Large', { 'Content-Type': 'text/plain' });
56                res.end(`File is too large. Max fileSize is: ${maxFileSize}MB`);
57                return;
58            }
59
60            if (err.message === 'Chunk is above size limit') {
61                res.writeHead(413, 'Payload Too Large', { 'Content-Type': 'text/plain' });
62                res.end(`Chunk is too large. Max chunkSize is: ${maxChunkSize}MB`);
63                return;
64            }
65
66			// this error is triggered if a chunk with uploader-chunk-number header != 0
67            // is sent and there is no corresponding temp dir.
68            // It means that the upload dir has been deleted in the meantime.
69            // Although uploads should be resumable, you can't keep partial uploads for days on your server
70            if (err && err.message === 'Upload has expired') {
71                res.writeHead(410, 'Gone', { 'Content-Type': 'text/plain' });
72                res.end(err.message);
73                return;
74            }
75
76            // other FS errors
77            res.writeHead(500, 'Internal Server Error'); // potentially saturated disk
78            res.end();
79        });
80    }
81
82    // unknown route
83    else {
84        res.writeHead(404, 'Resource Not Found', { 'Content-Type': 'text/plain' });
85        res.end('Resource Not Found');
86    }
87})
88.listen(8888, () => {
89    console.log('Listening for requests');
90});

Also, if the uploader is not on the same domain, don't forget to set a CORS policy. Either directly on node or on the reverse proxy. Here's an exemple for Node:

1res.setHeader('Access-Control-Allow-Origin', 'https://my-super-domain.com');
2
3if (req.url === '/upload' && req.method === 'OPTIONS') {
4    res.setHeader('Access-Control-Allow-Methods', 'POST,OPTIONS');
5    res.setHeader('Access-Control-Allow-Headers', 'uploader-chunk-number,uploader-chunks-total,uploader-file-id');
6    res.setHeader('Access-Control-Max-Age', '86400'); // 24hrs
7    res.writeHead(204, 'No Content');
8    res.end();
9    return;
10 }
11

Options

They aren't many options (all are required). As shown in the example, you pass the function:

• the request object,
• a directory to write to { String },
• the maximum total file size for the upload { Number },
• the maximum chunk size { Number }.

Be warned that total file size is computed by multiplying maxChunkSize by uploader-chunks-total header. So if the client is splitting files in chunks smaller than maxChunkSize, leading to a situation where uploader-chunks-total > maxFileSize / maxChunkSize, the upload will be refused although it might be smaller than maxFileSize.

Garbage collection

As said in the exemple, the module takes care of cleaning the successful uploads. But if an upload is paused and never resumed, its files are going to stay forever. So you should create a script called via a crontab that will erased directory older than the time you're willing to keep them.

Example bash script:

1#!/bin/bash
2
3find /var/www/tmp -type d -mtime +1 -delete

How to setup with the frontend

This module is made to work with huge-uploader frontend module. In case you would like to develop your own frontend, this module needs three specific headers to work:

• uploader-file-id a unique file id that's used to create temp upload directory for this upload,
• uploader-chunks-total the total numbers of chunk that will be sent,
• uploader-chunk-number the current chunk number (0 based index, so last chunk is uploader-chunks-total - 1).

Any other header will be ignored. Also, you can send POST parameters. Parameters send with the last chunk only will be processed.

Contributing

There's sure room for improvement, so feel free to hack around and submit PRs! Please just follow the style of the existing code, which is Airbnb's style with minor modifications.

To maintain things clear and visual, please follow the git commit template.