NodeJS: Uploading file to S3
May be we have some user generated files, uploaded via a form. Or may be we’re generating some files for our users to download. Either way, we need a place to store these files. Conventionally we store these files on our local disk. This works fine if we only have one server. Or if we can manage to share a disk with multiple servers. A much easier solution is to use the cloud to store our contents.
For storing files, my personal choice is AWS S3. In this quick post, I am going to show how we can easily upload files to S3 from our Node app.
To upload files to S3, we will need the aws-sdk package from npm:
npm install -S aws-sdkNow we can start writing our actual code:
The code is in TypeScript but if you know JavaScript and latest ES features, it should not be difficult to understand.
In this code example, we’re going to upload a file that’s already on our disk. At the top, we have imported necessary modules for this task. We need the fs module to read the file, aws-sdk for uploading to S3. We then defined our AWS credentials and the bucket in which we would upload the file. Ideally, we would read these values from environment variables or a config file.
Then we initialize a S3 client with AWS credentials. The uploadToS3 function does the actual file uploading. Inside it, we at first create a readable stream from the filename. For brevity, I skipped error checks. We’re then constructing the options we need to pass to the upload method on the S3 client. This includes the bucket name, an unique key for the filename and the file content.
As the file content, we’re passing the readable stream. Please note, we could have read the file synchronously and pass the entire file contents directly here. That would have worked fine for small files. But if you’re trying to upload a large file, streaming upload is much more efficient.
We’re calling the upload method with the parameters passed. In this case, we’re wrapping a callback based API to return a promise. When the operation finishes, inside the callback, we handle any errors and then resolve the promise with the response returned from S3. By this time, the file has either finished uploading or failed. We no longer need the file stream. So we destroy the readable stream.
The uploadToS3() file returns a promise object which resolves with the data returned from S3 or rejects with any errors.
In our example, we uploaded a file from our local disk, which was already there. We can also adapt this code example to accept a readable stream directly and redirect uploaded files to S3 without first saving them on disk. But I would leave that as an exercise for the reader.
