Uploading files/images to MongoDB using GridFS

Hello,

Are you a Web Developer and wondering how to upload large files from your website to MongoDB, then you are at the right place. 😄

We will build a simple application where we will be able to upload files to the mongo database and retrieve files from there. We will do this using Nodejs with express framework and Mongo database. 💃

We will be going through various packages and knowing their uses while building a simple application where you can upload images and files to the Mongo database. To get an insight of how the app will work, move to the end of the post. 👇
We will be using GridFS to store files to the database (It is recommended to use GridFS only when the file size is at least 16mb).

Before going too deep inside the functionality, routings, rendering etc lets first understand the goal and shortlist the basic requirements of building this app, the goal is to take input file from the user, store it in MongoDB and then retrieve the files to display it to the user, and the basic requirements can be shortlisted as:

1. An HTML file to take the file input and view them.
2. A submission handler to send the files to the backend.
3. Express routes to receive the file uploaded and send it back to the frontend.
4. And a cup of coffee to keep you fresh and active. ☕ ️

Things We Will Use

• NodeJS : A platform for building fast and scalable network application.
• Express: Flexible NodeJS web application framework
• Mongo Database: for storing files
• Mongoose: people usually get confused between mongoose and MongoDB, just to make you sure that mongoose is a modeling library for MongoDB and nodejs.

• EJS: We will use ejs(embedded javascript templates) as our app’s view engine, this will enable us to generate HTML markup with plain javascript.
• GridFsStorage: This will be used for creating a storage engine.

We will learn about more packages as we will go on building our application. That’s all about boring stuff 😕, let’s start coding now 🎉….

The Setup

Setup a simple nodejs application using npm or using a manager of your wish. The basic structure of our application must be containing a view page called index.ejs which will contain the HTML code.

packaje.json containing all the packages used.

app.js which will look over all the functionalities as wells as routing, (note: we are not creating another file for routing as we have a very simple routing process.)

After the setup, the structure should be somewhat like this:

Structure

Here fileU is the parent of all the files of the work.

What we need here is a page where the user can submit the files and can also view them. Let us divide this page into two parts:

• Users will upload the file.
• Users will see the files he/she uploaded

I will be ignoring the bootstrap parts of the code, I hope you all will have a basic knowledge, and formally speaking it is just for styling purpose.

Now we will look after the first part among the parts mentioned above, for this, we will create a simple form with two inputs, one of type file and other for submit.

This code is a part of index.ejs

As of now, just understand that when you submit the form it will make a post request to the server, and it will send the files and get it uploaded to the database.

You must be wondering what does enctype = “multipart/form-data” mean, whenever you make a POST request to the server, you have to encode the data that forms the body of the request in some way and this encoding type allows files to be sent through a post. For more details about this, you can click here. We will use a middleware called multer to handle these type of encoded data, which will be discussed later.

Now, let’s go to the backend part, we will come back to the second part of the HTML page after uploading the files to the database.

To store this large sized files, it was earlier mentioned that we would use gridfs, GridFS is a specification for storing and retrieving files that exceed the BSON-document size limit of 16MB. Let’s understand why is GridFS efficient for storing large files.

Instead of storing a file in a single document, GridFS divides the file into parts, or chunks, and stores each chunk as a separate document. By default, GridFS uses a default chunk size of 255 kB; that is, GridFS divides a file into chunks of 255 kB with the exception of the last chunk. The last chunk is only as large as necessary. Similarly, files that are no larger than the chunk size only have a final chunk, using only as much space as needed plus some additional metadata.

GridFS uses two collections to store files. One collection stores the file chunks, and the other stores file metadata. When you query GridFS for a file, the driver will reassemble the chunks as needed and return the queried file.

files are stored as chunks in the databse fileU

Let’s look at our backend code first and then we will understand the functioning.

Let us understand the code line by line. First we are requiring all the packages which we will use, briefly talking about them: express is the framework we are using, bodyparser is required to get the content sent from the HTML forms, multer is required to handle multipart encoded data, multer-gridfs-storage is the storage engine for multer to upload file directly to MongoDB, gridfs-stream handles the chunking of the data, we will use gridfs-stream to view files.

Next, we are using embedded javascipt(ejs) as our view-engine and then creating a connection with MongoDB using mongoose by declaring the mongoURI ( mongoURI is the database location where you want to store your files, if you want to store it in the mlabs then you need to change this URI.)

Next, we are initializing a variable gfs and it is assigned to connection of grid to MongoDB is created and we will use this “gfs” for all further queries.

Then the most important part that is creating the storage engine. The main components here are the url, the file which is returned through a promise. Then we are declaring the function upload using multer, this function “upload” will be called when the user uploads the file.

Now its time to create routes and handle the files uploaded. Let us take a brief view of the routes that we will create in the app.

To upload a file or an image, we know that those files will come from the body of form that we have created in our ejs page, the submit button in the form will send the file to the backend. Check that in the form, we have set the action of form as “upload” and the method as “POST” (check the first snippet in the blog), so we just have to handle the POST method of the /upload to handle the file input.

Also, we have created a function upload using multer, we have to call this function when the user sends the file,(note that the action upload in the form and the upload function using multer, both are different).

Here we are calling the upload.single() function from the multer when the form submit button in triggered, the upload.single(‘file’) will handle the chunking of the file and upload it to the database. That’s it, your file is uploaded to the database! Hurray!

Now let’s check out the routings of the app, how to display files, how to display images, when to display them, how to send them, how to retrieve them from the database etc.

We will display the files/images in the index page it self, so in the get method of the index page, we will first find the files in the database, check whether the file is an image or not and then render those files to the HTML page.

Create a property .isImage for each file which will contain a boolean value of yes if it is an image else false, we will use this property later in the ejs template to check the file’s type and display accordingly.

Earlier I have mentioned the two parts of the HTML page we are building, we are already done with the first part, now the time has come to look after the second part i.e., displaying files :

After the files are rendered from the backend, first we will check if there are files sent by the server, and if yes we will check the file’s isImage property, if it is true, we will display the image in the HTML page else we will show the file name, and by clicking the filename you can view the file.

We will also embed a delete button with each and every file which will allow you to delete the file from the database.

In line 5 of the above snippet, we display the image with a image source code, in the line 7 we are creating a href to open the file with a href link code, and then we are creating a form with only a delete button for each file.

You must be wondering how these line of codes and delete button is working, let’s understand the last part of our backend code.

Earlier we have discussed about using GridFs-stream to display images. So whether it is an image or a file, we have to first read the file using gfs.createReadStream() function, and then return the file by piping. Piping is a great mechanism in which you can read data from the source and write to destination without managing the flow yourself.

In case of the delete function, we are just removing it by calling the function gfs.remove() by passing the required parameters to the function i.e., the files id and the root of the file.

That is all guys.. we have built an application where we can upload large files to the database, retrieve them, view them, delete them from the database. 🎉 🎉

After everything is done correctly the app will work something like this:

Working of the app

The GitHub link to the code for the above discussion is this.

Thank You for your patience for reading such a long post, clap if you like. 👏