How To Make a Media Streaming Server with Express and MongoDB
Hello and welcome! In this article we are going to focus on creating a media streaming server from scratch using the following libraries:
These are just to name a few. We are obviously going to use more packages that are going to help setup our server , but for the most part the ones mentioned above are going to give us the desired functionality.
At this point you might be wondering what type of media can we serve? Well, basically we can store and serve anything — you name it: images, audio, videos, PDFs, just to name a few. This can be extremely handy especially if you want to stream videos from a server similar to YouTube or Netflix (Obviously not at their scale).
Okay, enough chit-chat let us go ahead and create our project folder. I am going to call mine media-server, you can call it whatever you please.
Inside this directory go ahead and initialize a new node application by writing the following command:
yarn (I will be using this for the entire project)
yarn init -ynpm (included with nodejs)
npm init -yThat should create a package.json file in your directory. This file has all the configurations for our project. You should definitely read more about it if you have no clue what I am talking about.
Setting up Express
To set up a simple express server we first have to install the following packages:
yarn install express corsJust to give a little insight into what these packages do, express allows us to set up a nodejs server quickly and cors will remove Cross-Origin Errors (Requests to our server from unknown origins). I encourage you to read more about these technologies if you have not already.
Now, create a file named index.js and write the following lines of code.
This is the basic setup for an express server, now run the following command to run the server:
node index.jsGo to http://localhost:5000/ to verify it is set up correctly. You should see the following:
MongoDB
So now that we have the server running we can take a moment and set up our MongoDB cluster. First, make sure to register an account and sign in. You should be able to see the main dashboard like so:
Go ahead and click Build a Cluster. The following steps are just choosing a tier(in this tutorial I will be using the free tier) and then you can choose a cloud provider (I will be using AWS in us-east-1, you can choose whichever region is closest to you). Following those two steps, your cluster will build (this may take a couple of minutes).
Once the cluster is up and running you can go ahead and click on connect. This will prompt what type of connection you want to establish. You want to click on Connect your application. A connection string will show, go ahead and copy it for later use (remember to replace the placeholders in the string with your actual credentials).
Time to go back to our project and establish a connection between MongoDB and our application. Go ahead and open a terminal in the project’s root directory and install the following package:
yarn add mongooseMongoose is a very nice library that allows us to connect to a MongoDB cluster and establish schemas for validating incoming data in our database. Now that we have that package installed, let us create a new folder called services in our project. Inside that folder we declare a new file named db.js and write the following code:
Here we create a nice function that establishes a connection with our MongoDB cluster. You may notice I am using an environmental variable to establish the connection in the mongoose.connect function. This is to protect the URI from being seen. You can either paste your URI ( what we copied before) directly there or use a package such as dotenv to define environment variables in your application.
Let us go back to index.js and call our function to establish a connection:
If your application does not crash you are in the clear. If it does crash make sure to verify your URI or type a console.log statement in the catch part of the connectDB function to read out the errors to the terminal. There we go, our database is ready to accept requests! Now we can proceed to add the storing and serving properties.
GridFS & Multer
MongoDB on its own cannot store large files, so there must be a way to achieve saving large data. GridFS is the standard of storing large data in MongoDB. Essentially, very large data is divided into chunks that are later saved in the database itself and all have some reference to each other. This is a great standard since you do not necessarily have to wait for all the chunks to be collected to start streaming data when piping a request. Another question you may be asking yourself: well, how do I send my files from my client to my server?? . It is a great questions and the response is multer. You can pass data from the frontend to the backend appending the files to a form data object and passing it in the request. Multer is going to detect those files and store them either on your server or some other medium you provide. We are going to use a neat package called multer-gridfs-storage that is going to take those files and store them directly in our cluster. Now, to retrieve the files from the database we are going to need a package called gridfs-stream. Sounds easy enough right? Let us begin:
Install the following packages:
yarn add multer-gridfs-storage multer gridfs-streamOnce that is done, go ahead and write this code:
NOTE: Most of the code is wrapped in an async function called init which I later call below, this is done to benefit from the await keyword. We do not want to continue until all the essential connections are made.
Now, it might seem like a lot is going on but only three essential things are done here. We first establish a GridFS standard connection with our database so we can retrieve files and their chunks ( from the media bucket). Second, we initialize storage in our cluster. Lastly, we initialize multer and pass in the newly created storage.
Now we have everything we need to set up some routes and test it out. Let us go ahead and define a route POST route for uploading files, a GET route for retrieving all those files, and finally, another GET route for streaming media, and a DELETE route for deleting a file.
These routes are all in the init function right under the variable upload. The only worthy mention here is the read route (/read/:filename). This route basically opens a stream from the database to the API slowly passing bits of information that can be piped into the response, essentially passing bits of the file that can be passed to the client. So instead of waiting to retrieve the entire file, we can send to the client what we have so far. If you have ever noticed images loading by pieces or videos buffering, that is essentially what is happening.
Everything looks good and our server is operational, but we do not have a UI to test this out. Now, we could just use a view engine such as ejs, but we want to mimic an actual service that contains an individual client and server. So for this article, I am going to create a simple react project and do very little styling.
To start off go ahead and make a react project with the name of your choosing (I am calling mine media-client).
yarn create react-app media-client
Now, we are only going to modify the app.js and the app.css files. So let us first begin with the app.js file:
Here we have quite a lot going on here so I am going to explain briefly what is happening. To begin with, I added a package called axios which is for making HTTP/S requests. I declared two variables with the React-hook useState. This allows me to declare a state outside of a class. This is known as Reacts functional programming style. The useState hook has two parameters which are the variable itself and a setter function. Now, we have a variable called file and another files. The variable file is going to contain the file we want to upload. While files will contain the files that we fetch from the database.
The function fileHandler is in charge of grabbing the file passed into the input component and setting the file variable to that file. The uploadFile function is in charge of sending the formdata to the /upload route and the view renders the new file without having to refresh the page. The removeFile function does exactly that and it also removes in real-time the file from the view, so no refresh is needed. Lastly, we pass the /read route to the anchor tag’s href property. That means we do not have to make a request with axios instead the href will allow us to view whatever media we have there when we click it. Now, you do not have to use an anchor tag, if the media were an image you can use an img tag and pass the route as the src and it will show the image. The same thing happens with audio and videos, you can use the video or audio tags and they will show. The only exception might be some files that will try to download when you click them (power points, excel sheets, word documents, etc).
Before we show the final product, let us do some styling some we can appreciate better the application. In app.css write the following:
Now, we are finally done! Let us see how everything looks put together. Run the client with the command:
yarn startRun the server with
node index.jsBoom, this is what you should see:
Let us add a file, click browse, choose a file and click upload.
The file is up on the server and now we can go ahead and click on the link to see what it is.
A cute little bunny, awesome it works, let us try something else!
This time let us upload a video. Now, it is important to keep in mind that videos may take a while to upload especially if they are of very good quality and if your internet speed is slow. Let us go ahead and click on the video’s link:
The server immediately starts to play the video. Super cool, RIGHT? Awesome and there we go. That is how we make a simple media streaming server. Hope you guys learned a thing or two. Now, go ahead and apply what I taught you and make the apps of the future! Have a great one! CHEERS!
