Setting up Backend (Part-2)

Welcome back! Or if you only just finished the first part, welcome to Part 2! In this second part, we will see how to handle the chunks we received from the client & play the video back in various formats depending on user bandwidth.
I am hosting this on AWS; you can host wherever you want, but please ensure you have root access to the VM or permission to install the application. We will be installing applications for handling video, audio, and other multimedia files.
Let’s launch an EC2 instance with Ubuntu OS (t2.micro)

Please select Ubuntu Server (free tier)

Just accept the defaults on every page till security groups where we need to allow a few ports below. Note we are running the node server on port 5000 & as we will be hosting the app on Netlify, We might get Mixed content error if the server (EC2) is on HTTP. Hence, we also need to open port 443 as we will be installing a free OpenSSL cert on the server that's right HTTPS for free.

The final step will be to generate a key-value pair to connect to our instance through SSH.

Generating Key-Value pair

Please save the .pem file safely as you will be only able to download this once.

Next, If you are on windows you can use Gitbash or putty I will be using the latter, please migrate to the folder containing the pem file and type ssh -i name_of_pem_file.pem ubuntu@ec2–14–265–896–75.ap-south-1.compute.amazonaws.com

Note: all AWS EC2 ubuntu instance have ubuntu as a default user so the IP address you will connect to will be ubuntu@public IPv4 DNS address of instance .

Once you are in we are going to install a few applications

1. FFmpeg

sudo apt update
sudo apt install ffmpeg

once this is installed you can verify by running ffmpeg -version it should return something like this

ffmpeg version 3.4.8–0ubuntu0.2 Copyright © 2000–2020 the FFmpeg developers built with gcc 7 (Ubuntu 7.5.0–3ubuntu1~18.04)

FFmpeg is the go-to open-source application able to decode, encode, transcode, mux, demux, stream, filter and play pretty much anything that humans and machines have created. We will use it to convert the incoming video to various formats and separate audio from video.

2.MP4Box

sudo apt-get install subversion
svn co https://svn.code.sf.net/p/gpac/code/trunk/gpac gpac
cd gpac
./configure --disable-opengl --use-js=no --use-ft=no --use-jpeg=no --use-png=no --use-faad=no --use-mad=no --use-xvid=no --use-ffmpeg=no --use-ogg=no --use-vorbis=no --use-theora=no --use-openjpeg=no
make
make install
cp bin/gcc/libgpac.so /usr/lib

Verify Install

MP4Box -version
MP4Box - GPAC version 0.5.1-DEV-rev5619
GPAC Copyright (c) Telecom ParisTech 2000-2012

We need MP4Box to generate an MPD manifest file.

A Media Presentation Description is a manifest file for MPEG-DASH streaming. For example, Youtube adjusts video quality based on users' bandwidth, well it requires an MPD manifest file and a custom video player & tons of cash but let get back to the topic, So the manifest file contains information regarding the format, resolution, codecs, etc. If you want to know more about it please check this link out.

3. Nginx

The good folks at Nginx have a very Helpful page in Docs on how to install it on an EC2 Instance with pictorial representation.

We require this to redirect all incoming traffic to the port running Node and limit the client’s upload size but mainly as a reverse proxy.

What are we trying to do here?

Let me explain in detail how this works?

Client Uploads a Video file in chunks, we save all the chunks in a temp folder, and once the upload is complete, we will merge them all and get the original file back.

Once we have the file, we will convert the video into various formats with the help of FFmpeg. For example, 360p, 520p & 720p formats also separate the audio file as well.

After which, we will provide the file path of all those converted file formats to Mp4Box which will generate an MPD File like below.

Let's dive deeper into an MPD (Media Presentation Description) file:

<Period duration="PT30S">
<AdaptationSet segmentAlignment="true" maxWidth="320" maxHeight="240" maxFrameRate="11988/400" par="4:3" lang="und" startWithSAP="1" subsegmentAlignment="true" subsegmentStartsWithSAP="1">
<Representation id="2" mimeType="video/mp4" codecs="avc1.64000D"        width="320" height="240" frameRate="11988/400" sar="1:1" bandwidth="171132">
    <BaseURL>only240_test_dashinit.mp4</BaseURL>
<SegmentBase indexRangeExact="true" indexRange="1033-1136">   
<Initialization range="0-980"/>
</SegmentBase>
</Representation>

Periods

Periods — An MPD element is a part of content start time & end time. We can use multiple Periods for scenes or chapters, or to separate ads from program content.

Adaptation Sets

Adaptation sets are set of media streams. for example, a Period can have one Adaptation Sets & multiple audio sets

Let’s say you are watching a movie on Netflix for simplicity sake lets just say the movie is available only in 720p format and in multiple languages So when you request a change in language the Adaptation set for the video remains the same but a different set of the audio file which matches your request is sent.

Now we all know that Netflix will have more than just one format, but ultimately with various formats, it’s just a change in Adaptation sets for different video and audio requests by user-agent or user.

Then there is Bandwidth, resolution, etc. Here’s a good article by Brendan Long with in-depth details of an MPD file syntax and its meaning.

Now, once we are done with the setup, let’s set up a basic node server nothing too fancy.

var express = require("express");
var app = express();
const cors = require("cors");
const session = require("express-session");
const key = require("./config").db;
const connectMongo = require("connect-mongo");
const MongoStore = connectMongo(session);
const fs = require("fs");
var resumable = require("./resumable-node.js")("./tmp");
const os = require("os");
const formData = require("express-form-data");
var bodyParser = require("body-parser");
const ffmpeg = require("./ffmpegArgs");
app.use(cors({
        origin: "My URL",
        methods: ["GET", "POST", "PUT", "DELETE"],
        credentials: true, // enable set cookie
        exposedHeaders: ["Content-Disposition"], //this is very imp.
     }));
//Managing sessions
let sessionOptions = {
    name: "SESSID",
    secret: "mysecret",
    saveUninitialized: false,
    resave: false,
    cookie: { maxAge: 3600000, sameSite: true },
    store: new MongoStore({ url: key }),
};
app.use(session(sessionOptions));
// Host most stuff in the public folder
app.use(express.static(__dirname + "/folfer_name _to_save_files"));
//Creating a temp dir for all the uploaded chunks
const options = {
    uploadDir: os.tmpdir(),
    autoClean: true,
  };
//We parse the form data with express-form-data
app.use(formData.parse(options));
//all the uploaded chunks go to the below path
app.post("/upload", function (req, res) {
//Chunks come in req.body along with chunk number and chunk size & all details
resumable.post(req.body, req.files, function (
      status,filename,original_filename,identifier) {
//If the status is done we merge in chunks and create a single video file
      if (status === "done") {
      var s = fs.createWriteStream("./folder_save/" + filename);
       
         s.on("finish", function () {
       
        //On finish event we clean the chunks from the drive     
       resumable.clean(identifier);
       res.status(200).send();
      });
    }

Please check the below code for the resumable js server-side. It takes care of everything from merging the incoming file, even to delete the chunks in the temp folder.

Once we merge all files into one, the client sends multiple GET requests which act as a trigger to start various format conversions & also we can control if we don’t want a particular format.

app.get("/", (req, res) => {
  //Just To Test
  res.send("ok");
});
//Now we hit various endpoints one by one using Aync & await on frontend just to keep things simple to start the process to convert into various formats

// To extract audio only
app.get("/onlyaudio", (req, res) => {
  ffmpeg.onlyaudio(filename).then((res)=>{
    return res.status(200).send(response)
    })
 }
//Next we convert it 240p, 520p & 720p 
//please check the repo link at the bottom for detailed code.

Lastly, we Generate an MPD file by giving all formats including audio to the MPD file function.

To play the video we access the MPD file directly in the Public folder like URL/folder_where_files_saved/mpdfilename from a Video player that supports DASH streaming.

Nginx Config

server{
        server_name Url with SSL; # managed by Certbot
        client_max_body_size 20M;
         proxy_read_timeout 3000;
         proxy_connect_timeout 3000;
         proxy_send_timeout 3000;
location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                proxy_pass http://localhost:5000;
                proxy_set_header Access-Control-Allow-Origin *;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection 'upgrade';
                proxy_set_header Host $host;
                proxy_cache_bypass $http_upgrade;
                #try_files $uri $uri/ =404;
       }
}

Working Example

Finally, to view the magic please visit https://dash-uploader.netlify.app/ a small app to test these concepts.

*This link is no longer valid big server cost*

While playing the video please open network tabs you should see something like this if you throttle down to slow 3G you can see the format changing from 720p to 240p or 540p.

Please check below for the detailed repo

aroirocks/videoserver

For some reason, Safari has an issue playing this I guess as they support a different video codec.

Feel free to let me know if I got something wrong.