How to upload a file (video) in chunks and Play it back with Dynamic Adaptive Streaming over HTTP. (Part -1)

Sending data over wire is really breeze in modern frontend web application with libs using XMLHttpRequest like axios or fetch which somewhat differs from the former as its harder to track upload progress & it handles Errors same as 200 ok status which sometime is undesirable as you would want to abort the process once you receive an error or treat it diffrently.

We use multipart/form-data or application/x-www-form-urlencoded as Content-Type headers in POST requests. However, It depends on the Data one method is efficient than the other, for example:

When you post a request with payload for x-www-form-urlencoded Content-type , say you want to post the below:

 — — — — — — — - -
| username | xyz | 
 — — — — — — — - -
| password | pass|
 — — — — — — — - -

The payload of the request is sent to the server in one giant string & name-value pairs are separated by an ampersand(&) like username=xyz&password=pass

the server knows where to start and to stop as the parameters are separated by &. However, In multipart/form-data the key & value pairs are encoded in their own section and are separated by boundaries. The boundary acts as a separator for each chunk of data.

POST /test HTTP/1.1
Host: foo.example
Content-Type: multipart/form-data;boundary="boundary"

--boundary
Content-Disposition: form-data; name="field1"

value1
--boundary
Content-Disposition: form-data; name="field2"; filename="example.txt"

value2
--boundary--

You can find more about the multipart content-type here.

Now for more control over these chunks, while uploading a file, we will use a JS library, Resumable.js. Again, you can visit the website here.

The library is designed to introduce fault-tolerance into the upload of large files through HTTP. This is done by splitting each files into small chunks; whenever the upload of a chunk fails, uploading is retried until the procedure completes. This allows uploads to automatically resume uploading after a network connection is lost either locally or to the server. Additionally, it allows for users to pause, resume and even recover uploads without losing state.

At this point I am assuming you are already familiar with creating a react app if not please click here.

lets start by installing the library

npm i resumablejs

and import it like import Resumable from “resumablejs”;

Let us add an input type=file and a Select dropdown to get more control over how much data we can upload at once and an Upload button. The code looks like this

import Resumable from "resumablejs";
const FileUpload = () => {
 const [chunk, setChunksize] = useState(1048576);
 const [inputfile, setinputfile] = useState(null);
return (
 <div className="row">
  <div className="col-xl-2 col-lg-2"></div>
   <div className="col-xl-3 col-lg-3 ">
    <label htmlFor="exampleFormControlFile1">
    {"Upload an mp4 file < 20Mb"}
    </label>
    <input type="file" className="form-control-file"
     id="exampleFormControlFile1" onChange={(e) =>
     setinputfile(e.target.files)} accept=".mp4"/>
   </div>
   <div className="col-xl-3 col-lg-3 ">
    <label htmlFor="Compress"></label>
    <select className="form-control"
     onChange={(e) => handlechange(e)}
     value={chunk}>
     <option value="262144">0.25Mb</option>
     <option value="1048576">1Mb</option>
     <option value="2097152">2Mb</option>
     <option value="4194304">4Mb</option>
     <option value="6291456">6Mb</option>
    </select>
    <small className="form-text text-muted">Chunk size</small>
   </div>
   <div className="col-xl-4 col-lg-4  mt-4">
    <button type="button" className="btn btn-secondary"
     onClick={() => setData()}>Upload </button>
   </div>
 </div>
 );
};

Expected UI (Nothing too fancy)

Here we have an upload button with a select Dropdown with predefined values in bytes we have a default value of 1048576 Bytes . Once the file is selected we setinputfile to a files array and extract the first file with inputfile[0] and when the user clicks the Upload button we run setData() .

const setData = () => {
  if (inputfile != null) {
   var file = inputfile[0];
   var r = new Resumable({
   target: `${url}/upload`,
   testChunks: false,
   chunkSize: chunk,
   simultaneousUploads: 3,
   query: { upload_token: file["name"] },
   });
  r.addFile(file);
  r.on("fileAdded", function (file, event) {
    r.upload();
  });
  r.on("fileSuccess", function (file, message) {
   console.log("sucessfully file uploaded");
  });
  r.on("fileError", function (file, message) {
   console.log("error Uploading the file");
  });
  r.on("fileProgress", function (file, message) {
  r.progress();
  });
 };
};

Which Provide us access to events like fileSuccess(file) ,fileProgress(file) ,fileAdded(file) , fileError(file, message) etc.

Based on which we can track chunk upload progress or know if the files uploading has been completed.

Let us try uploading a Video Mp4 file in 2Mb chunks

Once we click on upload upon selecting a file, Resumable breaks down the file into multiple 2Mb chunks to the last bit and assigns a number to each also sends it over to the server, where we parse the data using express-form-data .

https://{url}/upload?resumableChunkNumber=1&resumableChunkSize=2097152&resumableCurrentChunkSize=2097152&resumableTotalSize=10546620&resumableType=video%2Fmp4&resumableIdentifier=10546620-samplemp4&resumableFilename=sample.mp4&resumableRelativePath=sample.mp4&resumableTotalChunks=5&upload_token=sample.mp4

We send the same data as form Data as well

Later we will collect all the files in a temp folder in a node backend, merge them, and then start processing them in various video formats to stream them back to the user.

The Repo for this can be found On github https://github.com/aroirocks/VideoClient.git

For Part Two please click here