Adaptive Streaming with Piper

Photo by Jake Blucker on Unsplash

In my previous installment I showed you how you can use Piper to transcode a single video file and how to scale that out when you need to transcode a large volume of content.

In this article I’d like to take it one step further and show you how to use Piper to create an adaptive stream. “Adaptive Streaming” is a technique used by streaming providers (e.g. Netflix) to provide video over HTTP to their clients which adapts to the client’s (computer, phone, TV etc.) bandwidth limitations.

This is achieved by encoding the source video multiple times using different bitrates (the number of bits per second that can be transmitted along a digital network), thus letting the client pick the optimum bitrate at any given time.

So, for example, if I start watching a movie on Netflix during my subway ride home I will most likely have a relatively limited connectivity to the internet and therefore the Netflix app will most likely pick a lower bitrate stream (and therefore a lower viewing quality) at that point in time. Later, as I exit the Subway the application will adapt and maybe pick a higher bitrate stream for me. Finally, when I get home and connect to my Wifi I will most likely get the highest possible bitrate stream and therefore the best possible viewing experience.

That’s the idea behind adaptive streaming.

There are many standards for Adaptive Streaming but in this article I will use Apple’s HLS (HTTP Live Streaming) due to its relative simplicity and its ubiquitous device support.

So without further ado, here are the steps to convert your video to an HLS stream:

Create a work directory:

mkdir work
cd work

Start Piper:

docker run --name=piper \
           -d \
           -e piper.coordinator.enabled=true \
           -e piper.pipeline-repository.git.enabled=true \
           -e piper.pipeline-repository.git.url=https://github.com/creactiviti/piper-pipelines.git \
           -e piper.pipeline-repository.git.search-paths=video/ \
           -e piper.worker.enabled=true \
           -e piper.worker.subscriptions.tasks=2 \
           -v $PWD:/videos \
           -p 8080:8080 \
           creactiviti/piper

This will start a single piper instance acting as both a Coordinator and as a Worker with two task consuming threads.

Depending on your load, you can easily distribute your workers to separate machines and use RabbitMQ to allow the workers to communicate back and forth with the Coordinator.

Obtain source input:

wget "http://ftp.nluug.nl/pub/graphics/blender/demo/movies/ToS/tears_of_steel_720p.mov"

Single Bitrate Rendition

Let’s start with a single bitrate stream.

From the directory you started Piper at, create a directory for the output:

mkdir output_single

Start the hls_single pipeline:

curl -s \
     -X POST \
     -H Content-Type:application/json \
     -d '{
           "pipelineId": "video/hls_single",
           "inputs": {
             "input": "/videos/tears_of_steel_720p.mov",
             "output": "/videos/output_single"
           }
         }' \
     http://localhost:8080/jobs

Using the Job ID you got on the previous step, check the server for status:

curl -s http://localhost:8080/jobs/06e728d52bb6410387fe4a194f48ab12

Eventually, the job status should change to COMPLETED.

Next you need to put your output behind an HTTP server. a simple way to achieve that is by using the node.js module http-server but any web server that can serve your files would be just fine.

npm install http-server -g
http-server -p 8000 --cors

Test your stream using an HLS player using the URL to your local webserver : http://localhost:8000/output_single/720p.m3u8

(make sure you choose HLS from the drop down box)

Multiple Bitrate Renditions

Generating a single bitrate stream does not give you the benefit of adaptive streaming though, it simply splits you video file so it can be served over HTTP in chunks.

To enjoy the benefits of adaptive streaming we will have to encode the stream to multiple bitrates.

Create the directory for output:

mkdir output_multi

Start the hls_multi pipeline:

curl -s \
     -X POST \
     -H Content-Type:application/json \
     -d '{
           "pipelineId": "video/hls_multi",
           "inputs": {
             "input": "/videos/tears_of_steel_720p.mov",
             "output": "/videos/output_multi"
           }
         }' \
      http://localhost:8080/jobs

Test your stream using an HLS player using the URL to your local webserver : http://localhost:8000/output_multi/playlist.m3u8

(make sure you choose HLS from the drop down box)

If you look at your web server console log you’ll notice that the playback starts from the lower bitrate (480p) and quickly adapts to the higher bitrates.

Happy streaming.