PHP Microservices — Video Processing With RabbitMQ

Devin Dixon
Jan 2 · 5 min read

This entry is part of a series of creating microservices with various PHP implementations. In the previous tuorial we discussed building a simple mail microservice with sockets.This part will focus on video processing as a microservice.

Media processing is time-consuming and expensive on your hardware. Media includes video, audio, and images. If done on a monolithic architecture, especially on web servers, processing these files can be crippling to your end users experience.

For example, a single video file can take up to 15%-20% of your CPU. A few video files being processes simultaneously and users may no longer be able to load your website. Therefore, we are going to show you the basics of creating a microservice to perform video processing.

RabbitMQ + ProdigyView + FFMPEG

In this tutorial, we are going to cover RabbitMQ, a messaging and queueing service. php-amqplib is a PHP Library that uses AMQP 0–9–1 protocol for connecting with RabbitMQ. A queueing service is important for video processing because:

  1. A server has limited capacity on how many videos it can process at a time
  2. Large videos can take hours to process, and other videos must wait their turn until enough resources are available on the server

Other tools include ProdigyView, a toolkit for building micro applications, will be used for video conversions and downloading the file, and FFMPEG will handle the processing.

Run The Code

The code for this example is available at:

Sending With A Client

A client in this scenario is the server, ie your web server, that is going to send a message to another server to perform a task. In our example, let us imagine that when a user uploads a video to our web server, we will tell our video servers to process them.

With RabbitMQ, we are going to start by setting up the client. Before you begin, you must have RabbitMQ installed and running on your system. If you have not already done this, you may do so here. After that, in our client.php we start with the following:

$connection = new AMQPStreamConnection('localhost', 5672, 
'guest', 'guest');
$channel = $connection->channel();

The above small snippet sets the messaging server and the port. The code also handles authentication — the guest, guest part represents where a login and password can go. Next, we setup declare a queue.

We have now declared a queue named ‘video_queue’ that we will send our messages too. With RabbitMQ, we can declare multiple queues (image processing, sending emails, etc) for sending messages to different locations. Now let’s create the data we want to send over. Imagine it as a user uploading a video and now we have to convert the video so it works on all devices. We can create a simple definition like below:

Take note above, we are sending the URL of the video we want to process. We DO NOT want to send an actual video file because of the amount of time, server resources and blocked I/O that would take, making it very inefficient and impractical. We want to send a reference, ie an url of the video saved to cloud storage, to our video servers.

Next, we are going to take the data, JSON encode it and put it into a MessageObject as a string.

$msg = new AMQPMessage(json_encode($data));

And finally we can send the message to RabbitMQ:

$channel->basic_publish($msg, '', 'video_queue');

And that is it for a client! Relatively straightforward, nothing too hard or out the ordinary. Next, we are going to create our server for processing. Before we get into our server, here is a complete example of the client:

Setting Up Our Server

Now let's dive right into the server side of RabbitMQ. Again we start with setting the host, port, and authentication information.

$connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest');$channel = $connection->channel();

Also again, we are going to declare which queue we want to be listening too:

$channel->queue_declare('video_queue', false, false, false, false);

Now, is where the fun begins. When a message is received from the queue, it will be passed into an anonymous function as a callback. The function to execute the callback is called basic_consume, like below:

$channel->basic_consume('video_queue', '', false, true, false, false, $callback);

Now let's define our callback. In PHP, we can define anonymous functions as such:

$callback = function($msg) {   //Our processing logic will go in here}

The first action inside our function is to take the $msg object received from the client, and json_decode it to get our data.

$data = json_decode($msg->body, true);

Next, we are going to use wget, a Linux server function to download the video onto the server for processing. Why wget and not a file reader object or a function like file_get_contents?

Any function that reads a file into memory is BAD for our servers. Video files can easily reach gigabytes in size, and that will eat through your memory. Instead, we want to stream files to our server, aka read a file byte by byte. While a php file reader using fopen and buffers is ok, wget does a very nice job at streaming large files and should have less latency than most functions in PHP. We can use wget with the exec command.

exec("wget {$data['video_url']} -O video.mp4");

Next, we want to convert the file using FFmpeg. ProdigyView offers an abstraction class that makes the relatively seamless of converting out downloaded mp4 to a mov file.

Video::convertVideoFile('video.mp4', '');

And we are done! In a production environment, you would store the converted file in a different location. For a complete file of the server, see below:

Wrap Up And Next Tutorials

To run the above example on your local, you can visit the repo here.

And that is how to create a simple video processing microservice! Now in a production setting, there are more steps that go into the solution such as:

  1. Converting to multiple formats to account for the many devices and browsers and bandwidths(IPhone, Android, Chrome, Safari, etc).
  2. Storing the video in a solution that can scale over terabytes of data, ie s3.
  3. Delivering the video through a service such as Content Delivery Network.

But the point of the tutorial was getting you started in how to take the processing off your main server and onto a microservice using a queue. As the tutorials become more advanced, the next ones are:

Helium MVC

The blog of the Helium MVC and ProdigyView Toolkit

Devin Dixon

Written by

Entrepreneur, Technologist, Runner.

Helium MVC

The blog of the Helium MVC and ProdigyView Toolkit

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade