How To Speed Up Download of Multiple Files By Parallelization In Symfony
It might be pretty annoying when a process failed which has long execution time and has to download something to perform an operation. Sometimes one of the download processes can fail on the middle of the file list and you may have to start everything from scratch. A little bit the parallelism increases your strength of speeding up the entire process. :)
Today, I wanna implement something with you guys together to bring a solution to this issue.
Ok. Let’s start with adding some dependencies to our project from the beginning. I assume the files we needed are in an FTP server. So I’m gonna use an FTP client to be able to download them. Also, we’ll use ShepherdProcessPoolBundle to be able to parallelize the download processes.
Run the following command to add the needed dependencies.
$ composer require ibrahimgunduz34/shepherd nicolab/php-ftp-client
I create a service definition for the ftp client to be able to inject it anywhere I needed.
I create a console command to be able to download the files from the remote server by the specified parameters.
Of course, we need to create a service definition to inject the ftp client to the console command we created.
We just created some basic components to separate the download process in a simple Symfony command to be able to run multiple instances of the download process. Right now, we are ready to start to the funny part of the job.
Now we need to create a new client class like the following to be able to download files in parallel.
As you see from the implementation, it just creates process instances to be able to call the FTP downloader command we created, by the process pool. Now we have to do three more things to use our new client.
First, you need to define the new bundle called
ShepherdProcessPoolBundle we installed before. The bundle setup methodology is completely dependent by your Symfony version.
Then, We need to create a new process pool to run the download processes in a separate pool. Add the following configuration to the project configuration:
Finally, we need to create another service definition to inject the new process pool to
And… Let’s see how we can use it easily.