Golang Concurrency — Worker Pool

Concurrency Patterns

A Worker pools are a “concurrency pattern” in which a fixed number of workers runs parallely in order to work in a number of task that are holding in a queue.

In golang we use goroutines and channels to build this pattern. Usually will the workers be defined by a goroutine that is holding until it get data through a channel which is the responsible to coordinate the workers and the task in the queue (usually a buffered channel).

To make it more clear let us have an example, Imagine that you need to process thousand of records from a CVS and we need to save the date in our database. There is where we can use a worker pool, because instead of have only one process handling the data we could have more than one making it faster.

Let us see a comparison of the example using workers pools and without use it. In the first part I will show you the code and the result to solve the problem without worker pool.

Whitout worker pool

output:

As we can see it takes 54 seconds to save all the records from the CSV, that is a lot of time and could cause a lot of performance issues and the users whus wants to upload csv files to process won’t be happy with the experience.

How to fix it ?, well, we can go ahead with the worker pool approach. In the next example we will solve the same requirement but not implementing a worker pool and we will see the big differences. Wel hands on!.

Implementing a worker pool

Output:

Awesome! do you see the big difference ?, now the same process takes just 8 seconds. As you can see the usage of worker pools are very beneficial when we need to process a lot of data.

Important things. We have to define a function that will be used to define the worker process and as you can see it receive a channel to handle the data. Also you may note that we have to start the worker goroutines before to pass the data to the channel it will be that as the channel get values the workers start to process them.

And the last part, you need to pass the values to the channel and that is the logic that we can see in the lines 65 to 68.

Easy right ?
-Great now you know how to implement a worker pool!.