FanIn-FanOut Concurrency Pattern in Golang
Understanding Go concurrency
Concurrency is a fundamental aspect of Go programming, and the Fan-Out, Fan-In pattern is a powerful tool for harnessing its potential. This pattern allows you to distribute work across multiple goroutines (Fan-Out) and then collect the results into a single stream (Fan-In), enabling efficient and scalable concurrent processing.
Fan-Out:
In the Fan-Out stage, a single task is divided into multiple smaller subtasks, which are then executed concurrently. Each subtask is assigned to a separate goroutine, allowing for parallel processing and improved resource utilization.
Fan-In:
In the Fan-In stage, the results or outputs from all the concurrently executing subtasks are collected and combined into a single result. This stage waits for all the subtasks to complete and aggregates their results, providing a unified output. By leveraging the Fan-Out, Fan-In pattern, you can effectively tackle a wide range of concurrent processing challenges.
Industry-Level Use Cases
According to my experience trying to understand something without knowing the use case is useless. Let us see some industry level use-cases of this pattern.
Web Scraping and Data Extraction:
When extracting data from multiple websites, you can use Fan-Out to distribute the scraping tasks across multiple goroutines, allowing for parallel data extraction. The Fan-In stage then collects and aggregates the extracted data.
Image and Video Processing:
In tasks like image resizing, filtering, or video encoding, you can use Fan-Out to distribute the workload across multiple goroutines, each processing a subset of the media files. The Fan-In stage then collects the processed results.
Distributed Systems and Microservices:
In a microservices architecture, a client request may require data from multiple services. You can use Fan-Out to invoke the relevant microservices in parallel, and the Fan-In stage collects and combines the responses into a single, composite result.
Log Processing and Analytics:
When dealing with large volumes of log data from various sources, you can use Fan-Out to process the logs in parallel, with each goroutine handling a subset of the logs. The Fan-In stage then collects and aggregates the insights generated by the worker goroutines.
Batch Data Processing:
In scenarios where you need to process large batches of data (e.g., financial transactions, sensor data), you can leverage the Fan-Out, Fan-In pattern to distribute the data across multiple goroutines, process it in parallel, and then collect the results.
Understanding the working principle
Here I have given the working principle of the Fan-In so that in the implementation section you will be able to grasp the code easily. I have given the workflow for only the Fan-In stage because you can easily understand the Fan-out if you know how Fan-In works perfectly.
Workflow of Fan-In
Here the Fan-In stage will receive multiple streams of data. What it does is spin up (n) new goroutines, hand over each stream to each goroutine, and do the processing. Finally, publish the outputs to the single output stream.
Implementing Fan-Out, Fan-In in Go
Fan-In
Let’s dive into the code implementation of the Fan-Out, Fan-In pattern in Go. We’ll start with the Fanin function, which merges multiple input channels into a single output channel:
// Fanin is a function that merges multiple input channels into a single output channel.
func Fanin[T any](channels []<-chan T) <-chan T {
out := make(chan T)
wg := &sync.WaitGroup{} // using waitgroups to close the output channel when process is over
for _, ch := range channels {
wg.Add(1)
go func(c <-chan T) {
defer wg.Done()
for val := range c {
out <- val // without simply passing the data, you can do the heavy work here
}
}(ch)
}
go func() {
defer close(out)
wg.Wait()
}()
return out
}The Fanin function creates a new output channel and starts a goroutine for each input channel. Each goroutine reads values from its respective input channel and sends them to the output channel. The main goroutine waits for all the worker goroutines to finish before closing the output channel.
Fan-Out
Next, let's look at the Fanout function, which takes a source channel and the number of output channels to create:
// Fanout is a function that takes a source channel of type T and a number n,
// and returns a slice of n channels that receive the same values as the source channel.
func Fanout[T any](source <-chan T, n int) []<-chan T {
out := make([]<-chan T, 0, n)
for i := 0; i < n; i++ {
ch := make(chan T)
out = append(out, ch)
go func() {
defer close(ch)
for val := range source {
ch <- val// here instead of just passing, we can do a heavy work
}
}()
}
return out
}The Fanout function creates n output channels and starts a goroutine for each one. Each goroutine reads values from the source channel and sends them to its respective output channel.
By combining the Fanout and Fanin functions, you can create powerful concurrent processing pipelines in your Go applications. Here's an example of how you might use these functions:
source := make(chan int, 10) // here instead of a number channel you can use any channel as our implementatino supports generics
for i := 0; i < 10; i++ {
source <- i
}
close(source)
// Fan-out
outputChannels := Fanout(source, 4)
// Fan-in
result := Fanin(outputChannels)
// Process the results
for r := range result {
fmt.Println(r)
}In this example, we create a source channel with 10 integers, then use Fanout to create 4 output channels. The Fanin function then merges the results from these 4 channels into a single output channel, which we then process.
The Fan-Out, Fan-In pattern is a versatile and powerful tool for building efficient, scalable, and concurrent systems in Go. By distributing work across multiple goroutines and coordinating their results, you can significantly improve the performance and throughput of your applications.
