Applying Modern Go Concurrency Patterns to Data Pipelines

Resilient, fast, efficient and concise? Pick all four! How to apply old and new concurrency patterns to pipelines in Golang

Photo by Victor on Unsplash

Table of Contents

A Simple Pipeline

Graceful Shutdown With Context

Adding Parallelism with Fan-Out and Fan-In

Before we move on to the next chapter, here’s a link to the full source code at this point in the blog post, so we’re all on the same page.

Error Handling

Removing Boilerplate With Generics

💡 Go 1.18 is now available and supports generics out of the box. You therefore no longer need the special -gcflags parameter when using Go 1.18

Maximum Efficiency With Semaphores

$ time go run -gcflags=-G=3 main.go
2021/09/09 13:51:17 sink: Bax
2021/09/09 13:51:17 sink: Bar
2021/09/09 13:51:17 error: oh no
2021/09/09 13:51:17 context canceled
________________________________________________________
Executed in 6.60 secs fish external
usr time 142.51 millis 0.12 millis 142.40 millis
sys time 282.23 millis 1.81 millis 280.42 millis

Summary

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
AMBOSS

Empowering all doctors to provide the best possible care. Join our team: amboss.com/us/jobs