Rust concurrency patterns: Natural Born Pipelines

A generic pipeline

  1. generate numbers.
  2. square them, using several workers.
  3. merge the results from the various workers.

Some “challenges” of a pipeline

Our chosen concurrency “primitives”

You are only as good as your idioms

  1. the Iterator trait.
  2. for loops.
  3. mpsc::Receiver's iter method, and especially it’s implementation of the IntoIterator trait.
  4. The Drop trait.

A forever-hanging pipeline in Rust

Our “generate” stage. Holds a sender to send generated numbers to the main thread.
“Square”, returns a sender on which work can be sent, and reiceive a “merge_chan”, on which to send intermediary results to the “merge” stage.
Our “merge” stage, is shared a “merge_result_chan”, to send final results to the main thread, and itself shares a sender with the outside world to receiver squared numbers on.

The pledge: create and share channels

  1. Results chan. Sender shared with the final “merge” stage, to send final results on back to the main thread.
  2. Merge chan. Sender shares with the “square” workers, to send squared numbers to the “merge” stage.
  3. Gen chan. Sender shared with the “generate” stage, to send generated numbers on.
  4. “Square workers”, simply a list of senders, through which we can cycle to distribute work among them.

The turn: start communicating

  1. Receive generated numbers via the “gen chan”,
  2. distribute the work by cycling over the “square_workers”.
  1. Start receiving over the “results chan”, waiting for the final results from the “merge” stage.
  2. In the background, each worker has a sender to the “merge” stage, hence each number that is squared is send to be “merged” via that channel, bypassing the main thread(see the “square” component code snippet above).

How to “Drop” our channels

The prestige: scopes

Moving on…

The End



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Greg Terzian

Greg Terzian

Software engineer living in Shanghai. I write in Javascript, Python, Rust, and English. Always for people to read.