Batch A Channel By Size Or Time in Go

Elliot Chance
Jul 23 · 2 min read
Photo by Perry Grone on Unsplash

There are some cases where you need to read enough items from a channel to begin processing. However, you also want to a TTL to expire with a smaller batch if there are not more items for a while.

Here’s an example of streaming a channel of string into an output channel of []string that guarantees:

  1. Each batch contains at least one string.
  2. A batch will never contain more than maxItems.
  3. If there is at least one string to be read at any given time, it will always send a new batch within maxTimeout .
  4. A batch will contain less than maxItems if maxTimeout has occurred since the the last batch finished (not since the first item in the batch). This is important because it means that if the stream is scarce and infrequent — such as one item per maxTimeoutBatchStrings will prefer to send a consistently timed batch of one item rather than more volatile intervals that might contain two items.

Try it now: https://play.golang.org/p/5NqYwehuxr-


The above example will halt when the channel is closed. It can also be written in a way that is context aware:

Now the stream can be stopped by either cancelling the context or closing the channel.

Try it now: https://play.golang.org/p/O7Wq1faP5qw

Elliot Chance

Written by

I’m a data nerd and TDD enthusiast from Sydney. I love exploring new technologies and working on modern ways to solve age old problems 🤓 elliotchance@gmail.com

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade