Golang batch processing with flush out within a time limit

Recently, my team was building the Kafka streaming consumers and using python Faust framework, that is a fantastic framework with many powerful features, somehow we were trying to write some other consumers in golang with kafka-go but it doesn’t have a beautiful Faust feature Take.

That Take feature is like a batch processing and flush data within a duration if the length of batched data is not enough so,

first, we were trying to imitate some library… (source)

so far so good, but what if we do it by ourself? (source)

what if we using some way like sync.Map to maintain two slices? using space to save time? (source)

seems great, what if we use channel instead? (source)

and memory usage

seems like the channel swap way wins! 🎉

Conclusion

the batch processing practice is very simple but combined with “flush no matter what within some duration” feature is getting more changeful.