Apache Kafka and Immutable Infrastructure + Plus an introductory Kafka talk!
The above video contains two talks presented at the Kafka meetup hosted at GumGum on 19th November, 2019.
Speaker: Alex Woolford
In this session, Alex Woolford will walk us through some practical near real-time examples that touch on analytics, systems integration, and machine learning. We’ll show you:
- how to build streaming apps using a familiar SQL interface (KSQL)
- how to capture events from external systems
- how to deploy a machine learning model in a stream of events
He’s been working with a diverse set of open source technologies for the past 7 years, specifically Hadoop et. al, StreamSets, and most recently Kafka. Alex maintains a YouTube channel of geeky practical examples which can be found at https://www.youtube.com/c/alexwoolford.
Outside of work he enjoys floating in a tank, playing guitar (very, very badly), and yelling from the sidelines at his kids sporting events.
Speaker: Karim Lamouri
Apache Kafka and Immutable Infrastructure
Today is the day that you have to upgrade your Apache Kafka production cluster. You trained well on your staging cluster but there are a lot of steps to upgrade a cluster.
You are afraid that you will miss some steps on some brokers leading to production failures or configuration drift. You wish that you could deploy Kafka just like any of your stateless application. Immutable infrastructure can help you. It means that components of your infrastructure get replaced at every deployment, not just updated in-place. It ensures you a consistent, reliable, simple, predictable deployment process.
At GumGum we are running 4 Kafka clusters along with Kafka Connect and Schema Registry — more than 90 nodes in total. Using the techniques discussed in this talk we improved, simplified and accelerated operations on this infrastructure we manage. What you can take home from this talk:
- How to build a Kafka ecosystem as an immutable infrastructure
- Considerations to take into account to be able to have auto-healing
- How to speed up the configuration updates, upgrades and recovery of
your streaming platform with minimal headaches
- How to monitor your stack in this immutable infrastructure.
Karim is a French Software Engineer SRE at GumGum. He holds a master’s degree in Computer Science and he has been in charge of GumGum’s Kafka infrastructure for the past 2 years.