The Rise of Streamzilla: Leveraging Real-time Data in the Cloud
As Product Manager for Data Streaming at Porsche, Sridhar Mamella has been responsible for real-time data transport in all areas of the company for two years now. Following his previous blog post on the basics of Streamzilla, he explains how the internal data streaming platform becomes a highly scalable and reliable hybrid solution with minimal support effort — a fundamental step for Porsche’s path to the cloud.
Data streaming is an important part of Porsche’s IT strategy. Since 2019, I have been working as a big data and data streaming expert at Porsche to build a central data streaming platform called Streamzilla, based on Apache Kafka. The aim is to minimize support costs, prevent future data loss, and provide all teams at Porsche with an optimal data streaming platform — whether in development, production, or customer service, creating an overall positive working atmosphere. New ideas and use cases become adopted and implemented at full speed.
Porsche’s path to the cloud
At Porsche, we work in a wide variety of business areas with large amounts of data that must be processed instantaneously. Leveraging real-time data in the cloud goes hand in hand with Porsche’s “Cloud First Strategy”, which aims to bring all services and infrastructure into the cloud. With Streamzilla acting as the interface for the overall data flow, enabling bidirectional data allocation to and from the cloud, the foundation for a successful cloud migration is laid.
When implementing new features, it is always important to consider the requirements of the current use cases and to keep a permanent eye on the available capacity. Taking one step after another is the motto here — with all the enthusiasm and will to innovate.
The next step in our data streaming journey
Step one is done: the onboarding of all basic use cases is completed. Now it’s time to decide which goal to tackle next. One planned component is an Apache NiFi implementation, as some teams prefer this lightweight streaming solution. Another plan is to connect parts of the SAP system to Kafka, which is fundamental to Porsche, so that teams in our production, sales and customer service environment benefit from Kafka in the future. Many of our systems are very SAP-intensive; the connection to Kafka means an extensive project and a significant milestone for a successful implementation.
Kafka in itself can solve almost all problems in the area of data exchange, but the system must be permanently maintained. Operations and support of the platform proved to be costly and resource-intensive. Ensuring a high level of service scalability was a challenge. Thus, we are now working on the second development step of Streamzilla: Kafka as a managed service. The provision of a scalable high-quality service for all data streaming needs has to be guaranteed for teams in Europe and our subsidiaries all over the world.
“Sometimes you just have to talk to each other”
From a DevOps perspective, the central learning of the transition process has definitely been that you just have to talk to people and ask them what they need. Often, no one outside the team knows in detail what they need to do to work at their best. As in every relationship, the same applies here: communication is the key to success! It is important for us to get a common ground of understanding and to create a baseline on which the product is built.
However, it is just as important to remain in constant exchange and not to leave the teams alone with a finished product; cooperation is a continuous process, not a one-time agreement. By talking to teams and asking where the pain lies, we have discovered many use cases that we would not have thought of otherwise. We can build the platform in such a way that almost every requirement is directly considered. Our platform is flexible and is continuously being adapted. When working with DevOps there is still the possibility to do something better than before. In other words, there is always an even more elegant solution for the most diverse problems.
Looking to the future
Our goal is to finish all implementations in the pipeline as soon as possible, guaranteeing a smooth system for customer satisfaction, whether it’s an internal or external customer. Our strategic goal is to offer this platform-as-a-service for not only Porsche itself, but also our subsidiaries all over the world.
We work a lot with CI/CD pipelines, which continuously correct occurring errors. This is how our systems currently work. Kafka in itself is a very stable system: when we reach the point where it is a self-recovering system, then the interim goal has been reached. No more data loss, no more downtimes, at least as far as data flow is concerned. I am thrilled that we are well on the way to becoming a “real-time company.”
Sridhar Mamella works as a Data Streaming Manager at Porsche.
About this publication: Where innovation meets tradition. There’s more to Porsche than sports cars — we’re tackling new challenges, develop digital products and think digital with a focus on the customer. On our Medium blog, we tell these stories. It’s about our #nextvisions, smart technologies and the people that drive our digital journey. Please follow us on Twitter (Porsche Digital, Next Visions), Instagram (Porsche Digital, Next Visions, Porsche Newsroom) and LinkedIn (Porsche AG, Porsche Digital) for more.
