Outsized Impact: How Confluent’s Engineering Team is Building the Infrastructure for Real-Time Event Streaming
Less than five years after Confluent launched as a “central nervous system” for event streams, the company’s platform now serves millions of people through some of the world’s top businesses — and the Engineering team still has plenty of interesting problems to solve. Below, Trunal Bhanse (Senior Engineering Manager), Gwen Shapira (Software Engineer), Guozhang Wang (Engineering Lead), and Victoria Xia (Software Engineer) reflect on what drew them to Confluent in the first place, share a few of the projects they’re currently tackling, and explain the unique opportunities — and challenges — that will set Confluent apart in the years to come.
For those who aren’t familiar, what is Confluent?
Guozhang: A lot of people think of us as the company founded by the original creators of Apache Kafka®, which is an open-source, distributed event streaming platform. But Confluent itself goes beyond that. We’re building an entire platform using Kafka as the core but supporting a lot of other technologies, too.
Gwen: The mission is to democratize real-time event stream processing — to make it possible for every company to get continuously updated data instead of static snapshots. For us in Engineering, that means solving some very difficult problems in distributed systems so we can maintain performance as we scale.
What are your roles on the team?
Victoria: I’m an engineer on KSQL, which is one of our product teams. If Kafka is a streaming platform, KSQL is a processing engine. It helps customers actually do something meaningful with their data.
Guozhang: I’m on the Streams engineering team. When a user enters a query, Streams takes direction from KSQL and does the behind-the-scenes legwork for operations like parallel processing, dynamic workload rebalancing, and fault tolerance.
Gwen: I work on the core functionality in Kafka, which is the server side of Kafka in a client-server architecture. It’s a distributed system, so our team tackles problems like replication and storage efficiency. We’re also doing a lot of work on elasticity and stability, to make Apache Kafka cloud-native.
Trunal: I’m a manager on the Experiences team, which works across all of Engineering to build great user experiences. We have members contributing to lots of different products, including KSQL, Streams, and Connect.
Tell us about your backgrounds and what brought you to Confluent.
Victoria: I joined last year after graduating from MIT. Before I interviewed, I didn’t really understand what Confluent was. But then I realized how many different things I could work on here. I had no experience with cloud, distributed systems, or open source. It was all new and exciting to me. And I could see myself learning a lot from everyone I met.
Gwen: I came here four years ago from Cloudera, where I was the first Kafka contributor. Then when I was moved to another project but wanted to keep working on Kafka, I reached out to Confluent. At the time, the space and the company were still very new, so I asked around about both the product and the team. Everyone said Confluent’s founders were the best people they’d ever worked with. I remember one person told me, “If you have a chance to work for them, even if it only lasts six months, it will be the best experience of your career.”
I also knew I could have an outsized impact here. Usually, you have to go to a massive company if you want your work to affect millions of people, and you might only get to contribute one small part of one small feature. But at Confluent, engineers get to write code that powers companies around the world, rather than the backend of just one company. Kafka is used by so many companies, and yet even today, our team is small enough that new hires get to work on some of our most challenging problems, and any idea could make the company twice as valuable.
“I asked around about both the product and the team. Everyone said Confluent’s founders were the best people they’d ever worked with.” — Gwen
Guozhang: I joined about four years ago, too. Before that, I was at LinkedIn, where I was working primarily on Kafka. Confluent was about to begin work on Kafka Streams, and I was intrigued by the idea of starting from scratch on something completely new. It was a chance to both learn more about and directly impact the whole modern software stack.
Trunal: I joined last year after five years at Airbnb, where I managed the Homes and Hosts organization. I knew about Kafka because I’d previously worked at LinkedIn. When I was looking at companies, I considered getting into another vertical, like autonomous driving or finance. But I realized that all of those companies needed something like Confluent to succeed. As Gwen said, you can have such a massive impact at a platform-level company, especially at this stage.
What are some of the technical problems you’re solving?
Trunal: People think being on a UI team at an enterprise company means building dashboards, but that’s only a small part of what we do. Because Kafka can support such large datasets in real time, managing that scale on a browser and through the APIs and infrastructure is the much more interesting challenge.
Gwen: We have so many projects going on, you can really pick whatever you’re interested in. There’s work to do on replication, consensus protocols, storage. If you like finding bottlenecks and optimizing the hell out of things, you can do that. Our team also works on solving some of the hardest technical problems, including exactly-once semantics and rearchitecting Apache Kafka® to be cloud-native.
“Because Kafka can support such large datasets in real time, managing that scale on a browser and through the APIs is an especially interesting challenge.” — Trunal
Guozhang: Absolutely — there’s a lot of project diversity. Not only because we’re building a platform for so many different industries, but because it renders from basically every tier of the stack. In our tech council meeting yesterday, we discussed how to make our network architecture more efficient; then we talked about the storage engine on top of that architecture and stream processing on top of that. Then there are the user experience modules. We’re serving a lot of sectors but still going very deep into the stack.
Victoria: KSQL is only about a year old, and it feels like new things happen every week. As we get more adoption, we’ll be able to iterate on feedback from those users and better prioritize new features. It’s a little overwhelming, but exciting to see the product grow.
What tools do you use?
Gwen: It depends on your team. If you’re working on the cloud, you use cool stuff like Kubernetes and Helm and Operators. For the core functionality in Kafka, we build our own tools for everything.
Trunal: We use top-of-the-line technology on the front-end side — React and basically anything else you can think of. We’re also diligent about keeping our tools up-to-date. We make sure we’re automatically on the latest version of all the open source libraries we use.
Guozhang: Confluent has always had a strong remote culture, and we work actively to maintain that. So across the board, we also use a lot of communication tools like Slack and Zoom, as well as our wiki. A third of my team is remote, and we never feel shy about hopping on a video chat for a few minutes when we have a question.
Tell us more about collaboration on the Engineering team.
Gwen: There’s a lot of collaboration. Probably 80% of my code is reviewed by someone who isn’t on my team because it’s a feature that will affect them. And with reliability engineering, for example, it’s not just throwing things over to the SRE team and forgetting about it. Someone on my team might build new monitors for Kafka, new metrics, new ways to operate or deploy faster to the cloud. A core value for Engineering, and Confluent overall, is that we’re all one team.
Guozhang: Depending on your role, you might collaborate with people outside Engineering, too. My team works a lot with Sales, Solutions Architecture, and Product.
Trunal: There’s a lot of cross-functional interaction on my team, too — we work with designers, product managers, Customer Success, Sales. For me, the “one team” mantra comes down to the fact that no one is serving their personal agenda. Everyone is working toward the same goal of making Confluent successful. That’s refreshing.
What’s the team culture like?
Guozhang: As much as I like the work, I think the people are the number one reason I’ve stayed at Confluent. I feel comfortable, encouraged, and challenged — it’s really satisfying. To me, that’s the definition of a good culture.
“I feel comfortable, encouraged, and challenged. To me, that’s the definition of a good culture.” — Guozhang
Trunal: We have some of the smartest engineers in tech, but everyone is incredibly humble. I didn’t know much about Kafka when I joined, and I’ve never felt awkward asking a question. People will take the time to explain how the entire ecosystem works.
I’ve also never worked at a company that supports remote work as much as Confluent does. The tools Guozhang mentioned make it so easy, there’s really no downside. We’re also careful about treating everyone equally. We have remote leads, and remote engineers get the same say as everyone else in how things are built.
What’s challenging about working at Confluent?
Victoria: There are definitely growing pains. You need to be okay with the fact that priorities shift, and you might start moving in one direction and get switched to another project. It can be hectic, but it’s fun. And as Gwen said, it comes with a lot of opportunity. If you see something that interests you and that needs to be done, you can go work on it. No one’s going to stop you.
“You need to be okay with the fact that priorities shift, and you might start moving in one direction and get switched to another project. It can be hectic, but it’s fun.” — Victoria
Trunal: You have to kind of embrace the chaos. If you want a place where everything is structured and you can just focus on one thing, this probably isn’t a good fit. You’ll be more successful here if you’re excited that 50 things are happening at once and don’t mind a lack of clarity on what you should chase next. Part of the job is to figure that out.
Gwen: We do get a lot of support — that’s part of the “one team” mentality. I got to learn Go from one of the biggest Go experts in the industry, and he was super generous with his time. But I don’t think you can depend on constant feedback. You won’t always be told, “Yes, you’re headed in the right direction.” The people who are most successful find their own resources, make their own connections, and speak up about what’s blocking them and what they need. You have to be independent and proactive.
Tell us about opportunities for career growth on the team.
Trunal: One great change our VP of Engineering, Ganesh, made when he joined was to establish clear paths for growth — whether you want to be a people manager or not. If you want to continue as an individual contributor, we have technical ladders, as well. There’s also an internal doc that explains what’s expected at each level, so it’s easy to see what you need to do to grow.
Coaching and mentorship is something we talk about a lot and take very seriously. We want to know what people are interested in so we can give them opportunities to explore. The management structure here is pretty light, and managers have a lot of direct reports, so doing weekly one-on-ones with every person can be a challenge. But we try. I have 12 people on my team, and we still meet every week. I want to keep doing that as long as I can. We also make sure every new hire gets a mentor they can go to with questions.
What’s most exciting to me is that as the team grows, we all have a chance to grow, too. As we bring in more smart people who have diverse backgrounds, they’ll lift up everyone around them.
Interested in joining the Confluent team?
Check out confluent.io/careers or email Stephanie Huie (steph@confluent.io).
