Conduktor Retrospective 2023

Here we are, welcoming 2024. Let’s take a moment to reflect on 2023, and see how far we’ve come: lessons learned, challenges tackled, and milestones achieved.

Here we are, welcoming 2024. Let’s take a moment to reflect on 2023 and see how far we’ve come: lessons learned, challenges tackled, and milestones achieved.

Our origin

Conduktor started by crafting a powerful Kafka Desktop application for developers. It was a hit! Before we knew it, hundreds of thousands of developers were using Conduktor Desktop to navigate their Kafka ecosystem.

This was only the beginning. Our vision is to accelerate real-time data adoption in organizations. To do this, we want to set platform teams up for success by assisting them in establishing security standards and rules and encouraging collaboration between isolated projects. At the same time, we ensure that developers maintain both the flexibility and independence necessary to avoid bottlenecks, allowing them to work efficiently.

Balancing the needs of developers with the broader concerns of organizations is tricky. Developers focus on their immediate needs and how to make their lives easier, while organizations have non-negotiable obligations to meet security, compliance standards, and data management policies. Striking this balance has led to our next-gen product, Console.

The need for a central Console

To solve this problem, we introduced Console in Oct 2022 (during our Milan offsite!) — a platform managed by the organization yet made for developers. Its mission? To allow the platform engineering teams to organize, secure, and support developers while eliminating any delays and frustration in the collaboration.

2023 was all about getting Console to market and helping our Desktop users migrate. It opened doors to learning from a broader range of customers, breaking into the larger enterprise scene, helping us clarify our purpose, and be more efficient. Sure, there were some hiccups at first because Console was new, but thanks to our customer feedback and hard work, everything was quickly fixed.

We’ve noticed a trend among enterprises: they’re moving away from using many highly-specialized products (where often only 20% of the features are used), and the rest are too complex or unnecessary. Instead, they opt for fewer, more versatile products covering a range of functionalities — all in one package. This shift is helping reduce costs and operational challenges.

Aligning Console with enterprise standards

As we shifted towards central teams and mission-critical ecosystems, we had to raise our game. Let’s look at some key aspects we introduced last year to support enterprises:

• Enterprise Environment: LDAP, RBAC, Audit system, Public APIs for automation and GitOps, OpenShift support, third-party Service Accounts.
• SOC2 Type II: We achieved SOC2 Type II certification, showcasing our strong data security and trust. https://www.conduktor.io/blog/what-we-learned-from-soc2-type2-write-what-you-do-do-what-you-write/
• Solutions Engineering: As we tackle more technical and organizational issues, we’re supporting our customers closely to solve their problems more than ever. We are building confidence for production use and critical path deployment. We don’t just sell products; we offer an experience (yes!). This includes our vision and best practice recommendations from real-world Kafka usage.
• Zendesk: We switched from Intercom to Zendesk. It better suits our needs and is more ready for enterprise use. The change has been a significant improvement in our Quality of Service.

A few numbers!

Let’s dive into numbers for 2023! The latest version of Console is 1.20 (changelog here). What changed since its inception?

• Docker size: 1.66GB to 800MB! We keep working on this to make it minimal.
• Dependencies Vulnerabilities: from 1C 27H 14M (sorry!) to 0C 0H 19M today (C/H/M stands for Critical/High/Medium).
• Onboarding: 5 screens to 1! In hindsight, it looks obvious, but it took quite some work to get there, and many constraints were relieved since. It’s really important for helping users start right. We’re still working on making it even better.
• Menu Product Switcher: 8 to 0! If you’ve used Conduktor before, you might recall the top-left corner menu that led to different features like Console, Testing, Data Masking, Topic as a Service, etc. It’s gone as we offer a unified user experience now.
• Topics: 1TB. The largest topic we saw running in Console. Performance matters!

We also all agreed we could offer a better user experience in Console. To achieve this, we focused on improving a few things:

• Faster Rendering & Unified Navigation: We improved the interface to make it more reactive and helpful for finding features. Good usability is key to building a positive reputation.
• Richer Controls: We added more options to customize the interface, allowing users to tailor it to their working style.
• Performance is a key driver: As our users sometimes deal with enormous Kafka clusters, providing them with a sub-par experience was unrealistic. We’ve worked hard to enhance this, ensuring a smooth experience for everyone: tiny or humongous cluster.
• Deployment Simplification: We made setting up our product on Docker and Kubernetes much easier. Initially, our setup was quite heavy on resources and caused many issues.

BUT, we knew Console alone couldn’t fully achieve our mission. An interface has its limits. To assist platform teams in enforcing data security standards and rules, we’ve introduced a new component…

The need to act on the Data Plane: Gateway

In mid-2022, we started R&D about developing a flexible Kafka Proxy called Gateway.

It’s deployed as a service in your infrastructure and connects to your existing Kafka clusters. It aims to solve problems that arise in organizations that need to scale Kafka across many teams and domains. In such cases, a company can implement organization-wide constraints and rules via the proxy, that align with how they wish to operate.

Gateway serves as a tool for Platform teams to set and enforce rules and modifications on the data going in and out of Kafka. It sits between applications and Kafka. We released it during Kafka Summit London 2023 (our review here).

A few use-cases it covers:

• Regulations and compliance: It enables encryption and data masking (independent of client programming languages), in a well-defined policy repository.
• Data Security: granular RBAC on all Kafka operations and resources, coupled with auditing capabilities.
• Federated Governance: It allows for logical clustering, introducing multi-tenancy and enabling the federation of multiple clusters as if they were one.
• Safeguarding: ensures that Kafka client configurations adhere to company standards by enforcing technical or data rules.
• Plug-and-Play: Gateway’s design capitalizes on the versatility of a proxy, allowing for various interactions with data in transit and Kafka operations. It relies on “Interceptors” enabling any developer to build one (in Java) and integrate their custom logic directly into the proxy. This flexibility allows for a wide range of applications and customizations.
• … tell us what you need, and we may implement it! product@conduktor.io

Note that we dogfooded it by using it at scale, hosting thousands of free multi-tenant Kafka clusters in our Cloud. We provided all our users with a free Kafka sandbox, avoiding them setting up a local docker to have a Kafka running (and avoiding the common problems of running Kafka on Windows). Thanks to this, we significantly enhanced the robustness of Gateway.

Lastly, Gateway and Console work together. Since Conduktor 1.19, Console can act as a configuration center for your Gateways. It’s possible to manage all interceptors from there, implement encryption, safeguard, etc. More direct integrations will come to make it even more seamless. We believe this holistic approach will facilitate user discovery and adoption.

Where do we fit in the data streaming ecosystem?

Our role is to complement your existing Kafka infrastructure with flexible tooling to help secure and federate it as a whole. We are not a Kafka infrastructure provider.

Conduktor fits into your Kafka ecosystem by augmenting and improving your existing setup. Whether you use AWS MSK, Confluent, Redpanda, Aiven, or another vendor, you can complement it with Conduktor.

We fully believe that data streaming represents the future of how organizations work with data. They must stay agile, responsive to market demands, and competitive. Our mission is to revolutionize how businesses interact with real-time data and speed up its adoption. The key lies in providing platform teams with tools that offer flexibility in implementing organizational standards and rules, leading to optimized data management. At the same time, we want to give developers the freedom to work independently through self-service and self-discovery, avoiding the common issue of a “Platform team bottleneck. Projects are delivered faster, and data is accessible in a secure way!

If Kafka is a car’s engine, we are the dashboard and the car’s gear shift: we help the organization get the most out of its engine.

Also, there are many open-source Kafka UI products available. Acknowledging their importance and understanding that enterprises vary in maturity levels, we’re moving towards a more inclusive free model that will cater to all developers and teams’ basic needs. This will set teams and organizations on the path to success sooner., offering them the flexibility to incorporate security and governance features at a pace that suits them.

Thanks for all the fish!

As you can see, 2023 was a breakthrough year for us. We learned a lot from the market; we put ourselves in a unique position and are definitely embracing enterprise challenges.

In 2024, we will double down our journey to manage and secure data at scale, embracing Data Mesh principles and helping developers make their lives easier while being autonomous. Our goal is to provide the maximum value to your Kafka infrastructure by streamlining the implementation of rules and processes.

It became clear that what we build addresses the challenges large organizations face in their day-to-day when leveraging their real-time data. As we expand, continue to develop our product, our impact on the data streaming ecosystem will become increasingly significant. You know the saying: “If a tree falls in a forest, and there’s no one around to hear it, does it make a sound?”. Let’s make some noise!

Oops

Hey, we also failed. We decided to discontinue two of our product lines. As you know, 2023 was a year of refocusing and efficiency for every company, and we were no exception.

• Conduktor Cloud was not a success. We faced challenges due to its lack of networking connectivity (local agent support, no VPC peering), reliance on multi-tenancy without single-tenancy options, and customer preference for on-premise solutions over SaaS, highlighting security and privacy concerns.
• Conduktor Testing aimed to make Kafka easily testable by testing the data within Kafka, running end-to-end testing, and generating testing scenarios. It failed to gain market traction, possibly due to inadequate advocacy or market readiness, leading us to decide to deprecate the product despite customer support and engagement.