Conduktor 1.17 brings programmatic access and granular RBAC to Kafka resources
Conduktor is the most complete interface for visualizing your Kafka data and performing Kafka operations
This month Conduktor released Console version 1.17, approximately 10 months since its general availability! Several key features have been added in this edition, including granular RBAC permissions, programmatic access via APIs, and a first step towards integrated service account management.
New to Conduktor?
Conduktor’s Console (UI) provides an unrivalled experience for developers that need to interact regularly with Kafka.
Conduktor also provides a market-leading Kafka proxy that sits between your client applications and your Kafka clusters. This transport layer can be enriched with features such as encryption, virtual clusters, chaos engineering and safeguarding rules.
1. Granular RBAC (Role-Based Access Control)
With the proliferation of micro-services, streaming applications and Kafka infrastructure comes an abundance of topics, schemas, consumer groups, connectors, ACLs and more!
At the same time, you have a bounty of different actors with varying interests in relation to the Kafka ecosystem:
- Developers that require autonomy within their scoped domain
- Platform Ops are responsible for ensuring the resilience and continuity of the Kafka platform
- Product Owners, Business Analysts, and Data Scientists that need to deep-dive and analyze data within topics
You might be scratching your head thinking, how do we please all these actors without compromising data security, ops efficiency, or platform robustness? The answer, of course, is Role-Based Access Control!
While Conduktor’s first iteration of RBAC focused on high-level read/write permissions on Kafka resources, 1.17 boasts the richest set of configurable permissions available in any Kafka UI.
With the new RBAC model, you can manage user and groups access to:
- Topics
- Subjects
- Consumer Groups
- Kafka Connectors
- Clusters (ACLs, Schema Registry Compatibility)
Within each resource, an additional layer allows administrators to grant specific operation access.
Let’s take Kafka Connect as an example. Bob is a support engineer on call over the weekend, responsible for recovering any connector task failures. Bob requires permission to:
- View the task & its status
- Pause/resume tasks
- Restart tasks
Bob is not permitted to edit the configuration, delete or deploy connectors, which goes beyond his responsibilities as a support engineer.
With Conduktor’s granular RBAC system, you can use pre-sets or define completely customized permission sets depending on the user or groups needs.
As Conduktor supports all Kafka providers, both cloud and on-premise, permissions can be granted across multiple clusters for a unified security model.
See the complete set of configurable resource permissions.
2. APIs for Programmatic Access / GitOps
Conduktor works best when deployed centrally for your organization. It integrates with your existing identity provider (IdP) so you can bring your source of truth user base.
Pair the above with the need to manage permissions at scale, and you have the perfect storm for automation. As a result, our product team's most frequent discussion with users is regarding APIs and GitOps.
For example:
- Synchronizing user or group-based permissions via an external application
- Application deployment performed through GitOps process
We are excited to announce that we have now opened the door to Console being driven via APIs. The first iteration focuses on:
- Managing clusters and certificates
- Managing users, groups, and their respective permissions (via the new RBAC model!)
The below shows an example request for listing permissions associated with a group ‘project-a’:
curl -X GET http://localhost:8080/public/v1/groups/project-a/permissions -H "Authorization: Bearer {token}"
[
{
"resourceType": "Topic",
"clusterId": "local",
"topicPattern": "projectA-*",
"permissions": [
"topicConsume",
"topicViewConfig"
]
},
{
"resourceType": "ConsumerGroup",
"clusterId": "local",
"consumerGroupPattern": "projectA-*",
"permissions": [
"consumerGroupView"
]
}
]3. Service Accounts
Kafka implements ACLs (Access Control Lists) to specify which users or applications can perform specific operations on resources.
ACLs have the following structure:
Principal P is [Allowed/Denied] Operation O From Host H On Resource R.Unlike the Conduktor RBAC model, they don’t cover subjects or connectors. Regardless, if you manage Kafka at scale, ACLs are likely already engrained in your organization. As such, it’s our duty to support them and help our users manage them efficiently.
The verbose structure of ACLs makes them difficult to digest at scale as humans. Many Kafka UIs will help you list ACLs but not effectively manage them en masse.
Console 1.17 introduces a new aggregation for viewing ACLs as service accounts.
ACLs are aggregated to a principal granularity in the new service accounts view. This enables a consolidated view of resources and operations, rolled up to the relevant principal, for all existing ACLs stored on your clusters. You’ll even be able to manage quotas, too!
While 1.17 introduces a read-only view of service accounts, soon, you will be able to use Conduktor as the sole interface for access management of applications requiring programmatic access to Kafka.
We also plan to integrate the service accounts feature with Confluent and Aiven via their APIs.
4. Quick Filters and Message Readability
The most core job-to-be-done within Conduktor is ‘finding a needle in a haystack’. Kafka can store an enormous amount of messages with often complex schemas. As such, when an orderId goes missing, or an application starts misbehaving, identifying the problematic message is time critical.
Console already supports advanced JavaScript filters for segmenting messages with specific criteria. However, 1.17 introduces a new method for displaying message structure and quickly filtering on specific fields and values. This provides an efficiency optimization for end users that need to navigate their Kafka data seamlessly.
Each key-value appears as a record in the flattened table structure. This works for most serializers that render JSON-compatible structures.
In-line with each key-value, you can also build a filter directly from the message data. If necessary, you can tweak the filtered value, without being concerned about the underlying message structure or the field path. This is super useful when dealing with deeply nested message data!
5. Performance boosts for larger environments
On top of the powerful new features already discussed, 1.17 also brings a major performance boost for larger environments.
Both the topic list and consumer group list screens are now served by snapshots. These snapshots are refreshed on a regular cadence, optimizing the time to display topics and associated metadata compared to making many parallelized, direct AdminClient calls.
This approach will be extended to all resource list screens, improving the time to render metadata and efficiently sort and filter through it.
What are you waiting for?
Regardless of whether you’re an individual, team, or enterprise, you can get started with Conduktor immediately and for free using one of these one-liners:
Launch Conduktor and connect it to your own Kafka:
curl -L https://releases.conduktor.io/console -o docker-compose.yml && docker compose upLaunch Conduktor with an embedded Kafka (Redpanda):
curl -L https://releases.conduktor.io/quick-start -o docker-compose.yml && docker compose upAlternatively, see our quick start for Docker or Kubernetes.
Ciao!
—
We hope you enjoyed this article and that it gave you some ideas for your Kafka infrastructure. Contact us if you want to discuss your use cases. We are Kafka experts and want to build out-of-the-box and innovative solutions for enterprises using Apache Kafka, so we are very interested in your feedback.
