The economic viability of the open-source model for Apache Kafka and Confluent
A short case study of the tough act of balancing between OSS and proprietary
Disclaimer: This is my own analysis and is based on interpretation of public information. I’m trying to remain as impartial as possible and intend for this blog to be a case study around the OSS revenue model. If any corrections need to be made please let me know in the comments or on twitter @stephanemaarek
Apache Kafka has been open-source for over 8 years and will remain open-source forever. While open-source software (OSS) is free to use and resell for anyone, private corporations are spending huge quantities of money (the cost being their employees’ salaries) to maintain and improve open-source projects. Therefore these corporations must find a viable and healthy economical model around these open-source projects. This has been a topic of focus in the press recently.
Here’s my analysis on the strategy I think Confluent is following, and some recent developments. In the last part, I discuss the tough balance of choosing what is open-source and what isn’t.
Business Model 1: Sell support licenses for the open-source software.
This is something Confluent has been doing for a while. While Apache Kafka is free to use and setup, when set up within a corporation, it costs a lot to maintain as you usually need full-time employees to harden and monitor the infrastructure. On top of this, employees usually need to be well trained and will learn while on the job when encountering bugs or faults. As such, what makes sense is to rely on Confluent for support, because Confluent makes most of the contribution to the Apache Kafka project and will have the most expertise.
How viable is this model? It’s definitely possible for other companies to sell support, and this is something Cloudera (Hortonworks) and others are doing. Nonetheless, this licensing model works and applies to any Kafka setup: on-premise, in any cloud, on VM, on Kubernetes, or a mix.
Business 2: Software as a Service
Deploying software yourself, maintaining the infrastructure and paying for support may not make economical sense for every company. As such, corporations behind open source projects offer OSS as a service (MongoDB Inc., Elastic, etc…). In this “all in one package”, you get the software, an “elastic” consumption model, and usually some sort of tiered support based on your needs. Confluent Cloud is Confluent’s attempt at this and according to the statements made during the Kafka Summit of London 2019, business is going well.
How viable is this model? Selling open source software as a service is something ANY company can do, thanks to the permissive Apache 2.0 license terms. This is why if you need Kafka-as-a-Service, we have Aiven, CloudKarafka, Instaclustr, Confluent, and other players.
One very notable player is Amazon Web Services (AWS). AWS has a good track record of piggy-backing on OSS projects and selling them on their platform, usually being integrated with their VPC offering and possibly with IAM for security. AWS has had a track record doing so with Redis, ElasticSearch, PostgreSQL, and MySQL. The community and corporations backing these projects usually call for outrage as AWS has not had a good track record at contributing back to these projects. AWS seems to attempt to play nicer from now on, but time will tell. AWS also has the means to make their own proprietary software to mimic any API, such as Aurora for PostgreSQL (meant to compete with Oracle), or DocumentDB to compete with MongoDB.
Now, back onto Confluent vs AWS. AWS announced MSK as their own Kafka as Service offering a year back. This makes it easy for any of their customers to obtain a fully working Apache Kafka cluster in minutes. As a reaction to this, Confluent made some of their satellite open-source projects “source-available” to protect their IP and prevent AWS from using those directly (although that wouldn’t prevent AWS from re-implementing those with a compatible API, like mentioned). The change of license from open-source to “source-available” is what I’m discussing next.
Business 3: Freemium and weird licenses
Confluent has tried to build many satellite projects around Kafka. They started being open-source (REST Proxy, Schema Registry, KSQL) but most of them are now moved into the “source-available”. This new kind of license is specific to Confluent as far as I know, although most of the other companies (Elastic, MongoDB) have their own flavors of it; lawyers are hard at work. In Confluent’s case, this means in a nutshell that the code is available for everyone to see and use, although not resell as a service unless a special license is obtained. I don’t think Confluent has issued such a license or ever will. This blocks cloud providers such as AWS to offer managed Confluent Schema Registry or KSQL without doing a full re-implementation of these. Other companies such as Aiven circumvented this by re-implementing their own Schema Registry for example. As a customer of AWS though, you are free to deploy your own schema registry or KSQL set up on top of Amazon MSK, because you would own and manage the deployment of those.
While this won’t affect most companies, if you’re using the Confluent “source-available” products, this leaves you with “do-it-yourself anywhere” or Confluent Cloud as only options for migrations, which is in some ways a form of light lock-in.
Business 4: Proprietary software
This brings us to what I’d like to discuss in depth. Creating your own range of proprietary closed-source software is one of the best ways to thrive and survive as a company in the long run. Elastic has done that with their security plugins, and Confluent is doing the same with two products: their security offering and their global Kafka offering.
And here comes the fine game to play. As a private corporation that is VC-backed, what do you choose to commit to the OSS project versus do in-house? These decisions have a huge impact on reputation and profitability. Do too much of in-house and the OSS community feels left out. Do too much of OSS and competitors can make the business models described above work.
I’d like to discuss global Kafka. It is enabled through using a proprietary broker that is a “fork” of Apache Kafka, named “Confluent Server”. Using Confluent Server, one can have “observers” that are replicas not counted towards the ISR and acks. These replicas can be deployed in other regions and serve data to local consumers there.
How does it work?
- This would not be possible without this KIP:
Allow Consumers to Fetch from the Closest Replica. This KIP is in the Apache Kafka project and therefore is open-source. It allows for better performance because consumers can fetch locally and usually save on a ton of cost if as an example you’re using multiple availability zones on AWS. It makes tons of sense, is a true improvement to the project, and also enabled for Confluent to offer Global Kafka. Fine line, one can see as a win-win for OSS and Confluent. - The “observers replicas”, as far as I know, are proprietary to Confluent Server but would work with vanilla Apache Kafka clients.
- To improve the stability, scalability, and ease of operations of a Kafka cluster, it’d be great to replace a ZK cluster with a Kafka cluster. This is the goal of the KIP Replace ZooKeeper with a Self-Managed Metadata Quorum. That one is a huge piece of work, that I’d love to discuss on its own in another blog, that will take well over a year to be fully merged probably. It’s not new as it’s been discussed for over 4 years within the OSS community and it’s finally happening. I think overall it’ll be an improvement as it’ll enable Kafka to run more easily on Kubernetes, enable better security models, improve partition scalability and so on. It also enables the Global Confluent Server cluster to be more stable. Fine line again, a win-win for OSS and Confluent.
It’s clear these contributions to the OSS project do improve the Apache Kafka project in meaningful ways. It’s also apparent they enable Confluent to differentiate further their enterprise offering by offering proprietary features that probably won’t be rolled back into the OSS project. It’s not something new, the Confluent Replicator was also another proprietary piece of software meant to replace the shaky OSS “Mirror Maker”. Since then, Cloudera submitted a KIP to create Mirror Maker 2 in ways that probably are similar to how Confluent Replicator works.
Now it’s clear Confluent is at least on(c)e step ahead.
Take-away
Building a business around OSS is not easy and something companies struggle to do. I think Confluent building enterprise features in-house is a great model that will for sure seduce large corporations. For now, all the contributions into the OSS project have been useful to the Apache Kafka project and so they’re healthy. Still, one should be aware that more than 75% of the contributions to Apache Kafka is made by Confluent employees, and as such, I hope they’ll keep their impartiality in the future years. This would include letting other companies make KIP for features that compete directly with what Confluent is offering in-house, including global Kafka. Calling for volunteers, time will tell!
Learn Apache Kafka like never before
My new learning resource, Conduktor Kafkademy, is the quickest, easiest and most effective way for you to learn Apache Kafka for free.
- It’s absolutely free.
- It’s open and there is no registration required.
- It’s packed full of deep-dive lessons and practical tutorials, perfect for everyone from beginners to experts.
