Free Software is the only winner in Elastic NV vs AWS
At the heart of Open Source beats Free Software values. The AWS announcement that they are launching the Open Distro for Elasticsearch, an
upstream-compatible distribution of Elasticsearch and Kibana, is a victory for
those values. So is the response by Elastic to stay the course on their current
trajectory, of releasing much of their work under proprietary license, and
intermingling that open source code with the proprietary source.
The core values of free software are embodied in the four freedoms:
- The freedom to run the program as you wish, for any purpose (freedom 0).
- The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1).
- The freedom to redistribute copies so you can help others (freedom 2).
- The freedom to distribute copies of your modified versions to others (freedom 3).
These values matter because they ensure that, no matter who you are, no matter your station in life, no matter what the reason may be: this software exists for you. That it exist in your world, and be useful to you — that is its
purpose. Strip away all the other concerns — about business, about collaboration — at its heart, Open Source and Free Software are about the freedom to make the system work the way you wish.
Open Source covers those core values with a sheen of business value. It says
that the collaboration and community fostered by those values is a better way
of building software, and a better way of building business value. For example, open source software very frequently has little to no acquisition cost — it is both free to receive and trivial to acquire. Open Source says this is a huge business value upside — that getting people easy, low friction access to your software creates a much larger pool of potential customers.
Riding atop both of these concepts is that of being a community. When we work together on a piece of software for a common purpose, we form a Community. In the case where our communities include commercial ambition, we move beyond just thinking about the software’s best interest. We also care about our own share of the pie: how much of the money is my money? How much is yours? Is it “fairly” distributed? If we find a way to work together, for the common good of everyone, we stay together. When we can’t, like any other community, we splinter.
Shay Banon (and many others) built Elasticsearch. He licensed it under the
Apache License. He started a company, Elastic, to monetize it using the tight
open core model. By putting the core of Elasticsearch into the open, we can presume he wanted the business value benefits of Open Source — collaboration in the commons, low friction acquisition for users, and hopefully the growth of an ecosystem around it. He got it — Elasticsearch was widely adopted, tools like Kibana and Logstash were built around it, and it became the most common method for getting at-scale full text search. They have been wildly successful, going IPO in 2018, and doing quite well as a business.
Their business model utilizes a strategy I term “tight open core”. This means that the primary functionality (the search engine) is under an open source license; but that direct, and often critical, features are only available under a proprietary license. Elastic is the strongest example of this model that I know — critical features like authentication are proprietary plugins. Over time, Elastic has opened up these features somewhat — they produce them under a source-available, proprietary software license. They also co-mingle the source code for these features in the primary Elasticsearch repository, and
bundle them in the default Elasticsearch distribution (there is also an elasticsearch-oss, which removes those components).
At this point, ask yourself two questions: to whom does Elasticsearch belong? The community, or Elastic NV? Why do you answer the way you do? I’ll wait.
Got your answer? Here is mine: Elasticsearch belongs to Elastic NV. Why do I say that? Because, for all that Elasticsearch is Open Source, it exists primarily to fuel the commercial ambitions of Elastic NV. That those ambitions embrace the value of being Open Source, and thereby embody some of Free Software’s values, ultimately aren’t the reason the software exists in the world as it does. If I, as a contributor, want to change the course of Elasticsearch in ways that benefit me (and perhaps others), but does so at the expense of Elastic NV, will I get that opportunity? The answer is most likely no — you will not.
Elastic are generous in many ways here — their proprietary functionality is
provided via clear plugin APIs to the core itself. However — the moment they
adopted the tight open core model, it was clear that Elasticsearch, and any
other component like it, wasn’t primarily designed to be a sustainable
community resource. It was designed to get the benefit of open source
collaboration and reach, of development in the commons. But it isn’t now, and likely never was, intended to be something that was shepherded by the community for the mutual benefit of all. Because when push comes to shove, Elastic NV is Elasticsearch. If it’s your interest against theirs, their interest will
win. That truth is ultimately corrosive to sustainable communities. The only question is whether or not a situation will present itself where that corrosion is fatal (and by fatal, I mean fatal to the idea of an Elasticsearch community, not the company, or the software).
Lets bring AWS into the picture. In October 2015, they launched the Amazon Elasticsearch Service. Built on top of the Apache Licensed Elasticsearch core, it brought automatic scalability and near-instant provisioning to their customers. They gained a lot of customers, because, like I said earlier, Elasticsearch was quickly becoming the de-facto choice in the space. AWS has their own commercial interest in Elasticsearch at this point: they monetize its existence directly. Of course, they must provide many of the features that Elastic keeps proprietary — authentication, encryption, etc.
Lets spend a second talking about AWS. They are a company driven by a deeply held belief in doing what customers want. They stand behind the services they launch, no matter what. They take responsibility for their customers experience. Once they launched the Amazon Elasticsearch Service, you can bet that they had an internal understanding that they were responsible for it being great, and meeting their users expectations, until
the end of time. That drive and focus guides a huge amount of their decision making, and it’s critical to understand what happens later.
The status quo is now different — we have an asset, undeniably Open Source, but that is fundamentally aligned with having the bulk of the monetary assets flow to a single entity (Elastic NV). We have a much larger company, Amazon, now monetizing that asset directly, by offering it as a service to their customers. If you are Elastic, what do you do with your asset?
The answer is, you differentiate it. You put as much distance between your product, which you control the destiny of, and your new competitor as possible. You do this through a combination of things: release cadence (you have newer versions), proprietary features, and deeper, more thorough support. If you’re smart, you embrace the validation that the AWS Elasticsearch Service brings your product, while also pointing out that if you want the Real Thing, you should get it from the Elastic NV distribution of Elasticsearch.
Over time, Elastic made the default distribution of Elasticsearch be not only the open source core of Elasticsearch, but also their suite of proprietary plugins as well. This distribution is not available under open source terms — it has its own license. In addition, they co-mingled the open source code with the proprietary plugins they include in the distribution. For many of their
users, this is a welcome convenience — they don’t have to bother with complex installation and plugin management for features they want and need.
But for AWS, this is a massive risk factor. They have no choice but to provide similar functionality to their customers. By having all the code in the open, in the same repository, but with mixed licensing terms, Elastic NV creates a world where it is very, very difficult to collaborate only on the open source pieces. Any AWS engineer responsible for their authentication plugin, for example, risks being tainted by the proprietary versions in the upstream. How can they support their customers without the intimation that they are violating Elastic NV’s proprietary license? It’s a nightmare supply chain scenario — the root of the well is poisoned.
AWS has customers to support. They have revenue to protect. They built a service on top of an open source project, but not on top of a sustainable open source community. When it comes down to an argument on who gets to decide whats best for Elasticsearch, the decision maker is, and will always
be, Elastic NV. AWS might get a voice, or they might not, but if it harms Elastic NV, they have no reason to do it.
So what can AWS do? They can go straight back to the beginning of this blog
post, and take solace in the existence of the four freedoms. They can take the program, and change it so that it does what they wish. They can then redistribute copies, so that they can help others with the same desires (in this case, people who desire built in authentication, encryption, etc,
on a free-software stack.) Because Elasticsearch was Free Software, nothing Elastic NV does can ever completely poison the well. The software is an infinite resource.
That is precisely what they did. They built a distribution of Elasticsearch, open sourced their plugins for providing functionality covered under the proprietary license, and will use that distribution to manage their own AWS Elasticsearch service. In many ways, I believe AWS had no choice.
AWS called this new distribution “Open Distro for Elasticsearch”, which I’m sure they have every right to do. Whats troublesome here is that it opens the door for confusion about where the “real” Elasticsearch lives. The answer, of course, is that it belongs to Elastic NV. It always has. If AWS doesn’t like the rules of the game they set up, they have every right not to play it — but to me, they should be clear that it means creating a new playing field, not trying to assume they have a more fundamental right to what the future of Elasticsearch is than Elastic NV does. The value of the Elasticsearch brand should not belong to Amazon, no matter what they called their service.
Let me be 100% clear: this is not a failure of Open Source. This is the deepest, most fundamental truth about Open Source and Free Software in action. That you, as a user, have rights. That those rights are not contingent on the ability of someone else to capture value. That those rights extend to everyone, including AWS — or they don’t exist at all.
From the point of view of Elastic NV, this was deeply hostile. AWS, a larger vendor whose value proposition is that they run all your infrastructure, decided to:
1. Write open source versions of their proprietary functionality, and distribute them as open source plugins.
2. Create a distribution of their primary software, with those plugins included, to compete directly with their own.
3. Demand that they change the way they manage and build the software project, so that they can more easily do the previous two things.
I know, dear reader, what I would tell Amazon in this scenario if I was
Elastic: Get. The. Fuck. Out. Of. Town. With. This. Bullshit. If I may project
a moment, this is because, in my deepest dark heart, Elasticsearch has always existed for my primary benefit — that it was made available to AWS came from my largesse. I would feel differently if this was a sustainable community asset. But it isn’t… so, get the fuck out of here with that. If it’s AWS or Elastic NV, Elastic NV will choose Elastic NV every time. For the record: so will AWS. Capitalism in action, folks.
What can we learn from all this?
Be wary of building a business on top of other peoples open source software
that is not itself a part of a sustainable open source community. Without
a shared understanding of our core values, we cannot come to amicable
conclusions in the face of deep divisions. Especially when money comes into
play. AWS should make this a core criteria for any service they launch on top of a piece of open source software.
Tight Open Core is antithetical to the creation of sustainable open source
communities. The reason to adopt this model is to drive deeper, more coupled
monetization of a target market to a single benefit. When you see core
features, like authentication, being part of the proprietary stack — you know
that the software, and any community around it, drives solely to one parties
monetary benefit. Once that happens, we know the community cannot be sustained as it stands without that benefit being prioritized. It is no longer a sustainable community resource.
Reasonable people can all believe they are doing the right thing by the user.
I believe Elastic thinks that the upside to users in being able to use
proprietary features out of the box is critical to their users experience.
I believe that AWS thinks co-mingling those features, and making such
fundamental features proprietary, is hostile to their users. All depends on
which hand is in whose pocket.
Companies who decide to build their business on Open Source cores need to get much more aggressive about their trademark policies. It should be clear and unambiguous that your trademark cannot be used for another product without your permission. If I may go further, I would make it clear that nobody but your company can create a distribution with your trademark on it at all, without your permission.
The only winner in this whole scenario are the values of Free Software. It shows why it is important that software be open — because, unlike other kinds of resources, it can be infinitely available to us. Even if a sustainable community fails to form once, it can form again, under more equitable terms (and it is not at all clear that Amazon has created that community). The users can choose the software that best fits their needs. Or create it themselves.
