In favor of CentOS Stream
If I told you about a great new operating system, which released a new version roughly every six months and supported each release for 4.5–5 months before they went EOL, you might have questions. After all, any systems running such an operating system would be unsupported for 4–6 weeks, until they were upgraded to the next release and started getting patches again. But maybe there’s a use case for non-critical roles. If I then told you that this was an Enterprise-ready system that should be your production platform, you might look at me like I had two heads. Who would run a system that was unsupported, and whose vendor would not immediately provide security patches for 8–12 weeks out of the year? And yet, all of this was objectively true of CentOS for more than 15 years.
CentOS Stream fixes that problem, delivering a stable LTS with a continuous 5 year life cycle. It offers users a more reliable and more secure platform, while offering developers the ability to directly suggest appropriate changes for the platform. It maintains the RHEL compatibility that made CentOS desirable, in an open development process that was previously very closed.
In order to make sense of how CentOS Stream is being modernized, I think it’s important to define and explain some terms and concepts, especially for the benefit of readers who aren’t developers with a background in maintaining stable software releases.
Background
Enterprise Linux minor releases are branches of the major release
One of the things that differentiates Red Hat Enterprise Linux from most distribution release models is that a minor releases aren’t merely milestones along in a major release. Each minor release is actually a branch of the major release which will be feature-stable after its initial release, and which has its own independent life cycle. This result directly mirrors its implementation. In its source code, each component has an associated git repository with a branch for the major release, and in order to create a new minor release, Red Hat creates a new git branch off of that “main” branch.
Semantic versions provide a simple change taxonomy
Semantic versioning is one type of versioning system that allows a vendor to communicate to users what types of changes are included in a new version relative to the version before.
In semantic versioning, changes can be described as primarily three types. The simplest type of change might fix a bug or cover a new case in its input. This type of change doesn’t modify the software’s interface, and doesn’t generally affect compatibility with other software components. The second type of change adds new functionality, but is backward compatible. Other software components that worked with the previous version are expected to work with the new version as well. The third type of change is a change to the interfaces that is incompatible with previous versions. Other software components that worked with previous versions of the software may or may not work with the new release, depending on whether or not they use the interface that received a backward-incompatible change.
These three types of changes are encoded into the version used for releases of the software, which is expressed as numerically as MAJOR.MINOR.PATCH. If a new release has an increased MAJOR version, then users know that some interfaces have changed in incompatible ways and that other software might need to be updated in order to work with the new release. If the new release has an increased MINOR version, then users expect their other software components to continue working, but they also know that there are new features in the release. If the new release has an increased PATCH version, then users expect only fixes and not any new features.
One of several reasons that this system is helpful is that not only does it describe the types of changes that users should expect, it provides a rational system for third-party software developers to indicate a minimum and a maximum version of software required for software they interface with. If a third-party developer builds and tests an application on SuperOS 1.2, then they might require features that weren’t present in SuperOS 1.1, but the application might not be compatible with a future release of SuperOS 2.0.
What are git branches, and how do they work?
Software developers will (often) begin with a source code repository which provides several basic types of change control and history. One of the basic mechanisms that exist to help developers work independently as they collaborate is “branches”. Branches allow a single repository to reproduce multiple different tracks of development that diverge and and converge as developers work independently, and together.
In git, the default branch is “main”. One common workflow for git is that developers can start from the main branch and create a new temporary branch where they can work on a specific change (e.g. a new feature or a bug fix). In their branch, changes that other developers make to the main branch will not appear, and when they are done with their work they will “merge” their changes back into the main branch. In a single step, all of their changes are added to that branch, and it will then contain the feature or bug fix that they developed.
Because they are isolated from the changes in other branches, not only can branches be used for short-lived versions of the source code repository, they can also be used versions of the repository that the developers want to maintain and support for a long time. At some point in time, perhaps the developers agree to release version “1.0” of their software, and in their repository they might create a branch for that. That branch will contain the exact state of the source code at the time that it was created, and won’t get any features or fixes that that developers add to the main branch.
This ability is recursive. Developers do not have to create new branches from “main” alone, they can also create new branches from a stable branch. This allows changes to progress outward from the “main” development branch into other branches that are used for supported versions.
For example, developers might create a “release-1” branch from main, and then branch “release-1.0” from that. Later, any bug fix changes would be added to main, and then merged to “release-1” from there, and then merged to “release-1.0” from there. Any backward-compatible new features would be added to main, and then merged to “release-1” from there, but not to “release-1.0” because it does not receive new features, only bug fixes.
This allows the developers to later create a new branch, “release-1.1” from “release-1”, and that branch will contain all of the new backward-compatible features that accumulated in “release-1” up to that point. As bug fixes are added to “main”, they can be merged into “release-1” from there, and then into both “release-1.0” and “release-1.1”, while new backward compatible features are only merged into “release-1”, just as before.
This practice allows a software vendor to simultaneously support multiple releases, including incompatible releases, for a variety of customers.
If Minor updates are backward-compatible, then why do previous minor releases need to continue to receive support?
Largely, this is a matter of contract terms and risk management. Some customers might want to limit the types of changes that are made to the software they use due to the perceived risk that the addition of features creates the risk of new bugs. They might ask for two years of updates to version 1.0 that don’t include any new features.
The same customers might have a validation process for moving forward from version 1.0 to version 1.1, and they want to ensure that they continue to receive bug and security fixes for the 1.0 release series while they perform that testing. Overlapping life cycles support the process of testing before deployment in these very risk-averse environments.
However, many environments (probably most environments) don’t strictly need concurrent minor releases, and simply adopt the practice of always updating to the newest release that doesn’t include any backward-incompatible changes. The extended, overlapping life cycle of previous minor release series isn’t a high priority for them.
An argument in favor of CentOS Stream
With that background, I think we can start to talk about a lot of the frequently asked questions that come up regarding CentOS Stream and the publication of RHEL source code.
What is CentOS Stream?
Referring back to the discussion of git branches earlier: Stream is a distribution built from the the git branch that is effectively “release-MAJOR” for the major release of RHEL. It is not the high-churn “main” branch where new feature development is done. It’s the branch where stable features and bug fixes are merged after extensive testing, and evaluation of their suitability for the enterprise customers who pay for RHEL. Everything merged into the “release-MAJOR” branch has already been fully vetted, and will appear in RHEL the next time a minor release is created by creating a new “release-MAJOR-MINOR” branch in all of the software components.
Where does the value of the enterprise release model come from?
Enterprise support contracts bring a wealth of benefits to customers, and I won’t try to enumerate all of them. But one of the things that Red Hat Enterprise Linux and SUSE Enterprise Linux have in common is that they apply the concept of semantic versions to the entire distribution. Doing so makes them a partner in their customers’ approach to risk management, and allows them to offer contracts to support a large coherent collection of software components for specific life cycles, even where the upstream vendors discontinue support during the life cycle of the Enterprise vendor’s product. Minor releases have overlapping support cycles, so that customers who need to remain on a feature-stable release can continue to receive security patches for all of those components until they’re ready to rebase their systems onto a new minor release with a new feature set. This differentiates them from most stable LTS distributions, where there is only one release channel for any major version, and where minor versions are merely informal milestones — if they exist at all.
Wasn’t CentOS also an Enterprise release?
Not at all. Not only did CentOS minor release life cycles not overlap, they didn’t even meet. Every time RHEL released a new minor version, the version of CentOS built from the previous minor version would immediately stop getting any kind of updates while the maintainers prepared a new release of CentOS. This involved a lengthy release engineering process with testing and QA that would continue for typically 4–6 weeks. Any security updates that were published to RHEL in the new minor release, or after the minor release would be delayed until the new minor release of CentOS was finally ready. And they had to be, because any security update might rely on new features that were introduced in the new minor release. (Remember that earlier I mentioned that minor versions provide a means for third party vendors to express a minimum version requirement for their software, which reflected the feature set present in the minor version. In this case, the same thing is true for the patches included in the product itself.)
The idea that CentOS was an Enterprise-ready product because it was rebuilt from the source code of an Enterprise-ready product is a dangerous myth.
Wasn’t CentOS bug for bug compatible?
It wasn’t that either, because Red Hat has never published information required for reproducible builds. While end-users often don’t appreciate that, the CentOS developers did. Ask the CentOS QA people, and they’ll tell you, “”We came up with the phrase “”bug-for-bug”” compatible during EL5 as a GOAL to aim for. CentOS was NEVER bug-for-bug compatible””
That might seem like a minor point, but I don’t think it is. Differences between CentOS and RHEL were rare, they impacted very few users when they existed, and they were resolved quickly when they were found. And because Red Hat’s testing includes automated review to ensure interface compatibility when components receive minor-version updates, there’s very little reason to believe that Stream will introduce compatibility issues.
It’s true that extremely risk-averse environments might want the support contract that includes extended support for feature-stable minor releases, but it’s also true that rebuilding RHEL doesn’t provide the level of risk reduction that those environments require, because they don’t have reproducible builds.
Is Stream a Beta or QA product?
No, it isn’t. Anything in Stream has already completed testing and QA and been accepted for inclusion in the next eligible RHEL minor release.
Q5: Does this mean that CentOS Stream is the RHEL BETA test platform now?
A: No.
There is a widespread belief that Stream will be less reliable than RHEL, but there is no coherent explanation for how that could happen. In the past, when a change was accepted and merged into the git branch for “release-MAJOR”, it was idle in that branch until the “release-MAJOR-MINOR” branch was created, at which point the change would be available to RHEL users who upgraded to the new minor release. Git commits do not age like wine. They don’t get better or more stable with time. Testing is done before the merge, so once an update is ready for “release-MAJOR”, it’s ready for most users. There just wasn’t a channel to publish it in RHEL’s model.
For most use cases, Stream represents a significant improvement, delivering many types of features and bug fixes much earlier than RHEL systems can receive them.
Can rebuilds use source rpms from the customer portal?
Probably not. Red Hat’s business agreement prohibits the use of the subscription for the purpose of redistributing the source. While there’s some speculation that the restriction conflicts with the GPL, that probably isn’t relevant because the git repos that derived distributions use to rebuild RHEL source code aren’t licensed under the GPL, and neither is most of the source code in the distribution. Even if it is true that the GPL protects the rights of customers against any additional terms imposed by Red Hat, you cannot build a derived distribution using only GPL code.
It might also be important to note that the terms which limit redistribution of content received through the subscription have been in place since the beginning of the Enterprise distribution.
The only thing that Red Hat doesn’t publish to the public through the Stream git repositories today is updates to packages that have been updated in the Stream git branches (the ones in “release-MAJOR” that are newer than in some “release-MAJOR-MINOR” branch.) This is not fundamentally different from the past, where Red Hat did not publish to the public any updates that were unique to packages in a minor branch other than the newest one.
Can rebuilds use the Stream git source?
This is the source of a lot of disagreement, but I will argue that “Yes,” the Stream git repos now available are totally sufficient to produce a derived distribution that is ABI/API compatible with RHEL. A derived distribution would mostly need to simply mirror the Stream git repositories, and branch them at the same time as RHEL. After that point, many bug fixes can simply be merged from the “release-MAJOR” that Red Hat provides via the Stream git repositories, except for bug fixes to the small number of packages that have critical bugs or security issues in an existing “release-MAJOR-MINOR” branch which have been updated to a new version in the Stream git branch. As long as downstream vendors don’t make any interface changes to the packages when they fix a bug or security issue, they’ll remain compatible with RHEL.
I won’t say that this work is entirely trivial, but professional developers should not struggle with that task, and I don’t think it’s entirely unreasonable to expect that vendors who sell long-term support for software should actually be engaged in providing some support for that software.
Can the companies rebuilding RHEL create supportable distributions?
While some people are upset that publishing only Stream’s git branch negatively affects downstream distribution vendors, I will tell you that from the perspective of a developer, it is much easier to build a real Enterprise system using Stream’s git repos than it was under the old model. Far from shutting down derived distributions, this move is enabling them.
Previously, the git repos that Red Hat provided weren’t even really their own dist git. They were completely separate repositories created by unpacking RHEL’s source rpms, removing the primary source from the working directory, committing whatever was left, and pushing the result to a public git forge. If you are not a developer, you might not appreciate just how fragile and weird that process is, but most developers will probably recognize this as an astoundingly weird Rube Goldberg pipeline that treats git like a fancy FTP service.
The new source code publication process is so much better. It’s the actual git branch published to a public git forge through the normal git channels built in to every git forge. I can’t believe there are people complaining about this.
Did Red Hat buy CentOS to kill it?
I think that’s a ridiculous idea. If Red Hat wanted to prevent derived distributions, they could simply not publish their dist-git. They’re not under any obligation, under any license, to publish the automation that they use to build source code provided by upstream source code vendors.
I cannot name a software development company more dedicated to open development than Red Hat. They have a long history of acquiring products which are not under a Free Software license and doing the work required to re-license those products for the community’s benefit.
If all of that is true, why are so many people insistent on using a rebuild of RHEL?
At this point, it becomes sadly necessary to define one more term: Cargo Cult.
A cargo cult adopts the practices they observe from successful organizations, without understanding the context in which those practices evolved, or what purpose they serve, or how the practices support and enable each other. As they adapt those practices to their own needs, the benefits they deliver to the original organization typically fail to materialize.
In our industry, we talk about cargo cults pretty often, because it is entirely possible (and common) to successfully do all of the things that one sees successful organizations doing, expecting to achieve the same result, and failing to get it. This is often true if you use the process that one groups requires in an organization where it isn’t required and can’t deliver any benefits. It’s also often true if you apply a development process (like distributing software with minor releases), but skip the part that makes that development process really valuable (like continuing to support an old branch so that the risk-averse customers that need feature-stable branches have time to test their services before upgrading to a new release.)
The idea that CentOS was an Enterprise-ready system was a myth that was unchallenged for far too long, creating a cargo cult, primarily among people who have never actually worked with an Enterprise release and have no basis for comparison.
That cargo cult now argues that CentOS Stream, which fixes major flaws in the old CentOS process, is unusable, despite the plain fact that it is a stable LTS, very similar to other stable LTS distributions that are broadly successful, and despite the fact that both the new model and the old one deliver fully vetted updates to users in a single major-release channel.
The myth is simply a barrier to progress. The new model provides a more reliable, more secure system for virtually all use cases that don’t require real, authentic RHEL and not a best-effort rebuild.
Is Red Hat killing derived distributions? I don’t think so. I think it’s much easier to build a derived distribution from Stream than it was in the past. What they’re killing, though, is the myth that it is necessary or sufficient to build components from RHEL, without any additional engineering, to sell a supportable production platform.
And it’s important for that myth to die in order for the distribution to modernize, because the Stream model is so much better for nearly all of the user community and the developer community than the CentOS model was.