CentOS Stream: Why It’s Awesome
Since the CentOS Stream announcement, I’ve been observing the discussions in the community and seen a lot of fear, misunderstanding, and even anger about the decision. During my nearly nine years at Facebook, I was a production engineer on the Operating Systems team, which ran what is likely one of the largest CentOS installations in the world. The team’s members believe in Open Source and are passionate about being active contributing community members of the OSS technology that they use. This experience provides me with a unique perspective on CentOS Stream; I believe it’ll be a huge win for the CentOS community, RHEL users, Red Hat, and for the Linux distribution community at large.
Some History
Let’s begin with a quick history refresher. CentOS began as a community project to pull down all the source RPMs from RHEL and build a usable distro out of them. The only changes made were to remove Red Hat’s branding, per the legal requirements. Other than the branding changes, CentOS Linux was as close to a carbon-copy of RHEL as possible; it didn’t carry fixes, changes, or patches of its own.
The project encountered some challenges familiar to those in Open Source: AWOL founders, budget shortfalls, and personnel issues. In particular, because the RHEL build system wasn’t public, CentOS had to invent their own, which required significant changes to accommodate each major release. Those changes required development time that wasn’t always available from volunteers. Leading up to RHEL4 and RHEL5, there were often questions on if the CentOS versions would happen or not and calls for help on the mailing list. Since CentOS didn’t sell a product or service, it relied on volunteers and donations like many other community projects. Proclamations of CentOS’s death were made by onlookers many times over the years.
Eventually Red Hat acquired the CentOS project for a variety of reasons, which, among other things, provided critically-needed resources. After what was apparently much internal discussion about how best to house CentOS, Red Hat decided to effectively firewall it off so that CentOS had no “special” access to RHEL that other downstreams couldn’t get. This often meant CentOS was still having to reinvent wheels that RHEL had already invented. Further, it meant CentOS stayed as a “carbon copy” of RHEL.
A Big Problem
While the Red Hat acquisition kept CentOS funded and staffed, it didn’t solve many of the existing, underlying problems facing the project. Let’s look at a specific example illustrating one of CentOS’s core problems: attempting to fix a regression.
Between 7.4 and 7.5, RHEL rebased their version of nmap and brought in a stray regression. After some debugging, I found that regression was a single errant character, so I sent the one-character PR to upstream nmap. The change was straightforward enough that it was merged the same day. So far, so good! But then I hit the real challenge: there was no way to get the fix into CentOS. Remember from our history lesson that the only code in CentOS was released RHEL code. In order for the fix to get in CentOS, I had to get it in RHEL. I had two options: I could either wait for the next major RHEL release (and hope it hadn’t already been forked from a version of Fedora built prior to my fix) or try to get it in this RHEL release. Waiting wasn’t an option for me, so I filed a bug and requested the one-character patch be pulled in for RHEL’s next point release.
Nine weeks later, the bug didn’t have any updates indicating if my fix would be included in the next release so I asked about the status. The package maintainer responded and suggested I create a support ticket. The RHEL nmap maintainer is an active participant in upstream development (and in fact submitted an upstream fix for a related bug that my PR was linked to). However, the role of RHEL gatekeeper necessitates using different evaluation criteria than that of an upstream developer. When I reiterated that I was solving a problem for RHEL customers, I was then told the fix would not be making it into the next point release.
At this point I started reaching out to contacts within Red Hat trying to see who might be able to squeeze this tiny thing between the cracks without drawing too much attention to themselves and getting asked too many questions by gatekeepers. Five months later the patch was pulled in, then another 6 months after that — one year after original filing — the bug was updated to confirm the fix would be in 7.7, which was about to drop.
And to be clear, it’s not the maintainers fault. RHEL was expressly built to not accept changes that didn’t have significant customer needs behind them. There were processes to follow to get changes in and those processes were more focused on the business and customer-driven justification for the cost of importing, testing, releasing, and carrying a particular change than a technical bar of correctness, lacking regressions or maintaining compatibility.
In other words, it was very, very, very difficult to fix a regression that found its way into RHEL (and its downstream CentOS) with a one-character fix.
Again, I could build that package myself, but not only would I have to maintain it for the life of that RHEL major version (which can be years), so would every other CentOS user who needed the fix.
But there’s an important flip side to this: RHEL, and by extension the RHEL customers, could not benefit from the experience of the wider RHEL+CentOS user base. Consider the bugs and fixes found by CentOS users big and small but never shared for lack of a venue to do so. Instead of those users forming a community that could contribute via filing bugs, sending PRs, fixing docs, helping debug/repro issues, creating tests, etc., they were just silent consumers.
The end result of all of this was that there was no ground under which a community could grow around either CentOS or RHEL.
What’s Really Changing?
The Twitterverse is all a-flutter with gross misrepresentations of what is changing. The core change is simple:
This means if you thought RHEL was stable enough before, then CentOS Stream is stable enough now. The testing framework RHEL used then is what both RHEL and CentOS Stream will use going forward. However, swapping of two elements in the chain doesn’t adequately show the positive impacts. Let’s look at more as a timeline. This is how it’s been until now.
The CentOS boxes here are dashed lines because it’s a straight rebuild with no “way in.”
Remember that one-character fix we were discussing? Had it been in RHEL7 and made it into, say Fedora 22 (a reasonable way into the RHEL7 lifecycle), without a lot of effort, it likely would not have made it in until RHEL8, which was based on F28! That’s a long wait for a one-character fix!
Let’s see what it looks like now.
This Is Awesome!
There are four elements of note above. The first is that the same testing and verification steps happen for both RHEL and CentOS Stream. The second is that you can make a PR and it will go in both RHEL and CentOS! You can now contribute directly to RHEL! As open source users, don’t we want software with more contributors behind it? Ultimately this means RHEL and CentOS Stream can now build a community with all the associated benefits possible: better docs, more/earlier testing, quicker fixes/responses, more community support options, etc.
An important clarification: there are two bars contributions need to hit. First, the maintainer of that project within Red Hat needs to approve it — find it to be a good change, not buggy, matching style, all the normal requirements for OSS contributions. Second, it has to fit in the RHEL “space.” While this new process opens up the possibility for you to contribute something to RHEL and CentOS Stream that otherwise might have been excluded, it still must “belong” in RHEL. A new useful feature in dnf is probably going to make it in. Re-versioning a bunch of stuff because you prefer monotonic versions to semantic versions is probably not going to make it in. That said, for things that don’t fit the RHEL space, SIGs are often a great solution.
The third noteworthy element isn’t explicitly seen on the graphic: all of the packaging repos here are moving to Gitlab. Previously, contributing to distribution content meant looking at read-only repositories in git.centos.org and attaching patch files to email or bugzilla.. Now they will be in one place with a single, easy way to contribute where pull requests can directly affect RHEL and CentOS once they’re merged.
Finally, in addition to contributing code or spec improvements, you can also contribute tests to CentOS Stream today, and there’s work going on to store the tests in dist-git alongside the code! Worried about one of those pesky regressions that got into RHEL/CentOS 7.2? Maybe one of them didn’t affect Red Hat’s customers and so it didn’t get a test? You can send in your own test to make sure it doesn’t happen again!
If you’re interested in the technical details you’ll want to look into dist-git and its up-and-coming counterpart src-git — the systems they’re using to tie upstream sources with packaging sources — which make it all much easier to contribute to.
One final note: src and debuginfo packages will be easily available for CentOS Stream.
Concerns
While there’s been a lot of FUD around the CentOS Stream announcement, there are also valid questions and concerns that are worth addressing.
The first concern I’ve already covered quite a bit: stability. A lot of people saw CentOS becoming an upstream for RHEL and assumed that meant it was a place to stabilize a Fedora cut into a RHEL release à la Debian testing. That’s not true. Recall that CentOS Stream will now be where RHEL used to be — so unless you thought RHEL was unstable, there’s no reason to think CentOS Stream will be unstable. Moreover CentOS Stream will now be subject to the full gauntlet of tests that RHEL uses, meaning it’s better tested. And, as previously mentioned, you can now contribute tests if you have a specific risk to mitigate.
I’ve also heard worry around vendor support. Some vendors who support RHEL could be convinced to support CentOS with the argument that it’s just a rebuild of RHEL and “the same.” RHEL and CentOS Stream will still be nearly identical, but instead of CentOS trailing RHEL slightly, RHEL will trail CentOS Stream slightly. In fact, vendors who care about RHEL can better guarantee RHEL support by testing on CentOS Stream, thus ensuring RHEL point-releases work when they land rather than playing catch up. While I can’t predict the choices vendors will make, I suspect the vendors who explicitly supported CentOS won’t drop it, those that mostly turned a blind eye to CentOS will continue to do so, and at least some who religiously only supported RHEL will add CentOS Stream support.
Some conspiracy theories have even developed implying that Red Hat is trying to kill CentOS or doing this in spite of what the community wants. It’s worth pointing out that both Facebook and Twitter advocated for this change as it not only makes it easier to contribute, but also easier to share their work with others! Full disclosure: I was a member of the Facebook team that supported this early on.
Finally, I’ve heard concerns around the “rolling” nature of CentOS Stream. First, let’s put that in perspective: updates for CentOS Stream aren’t exactly daily occurrences. As it stands they happen about 2–3 times a month. This isn’t any different from Red Hat adding packages to the ‘updates’ repo as before, the only difference is they’re all in one repo now. This means you no longer have to deal with the song-and-dance when all the updates get dumped into ‘base’ and a fresh ‘updates’ directory is created. For most people this will not look any different than CentOS 7 or CentOS 8.
In fact, haven’t we all been touting the virtues of smaller more regular releases of our own deployments for years now? We talk about the benefits — fewer things to break, easier to know what changed, etc. Isn’t it time we apply that to distros?
In the event you absolutely cannot have any packages be added to the repo outside of when you’re ready (CentOS 7 and 8 were already violating that) you can always mirror locally and snapshot the repos and update them every month or two or whatever works for you. Pulp, mrepo, and Foreman all make this easy, but even an rsync to an Apache server will work.
Let’s Recap
So, one more time:
- People can now contribute to RHEL and CentOS Stream adding value to both projects and solving real problems for users
- Sources will be all in a single place making it easier to contribute
- RHEL testing now runs on CentOS Stream, adding additional stability
- CentOS Stream receives smaller, more regular, well-tested updates
- src and debuginfo packages for CentOS Stream will be consistently available
- Red Hat and the CentOS project can now foster a healthy, diverse contributor community
- That community can leverage the work of all of its members from individuals to tech giants
- Red Hat and the CentOS project can now fully leverage each other’s experience in building an OS
Thanks to Rich Bowen, Davide Cavalca, Brian Exelbierd, Carl George, Jim Perrin, Brian Proffitt, and Brian Stinson for their fact checking an edits. Extra special thanks to editor extraordinaire Naomi Reeves.
