If you don’t understand its purpose, you can’t improve the process.
There’s been a lot of discussion over the last week regarding CentOS, Stream, rebuilds, and Red Hat. One of the aspects common to all of the conversations I’ve seen has been the blanket assertion that CentOS Stream isn’t a viable platform because it isn’t the same thing that CentOS was. That claim is made without an explanation, as if it is self-evident. There are a lot of misconceptions about what Stream is (not to mention what CentOS was), but I want to set those aside for the moment and talk about a different problem that affects how we talk about change.
In 1976, Rear Admiral Grace Hopper called the phrase “We’ve always done it that way” the most dangerous phrase we can use. That theme remains a constant in technical conferences today, nearly 50 years later. Every improvement we try to make as an industry and as a community has to push against the same “we’ve always done it that way” barrier. It is a natural mistake — but a mistake nonetheless — to believe that the process we employ is the best process available, and the fact that we use it is proof that it is ideal. The more successful a process is, the longer we will employ it. And the longer we repeat a process, the less likely we are to remember the historical context in which that process was developed, what problems it was designed to solve, and what compromises were made among opposing pressures.
All of those details are left out of discussions of CentOS and Stream (and the process of releasing software in each) when participants argue that Stream “isn’t the same” or generally isn’t viable. It is true that Stream is not literally the same process as CentOS; no process improvement is ever literally the same as the process it’s improving.
In order to understand CentOS and Stream, we need to start with a review of what problems are solved by the release model used for Red Hat Enterprise Linux — we need to understand the purpose of different elements of the process. The following list isn’t intended to be complete, nor is it provided by Red Hat; it’s merely intended to be an illustration of the types of opposing pressures that can lead to a release model like RHEL’s.
- Enterprise customers want to receive security fixes very quickly for legal and policy compliance reasons, as well as fixes for critical bugs that affect their production environments.
- However, some customers prefer to minimize the number of patches they receive, especially if their testing is conducted manually.
- Enterprise customers want to receive updates on a regular schedule so that they can schedule their testing and release schedules in advance. Planning ahead helps these customers maintain their development velocity, especially when they are coordinating workflows that involve many groups in a large organization.
- Enterprise and government customers may need software certified to meet various standards for legal and policy reasons. Certification is an expensive and time consuming process, and it applies to specific versions of certified components, so there’s a strong pressure to provide a long life cycle for those components to maximize the value of each certification.
- Third party software developers want a feature-stable build platform so that the dependencies linked to their application are consistent from build to build. The feature-stable package set also helps them express a minimum and maximum expected compatible versions to their customers.
- Maintaining multiple releases simultaneously means proportionally greater engineering resources required from Red Hat to back-port, test, and release security and critical bug fix patches to each active release.
- The process of preparing a release may be time consuming for Red Hat’s staff who may have to prepare release notes and update documentation for the product, compose new installation media, perform QA on the composed media, publish the whole update for early access testers, etc. The more frequent releases are made, the more time their engineers spend doing release preparation rather than improving software.
These competing pressures result in a release model that provides major versions that now occur every 3 years. While each release appears to be a single version, they are actually made up of a sequence of minor versions that occur every 6 months.
Some minor versions have extended life cycles of two or four years, depending on a customer’s support contract (and 5 years for the last minor release in the major). Not all minor releases get this support, which helps Red Hat keep the number of actively supported releases from growing too quickly. The six month release cadence allows Red Hat to publish critical fixes quickly, while reducing the update volume for customers that want extended support periods for a stable feature set. It balances customers’ desire for general bug fixes at an acceptable pace with the work that Red Hat has to do in order to prepare and ship them. It also helps more mature organizations plan their testing and validation of new feature sets in advance, which aids their general work planning. Maintaining multiple releases in parallel helps that group of customers remain patched for security issues while they validate new feature sets. Similarly, the cadence for major releases balances the concerns of shipping major new platform improvements against the number of active releases that Red Hat can support. The feature-stable extended life cycle of minor branches allows customers who need standards-certified software maintain consistent environments for several years. It also provides third-party software vendors with a build platform whose dependency set is consistent over the life cycle of the minor release. Those software vendors get a specific ABI guide from Red Hat that indicates how long supported libraries within the distribution will be stable, which they can use to project what versions of the distribution their software should be compatible with.
There are compromises visible throughout the model. Not all bugs can be fixed quickly due to the release cadence and update eligibility policy, nor can new features be released quickly to the users who need them. Not every minor release gets extended support. Not every library included in the distribution gets the same life cycle across releases.
Many aspects of that release model are mutually reinforcing, and don’t deliver the same benefits unless every piece is present.
When CentOS is described as a “1:1 rebuild of RHEL,” it is implied that CentOS supports the same business processes as RHEL, but that isn’t the case. CentOS was built on a different set of compromises, and never provided most of the benefits described above.
CentOS aimed for a similar release cadence (with varying levels of success), but Red Hat has only ever published the source packages for the newest minor release in a series. That means that CentOS only had to maintain one minor release within a major at a time, which reduced their overhead, but it also meant that they could not continue supporting a minor release while they prepared a new one. Their delivery of minor releases tended to lag weeks behind Red Hats, and because they stopped maintaining the previous minor release once they began work on a new one, users didn’t get any updates — including security updates — during that window. The lack of overlapping release branches led to a poor security posture for all users. Delivery of those minor releases was also unpredictable, so users couldn’t plan ahead for their testing and validation of new minor releases bringing new feature sets. Because the old minor wasn’t maintained, and security updates might depend on accepting the new major version, that validation needed to be scheduled very quickly as soon as it happened. Minor updates, then, were both time sensitive and unscheduled, which was disruptive to advanced planning. Red Hat’s software certifications were not transferable to CentOS, because certification isn’t an aspect of the code; it only applies to the specific binary modules that are certified. And even if users ignored the lack of transferable certifications, CentOS didn’t provide extended life cycles that allowed customers to run certified stacks for years at a time. And because the life cycles for minor releases were short, CentOS didn’t make a very good build platform; software vendors would need to choose to either run an unpatched old minor release, or to build on a patched system and tell their customers that the minimum platform they supported was the latest one.
If we’re charitable toward CentOS, it was a stable LTS that was usually closely compatible with RHEL’s runtime interfaces. Many users found it good enough to run software that had been built and tested on RHEL, and often chose to ignore the tardiness of security patches. These users typically weren’t large organizations that needed to coordinate workflows across dozens of groups, and ensure that they all met patching SLAs for their customers. They often had either no formal testing or highly automated testing, so they could accept updates quickly when they were provided. They didn’t have legal requirements for certification. Because their needs were different, they were able to use processes that they could support using a single update channel with a somewhat unpredictable release cadence.
However, some of the problems with that model, especially the security posture, needed to be solved. The process needed to be improved. Merely rebuilding RHEL source faster would be an improvement for the security problem, but it wouldn’t be an improvement that considered the purpose of a free variant of RHEL overall. Replacing the process of rebuilding portions of Red Hat’s minor releases with a process that continuously delivered a build of the RHEL major branch improves both CentOS and RHEL by expanding the number of purposes it can serve.
CentOS Stream is the result of automating the process of composing and testing a package repository and installation media from the major stable release branch of Red Hat’s source code repos for RHEL. Because minor releases don’t serve the purpose of extended support for certified components, or for consistent build environments for third party software vendors, or predictable delivery dates, minor versions are eliminated from the model entirely. This change allows many types of bug fixes which previously had to be queued for up to 6 months to be delivered to users immediately after testing and QA are complete, leading to more reliable systems. It also removes the process that created the longest delays for updates delivered in CentOS, leading to significantly more secure systems. Updates are delivered in much smaller sets to users, which makes them easier to test, and makes it easier to identify the source of incompatibilities if they arise. Early access to updates also allow third-party software vendors to test those changes to ensure that their software will work correctly when users upgrade their platform, and maximizes the time that they have to resolve any incompatibilities that arise either by working with the vendor or adjusting their applications. Furthermore, it creates a much more open development process for RHEL. It allows third-party distribution builders to build new distributions based on RHEL in a sustainable fashion, and provides them with the tools and infrastructure to collaborate with Red Hat. It allows third-party support consultants to directly propose improvements to the platform, which significantly improves the value that they can deliver to their customers. And it does all of that while preserving CentOS as a stable LTS that’s interface-compatible with the RHEL major version, with all of the testing and QA that made CentOS a usable platform in the past.
CentOS Stream does not provide all of the benefits of RHEL, but CentOS didn’t either. It isn’t a direct replacement for RHEL. But for most users — arguably all — it’s a significant improvement over CentOS, and its position in the Red Hat ecosystem is a significant improvement for all of Red Hat’s families. It isn’t the way we’ve always done it. It’s an improvement.
Grace Murray Hopper’s image was provided to WikiMedia by Lynn Gilbert, and is licensed under the Creative Commons Attribution-Share Alike 4.0 International license