Open Source Deployments on z/OS

🖥🖥🖥

I”m going

Open Source Software (OSS) is the means by which most modern software is developed, and it’s the only environment that today’s generation of software developers have ever known. As we continue to modernize, simplify, and enhance workloads on z/OS, it’s unavoidable that our platform must evolve to accommodate open source software.

OSS generates a lot of discussion about how things ought to be done on our legendary, largely closed platform. Unlike Linux, which is open source by definition, we sometimes struggle to agree about what the OSS footprint should look like on z/OS. Most of us are all too aware of ongoing debates about using UNIX System Services, which is now more than 30 years old. It’s worth noting that when we implemented UNIX System Services on what was then OS/390, we were at the forefront of open software, helping to define what common standards like POSIX and XPG4 even meant.

Given all the different debatable points about the shape of OSS on z/OS, there is one topic that everyone generally agrees upon — the security of the platform and the workloads it supports is essential. Starting from this common ground, there are still a number of directions we can take.

Stability Shouldn’t be Mistaken for Security

Many administrators look at their mission-critical mainframe environment like a fortress — create a working environment and isolate it from the outside world to prevent attacks. This does not address the problem of latent vulnerabilities emerging over time. Even if you have a protected environment in place with all the best maintenance practices, the number and variety of attacks and attackers has grown enormously over the last several years. Bugs and weaknesses that might have gone un-noticed are now often discovered and exploited by threat actors.

The Log4J/Log4Shell attack from December of 2021 was caused by the version 2 implementation of the Apache Log4J Java library. The vulnerability had existed for 16 point releases before it was widely exploited. Log4J provides common logging functions for Java applications, and the version 2 release implemented a means to log to a remote server. This allowed administrators to gather logs from several systems to a common location for review and analysis. However, it also enabled remote code execution from unauthorized users who deployed it as an attack vector.

The Apache Log4J V2 vulnerability caused a watershed moment for the software industry, prompting new requirements from the US Federal government, establishing new open security organizations, and driving commercial product providers, including IBM, to drastically overhaul their security vetting processes.

The first report, or CVE (CVE-2021–44228), for this problem was issued on December 10, 2021. Up to this point, any system administrator responsible for a running an enterprise server may have considered their platform stable and secure, and yet, it was still vulnerable to a problem of maximum severity — 10 out of 10.

Rapid Detection and Remediation

Today’s workloads — proprietary or open, involve complex and highly nested dependencies that can amplify a vulnerability’s reach. This is how a flaw in a foundational and pervasive logging mechanism like Log4J gets propagated throughout an entire software ecosystem. It’s not enough to detect that a given piece of code has a vulnerability, one must also understand where all references to that code are made across workloads and products. This is where the software industry has made substantial progress.

Today’s security frameworks are tightly integrated with source code managers like Git, and content repository servers that provide global control of the software used by an enterprise. Frameworks commonly in use can detect when a given CVE is a threat, locate all application source code within the enterprise that’s directly and indirectly vulnerable, and open problem reports to the developers who own the code. It can also quarantine any built code to prevent administrators from inadvertently deploying severe vulnerabilities that can be exploited. These frameworks are usually fully automated, allowing enterprises to respond within hours of the first detection of a problem anywhere in the world by any user who reports a problem.

Such frameworks usually enable policy-driven actions that give security architects control over all platforms in the enterprise, provided those platforms interact with the framework at certain well-defined points. One of those integration points is the package management system used to

deploy and service software on a given platform. These package managers acquire software from the approved content server that’s usually internal to the enterprise. Vulnerable software packages can be withheld, and fixed versions deployed through a common process from a central location.

These security frameworks are extremely agile. As problems are resolved and new fix versions are provided by an open source community, they can be detected, acquired, and deployed in near real time to an enterprise. This deployment strategy is compatible with conventional staging and air-gapped architectures where needed.

How z/OS Participates

It’s important for z/OS to participate in these security architectures to stay current with the latest open source code versions that protect the enterprise from attack. This means acquiring code from trusted, vetted sources through industry standard interfaces rather than proprietary z/OS package management systems. Requests are sometimes made to provide a PTF for a given open source fix. The downsides here are the significant amount of time required to build deliver a PTF, and the inability to integrate that fix with broader enterprise software management infrastructure.

Modern Security Posture

The pace of change for the software industry has accelerated over the last several years, often driven by the need for rapid response to emerging threats. Scheduling service updates for fixed service windows may allow a threat actor to operate for days before a vulnerability is closed. While we have had the means to provide out-of-band service for severe problems on z/OS in the past, this is becoming more of the norm these days than the exception. We have to be capable of deploying updates continuously.

Enterprise-wide, cross-platform security management provides the safest environment for z/OS operation. We have to engage with the open source community to actively participate in, and influence this architecture, or risk becoming a point of weakness in enterprise defenses against attack.