So you want to tackle the mainframe?

Before being able to modernize the Medicare claims processing system, USDS engineers had to climb the steep curve of learning to work with, and on, the mainframe.

Published in

U.S. Digital Service

7 min readFeb 3, 2021

By: Shelby Switzer (they/them), Software Engineer

Medicare: you may have heard of it. This is the federal health insurance program for individuals 65 and older, as well as younger people with certain chronic conditions. As of January 2021, the program covered more than 61 million Americans (19% of the total US population), making it the largest single insurer in the country. Every day Medicare pays out an average of $1 billion for inpatient and outpatient services. In other words, Medicare moves about 2 to 3% of the United States’ GDP every year.

Because of Medicare’s size, it is positioned to drive significant change in the healthcare industry. If you’re someone who wants to tackle the hardest problems or the most impactful ones, the Centers for Medicare and Medicaid Services (CMS) is the place to be. One of the projects our CMS team is tacking is a multi-year project to modernize the Medicare claims processing system. The age, complexity, and criticality of both the technology of this system and the healthcare system itself present unique challenges that require creativity and tenacity.

For many of the USDS engineers who work at CMS to modernize this system, it has also required climbing a steep learning curve: we’ve had to learn to work with, and on, the mainframe.

What is the mainframe?

As one of our colleagues at CMS likes to say, you can think of the mainframe as an operating system where all the code and data come from another planet, and that planet is the 1960s. As headlines during the pandemic have revealed, mainframe servers still power many levels of government, large corporations, and universities. Some of these organizations not only maintain mainframes but also still deploy new mainframes and build new apps on top of them.

Mainframes have two processing modes: batch and online. You can think of batch processing as highly efficient bulk data processing, where the workload scales with time and the size of the workload is measured in records processed. The online mode is the mode for user interaction: this mode powers applications and interactive, real-time transaction processing. With this mode, the workload scales with memory/CPU, and the size of the workload is measured in users.

Mainframes are mostly made by IBM, although others exist, and the programming languages they commonly support are COBOL and ASM, although you can run Java, C, and C++ on mainframes with compilers that must be installed and licensed. COBOL itself is an interesting and relatively simple language. As one of the first high-level languages, data is stored in “copybooks” and is written in sentences and paragraphs.

Like with many software systems, the programming language isn’t the hard part — although most of the engineers at USDS have never seen COBOL before, let alone programmed in it. It’s the systems themselves, which have been built and maintained and added on to for decades that are the hard part.

What is so hard about the mainframe?

1. It’s hard to learn.

Most software engineers in 2021 are totally new to the mainframe and to COBOL. Some folks on our team have used COBOL before, but that was over 15 or 20 years ago. Unfortunately, the learning resources are limited.

“I’ve been learning mainframes at CMS by trying to reverse engineer the Java code that has been deployed to the mainframes,” says Alberto, a USDS engineer who got so hooked on solving Medicare tech problems that he recently transitioned to be a full-time employee at CMS.

Other engineers have tried finding resources online to learn COBOL, without much luck. USDS engineer Jonny Eskew said the Coursera series on mainframe development, but didn’t find it helpful. “It was basically an IBM sales rep going over products you can add to the mainframe to make it better,” he said.

Another engineer on the team, Gina Maini, discovered the GNU Cobol community that helps generate documentation, however according to Gina “the community is smaller and has struggled with their own monumental modernization challenges, which tend to be a lack of portability in their monoliths that they are converting to Java or the proprietary data formats.”

The lack of learning resources led a former USDS engineer, Marianne Bellotti, to write a series of blog posts about it, which can help you get started with running a mainframe emulator and creating your first Hello World app. Even then, this is just a starting point.

2. COBOL is old and quirky.

The quirks of COBOL have tripped up engineers not used to it. For example, as part of the process of modernizing a claims pricing application to use Java, we discovered that COBOL math is fixed point while Java’s is floating point. In COBOL, numbers are rounded off between every compute statement, and you have to emulate that rounding in Java at every step to make sure the calculations come out the same.

Another “gotcha” on top of this is that in COBOL the precision constraint works on both sides of the decimal point. If you’re allowed a precision point of 3 digits, but then have a claim that goes up to $1,000, COBOL chops off the most significant digit and the value is left at 000. This obviously has huge implications for the calculation of how much a claim should be paid and our understanding of how to transfer this program to a more modern system.

At a higher level, when working with COBOL you may not be able to use the modern programming techniques you’re used to using elsewhere. Other relatively old languages like Java have had a lot added to it over the years to keep it modern. COBOL, on the other hand, hasn’t grown that much as language, and can’t be used in architectural styles it wasn’t built for, such as object-oriented programming.

COBOL also doesn’t lend itself to encapsulated reuse. There is no notion of importing a library into a COBOL application. Unless what you’re trying to do is a core feature of the language, you have to roll your own code, and copy and paste it around to where you want to use it.

On top of this, the mainframe itself is all file-based. If you want to get data in or out of the mainframe, you can’t use an API access token — you have to move files. This means that everyone is replicating the ETL process to get usable data — and one of our lofty goals is to eventually transition out of this to an API-first approach to reduce this duplication.

3. The mainframe is hard to test.

As we work to modernize the Medicare payment system and make it more modular, cloud-based, and API-first, we have to make sure we don’t break anything. This is incredibly difficult because of the absence of reusable executable testing. For over 50 years mainframe development has lacked not only testing tools but a testing culture.

Automated code tests that can be run locally or in a continuous integration pipeline simply don’t exist yet for the mainframe, which means testing changes is an arduous process. Virtually all testing of changes happens manually and takes up to 9 months. This in turn means that development and deployment of even small changes is a long and complex process.

What’s next for the mainframe?

If modernization is successful, the Medicare claims payment system will no longer be a monolithic black-box system on the mainframe but a cloud-hosted system implemented with best-in-class modern software development practices. The mainframe itself may become a big container for hosting environments, or it may become simply a set of databases with APIs providing access. Most importantly, the claims payment system will be ready to grow and support new payment models and innovation in policy such as value-based care.

This is a huge undertaking: USDSers have been neck-deep in the CMS mainframe for the past three years, and we expect to be working with our CMS colleagues on this problem for years to come. We work with CMS teams and contractors to understand, document, migrate, and sometimes even dramatically rearchitect these archaic and vital systems. It’s hard, interesting, and incredibly rewarding. Come join us.

The best of technology.
The best of government.
And we want you.
We’re looking for the most tenacious designers, software engineers, product managers, and more, who are committed to untangling, rewiring and redesigning critical government services. You’ll join a team of the most talented technologists from across the private sector and government.
If you have questions regarding employment with the U.S. Digital Service, please contact us at usds@omb.eop.gov and visit usds.gov/apply.