Tips for Modernizing your CI/CD Pipeline

coauthored with Siva Kumaran and Kim Peelman

Many companies are taking on a hybrid architecture as they modernize their technology stacks. This enables them to leverage the power and security of the mainframe, with the agility and advancements of cloud technologies. To meet this demand, as mainframe experts, we must expand our expertise. This blog shares the start of our DevOps journey to modernize the CI/CD pipeline for the development of non-mainframe components for a product called Gen™. We will share our pains, gains, and lessons learned. It is our hope that articulating these insights will smooth your journey toward a modern hybrid technology stack.

Background

Gen™ is a single, integrated development environment for creating and maintaining large-scale business applications for both mainframe and distributed systems.

Gen’s customers include federal and local governments, Fortune and Global 500 corporations, and industries ranging from banking to transportation. These customers have generated billions of lines of code with their Gen models. The proven stability and scalability of Gen’s model-based development are why it has been in demand for over 30 years.

Our Gen team has a wide range of experience, averaging over 15 years on Gen alone. A team with so much experience has a lot going for them and can do even more when supported by modern DevOps tools. This type of investment in the foundation of Gen ensures the product’s sustainability long into the future.

Pain points

To start our journey to modernize the CI/CD pipeline, we identified our biggest pain points.

1. The manual build processing was time intensive, and any error required starting over from scratch. This also created a bottleneck, with a single subject matter expert.
2. The mandatory SAST scanning was an intensive manual effort and prevented continuous scanning of every build.
3. Testing was done in a waterfall model due to the absence of predictable successful builds. This limited us in recognizing and addressing issues rapidly and effectively, in our preferred agile method.
4. All these deficiencies slowed our time to market.

Actions taken and lessons learned

When faced with all these pain points, we felt a little overwhelmed with all the technical debt to overcome. Here are the actions we took and the lessons we learned along the way.

Step 1: Adopt a distributed version control system (VCS)

Our legacy centralized SCM was organized as an unbranched monorepo. We chose GitHub to allow for flexibility in our CI/CD pipeline. We struggled with how to split up our codebase. Gen had too many projects to make each its own Git repository, so we divided our codebase into a handful of larger families of functionality. This was good enough to get us started.

Tip: Focus on the goal. Don’t spend too much time over-analyzing the details.

Step 2: Create a branching strategy

Now that Gen was in Git, we needed to establish a branching strategy. We used a variation of the Git Flow workflow to guide us. We also blended it with our preexisting version control process. This turned out to be very helpful as engineers were having to transition from muscle memory habits to new tools and processes. Training helped, but true success came from only making small process changes. For example, we retained the wording “Base” and “Stage”, so the new branching strategy felt more familiar. There are plenty of blogs and articles about the best way to organize branches and when to do merges, etc., but the best solution is one that makes sense for the way your team works.

Tip: Start with a process close to what your engineers are used to. Improve incrementally. Don’t let culture shock crack the team.

Step 3: Fix what we broke

Splitting the repos had some unexpected consequences, so we had to reassess our next steps. For example, our build scripts referenced specific file paths. These broke in the transition to a new repo. We could backtrack on splitting up our codebase or keep them in place and fix our build scripts. The latter was further complicated by a choice between patching aging scripts or looking at more modern build tools. Ultimately, we chose the simpler approach and updated our existing scripts. However, we noted that the scripts were something we really would need to address later in our modernization effort. It helps to focus on easy wins first and put the hard items in the backlog. We found that as we came back to the backlog for what to do next, we always found another easy task to take on. This is because with every step we learned something new so what originally looked impossible started looking simple.

Tip: It’s okay to break things. Don’t aim for perfection on the first step. Start with what is simple, and address advanced things later.

Step 4: Put artifacts in an artifact repository

To reduce the manual labor of SAST scanning and shorten feedback time, we wanted to integrate with the scanning pipeline already used by our organization. To do this, our deliverables needed to reside in an artifact repository. We made modifications to the build script which would deploy artifacts to Artifactory. This had beneficial side effects that we hadn’t predicted. We increased build reliability and painlessly created production builds from any branch we wanted.

Tip: Put your build deliverables in a repo that will make them easily consumable by any system. It pays off in ways you sometimes can’t predict.

Step 5: Check in and re-prioritize

There are many more steps we took to modernize our CI/CD pipeline, and we still have work to do. We found that with this long of a journey, it’s easy to lose momentum and focus. This is why it’s important to regularly question what we are doing, why we are doing it, and the priority. At first, we thought a quarterly check-in would be sufficient, but even after only a couple of months, we lost momentum and sight of our end goal. We then assigned a dedicated “strike force” to help drive decisions. The “strike force” meets at least weekly, and the broader team meets at least bi-weekly. These meetings have clear agendas to review what barriers we hit, solutions we found, new things we learned, and new items to add to the backlog.

Tip: Keep momentum with frequent check-ins, reprioritizations, and a reminder of what problem you want to solve.

Benefits

Here are the benefits we have seen so far:

1. Increased automation reduces errors and manual labor. Builds can now be executed by anyone on the team. This speeds delivery and eliminates bottlenecks.
2. The execution of SAST scanning has been fully automated, eliminating hours of manual labor. This gives us the freedom to execute scanning on-demand any time we want, rather than only at the time of release. This shortens our feedback loops and increases quality.
3. We have moved closer toward our shift-left strategy. With a faster and more reliable build process, engineers have the confidence to deliver a build for early testing. Defects are addressed earlier, requirements are clarified, and demos are more frequent.
4. Development practices have been strengthened with increased collaboration, recurring peer reviews, and test-driven design.

Conclusion

Technology is ever-changing. Adopting modern DevOps practices and expanding our expertise to support hybrid architectures, ensures the long-term sustainability of a product and speeds the delivery of enhancements. We hope these tips help you on your journey. The benefits will be greater than you imagined!