KeyBank Gets Deployment Control and Speed with XebiaLabs and OpenShift
“Our DevOps journey began by incident,” said Tom Larrow, DevOps Automation Engineer at KeyBank, one of the largest banks in the U.S., with revenues of 5.8 billion. The “incident,” which Tom described in a recent presentation at the XebiaLabs Leadership Summit, was a major outage that took their systems down for almost a day. While it made for flashy news headlines, it was, according to Tom, “a major reputational risk.”
But there’s a silver lining. Inside KeyBank, the outage spawned an investigation into their systems, which revealed just how complex they were. One online banking login, for example, required over 190 network hops across two datacenters and a minimum of 6 seconds to get in.
That kind of complexity is a fact of life for a major financial organization, especially one that’s seen much of its growth come from acquisitions. Last year KeyBank acquired First Niagara, which dramatically increased the number of customers using KeyBank’s systems.
“When you acquire companies and plug their systems into yours, you increase the complexity of your system exponentially,” said Tom.
Because of the complexity of their deployments, along with a lack of standard configurations and automation, they could only release software four times per year. Each release was a big deal — all-hands-on-deck, Sunday at 2am, 60 people on a call. It was, according to Tom, like “mission control, with everybody giving a go, no-go for changes.” Because the cycle was so long, they would do things like rush to include new features in the current release or scrap them until the next one.
But a major project was on the horizon: Tom’s team was tasked with building a new online banking platform, which was scheduled for completion by summer of 2017. And when the CEO announced the First Niagara acquisition — and moved the deadline for the new online system up by several months — the pressure was really on. They had to accelerate their process.
EBOOK
The IT Manager’s Guide to Continuous Delivery
Continuous Delivery allows you to get new features and capabilities to market faster and more reliably. This ebook helps managers understand the principles behind Continuous Delivery, explains the transition to a Continuous Delivery organization, and gives practical advice on how to start benefiting from the dramatic improvements Continuous Delivery provides.
Modernizing the Delivery Process
From a platform perspective, KeyBank knew they needed to change how they did business, and that meant modernizing the software delivery process. They began with an overhaul of their old platform, adding Docker containers to build all their online banking components, Kubernetes for container orchestration, and Red Hat OpenShift due to its strong enterprise support and value-added features.
For Continuous Integration, the team installed Jenkins, which they used in conjunction with OpenShift. This approach was great for improving coding, increasing the number and speed of deployments, and fixing issues faster. It was so effective, it allowed them to deliver the new online banking system to their existing customers a year early. The process also helped them meet their deadline for bringing First Niagara users onto their online services, and to rapidly fix any problems with little to no disruption to their millions of new and existing customers. Based on this success, KeyBank wanted to expand their new process to their other applications. “We had to keep improving, Tom said. “That’s the whole point of DevOps.” But spreading the success from the online banking system to their other applications demanded more change.
Reaching the Limits of Jenkins
Tom and his team realized that, while Jenkins worked well for Continuous Integration, it had its limits. Jenkins lacked certain software controls, which led to some problems. For example, there were a few incidents where someone would create an extra job to do testing, and the test code found its way into production.
As a bank in a highly regulated industry, they also saw the need for more mature access and process controls. Maintaining proper segregation of duties and ensuring all aspects of delivery were fully auditable was essential. They knew they needed something more robust — a full enterprise tool. That’s where XebiaLabs came in.
Finding an Enterprise Tool
KeyBank chose XebiaLabs’ XL Deploy and XL Release products to help them get the control of their delivery process and scale to include their other applications.
While KeyBank still builds Docker images and pushes them into their registry, they now use XL Deploy to deploy into their lower environments. Before XL Deploy, they were pushing a Docker image that was generically tagged “Dev.” Now, each image includes tags such as a build number and unique identifier. From there, XL Deploy creates a deployment package. The deployment packages are available in XL Release so that the team can select and control exactly which version of code is in any environment at any time. They can also easily roll back to a prior version and capture an audit trail.
By adding XL Deploy and XL Release to their delivery platform, KeyBank has achieved their goal of combining speed and agility with fine-grained control over which software versions go through the pipeline.
To hear KeyBank’s DevOps transformation story firsthand and learn about Tom’s “DevOps Do’s and Don’t’s, visit our DevOps Leadership Summit page.
