OPEN SOURCE CASE STUDY — How Walmart Scaled Hygieia to 5,000+ Dashboards

Learn How the World’s Largest Retailer Uses Hygieia to Monitor and Improve Their CI/CD Pipeline.

Case Study at a Glance

• Project: Hygieia
• Industry: Retail (Ecommerce, Grocery, Pharmacy)
• Company: Walmart
• Company Size: 2.3 million employees
• Hygieia Users: 5,000+

Background

Supporting ecommerce sites in 11 countries, a catalog of more than 1 million product SKUs, and a mobile app with on-demand features (like product scanning and prescription refills) meant Walmart needed a customizable dashboard to monitor DevOps across teams — and they found an enterprise solution in Hygieia.

“We were working to find ways to move to zero outage, on demand, and very frequent delivery,” says Bryan Finster, Technical Expert at Walmart Labs. “One of our top questions was how to give teams visibility to pipeline health to prevent bad changes from impacting production and our efforts towards improving our delivery processes.”

When Bryan and his team saw a demo of Hygieia at a conference, they knew it was the dashboard they’d been looking for — and that, because it was open source, they could alter it to fit their needs and test it with minimal effort instead of settling for an out-of-the-box solution and paying for upgrades.

Scaling to 5,000+ Dashboards

After initial small-scale tests by independent teams using available hardware (e.g., plugging Raspberry Pi units into spare monitors in scrum areas), they had both the data and buy-in to stand up a centralized instance of Hygieia supporting the entire enterprise, which meant scaling it to thousands of users across the organization.

To manage migration efficiently, they had to think beyond Slack channels and manual upgrades — automating the configuration process so new teams received onboarding emails and admin access automatically within hours of meeting their tooling minimums.

“Since we’re Walmart, we tend to stress tools,” Bryan jokes. “We have over 5,000 dashboards now with a continuing steep growth curve. Teams value Hygieia and challenge us all the time to improve it. We never have an empty backlog.”

Promoting Good Practices with Gamification

Part of that growth came from “gamification,” or scoring the use of Hygieia to encourage good practices across teams.

For example, teams need to use pull requests and integrate to master at least once per day per developer in order to earn five stars on Hygieia’s Source Code Management widget. This keeps the code manufacturing process visible to all levels of the organization and allows teams to see how they compare to other teams in their area and across the division.

“The response has been very positive,” Bryan says, “with teams either challenging each other or asking for help on increasing their scores.”

This visibility also enables them to have informed conversations about efficiency and value across teams — and update their scoring models in response.

Designing Dashboards and Support Teams

In addition to the automated onboarding process, Walmart also launched a configuration wizard to make it easier to create Hygieia dashboards and created a portfolio view showing aggregate scoring from the team level all the way to the CIO level — including data on security scans, dependency vulnerability scans, and team size and composition.

“This is all designed to help teams keep their repositories clean and to encourage the practices we need to maintain high quality,” says Bryan.

Walmart also plans to build a team dedicated to helping other teams with any constraints related to internal goals and challenges, further maximizing ease of use and minimizing the learning curve.

Results

While they’re still in the process of onboarding new teams and refining their scoring logic, Walmart has seen “concrete improvements as a result of using Hygieia,” and managers say it’s gratifying to see developers looking to their dashboards to monitor and improve the health of their pipelines.

And they’re pushing all of their modifications back to the Capital One Hygieia team for consideration, hoping to contribute closely to the pipeline of Hygieia moving forward and help other enterprises monitor the health of their DevOps more effectively.

Unless noted otherwise in this post, Capital One is not affiliated with, nor is it endorsed by, any of the companies mentioned. All trademarks and other intellectual property used or displayed are the ownership of their respective owners. This article is ©2018 Capital One.