One Walmart Team’s Journey to Faster Delivery
Interested in finding out how one application development team decreased deployment times to production by 350%, saving 10 days per release? My team works on technical solutions for Walmart home office associates. In November 2019, we stood up a small team and started development on a new internal Walmart app. Since it was a proof of concept, we had a ‘just get something working’ mentality to prove out features and get buy in from the larger team. We were focused heavily on feature development and not as much on automation. We pushed the first app version to production manually and continued to create manual builds and deployments for the following four months. The team started to feel the pain of how long it took to deploy a new version of our app into production. It was taking multiple days, but we didn’t have a clear understanding of just how many; we had to establish a baseline in order to improve the process.. For us, a value stream map made the most sense, as it visualizes the entire flow, it highlights any bottlenecks that slow down the process. We mapped out the build and deploy process which very quickly provided us with a baseline of the steps involved and how long each took (Figure 1). All build and deploy steps were manual and the total time to bring the app into production was 10 days and 3 hours. We had a tangible number that we could measure our improvement efforts against.
For the first several months of development, we deployed our app to the cert environment prior to production but we experienced many challenges. From the value stream map, we could see it took 2–5 days for our internal mobile device manager (MDM) team to manually deploy the update to cert. We also had a technical limitation that restricted our device pool and the diversity of testers that had mobile test devices configured to the test with. Typically, the development team was the only test group who had devices that we could use for testing. For these reasons, we made a decision to replace the cert deployment and instead use a platform that allowed us to share a URL link with anyone on Android or iOS. The testers would use that link and download the latest version and it instantly downloaded to their device. That opened the door for us to expand our UAT (user acceptance test) and exploratory testing since we could send the link to anyone on any device. That was a huge time saver, increased our number of test devices, and expanded the number of people who test our app updates. We added the pre-production deployment to the CICD pipeline and had a button that sent a new app update when we had new builds ready for testing. With an average of two releases per week, and a three-and-a-half-day cert deployment waiting period, we’ve saved nearly 28 days per month of time waiting on the releases to go to cert. Instead, we spend two minutes waiting on the CICD beta test deployment and the new release is ready for testing immediately(Figure 2). Reducing 28 days down to 16 minutes has made a monumental impact to our team when it comes to beta testing.
After improving the pre-production testing timelines, we were confident we could reduce the current deploy-to-production timeline, of 2–5 days, as well. We partnered with the internal mobile device manager team to make improvements. Their team had a rough draft of a pipeline that was about 80% complete for iOS. We took a deep dive with their tech team and established a level of trust that allowed us to help them build, test, and prove out the iOS pipeline in just 6 weeks.
Today, iOS deployments are fully automated with a manual post-deployment validation in production. iOS builds are sent to production in 20 minutes which is a drastic improvement from the previous timeline of 10 days and 3 hours. Our MDM team has an ongoing effort to develop an automated Android release pipeline as well which we’ll partner together to test and implement.
What are the key takeaways from the past year? We don’t see improvement as a standalone effort that takes weeks or months and that’s the key to seeing positive change over time. We’re doing better than when when we kicked off development but we’re not yet where we want to be. We must keep continuous improvement top of mind, keep development tasks small, bake in improvements into each development cycle and constantly learn new ways to improve our app and our code. Incremental changes add up to big results and we’ve seen the benefits firsthand since the time we invested in CICD.
