My DevOps Journey at VMware
A glimpse into the impediments experienced while starting the DevOps transformation and how we solved them.
Most companies are going through a DevOps transformation these days to adopt faster releases, provide better quality, increase team’s flexibility, agility and get faster feedback. VMware’s mobile products are no different. We started our transformation two years ago with these same goals in mind. This post will list the obstacles faced on our journey, as well as the solutions that kept us moving forward.
While starting with DevOps goals in mind two years back, we had some initial ingredients available to us. We had an agile process in place (it is debatable how effective this process was), operations team in place, automation in place, tools and technologies available, but none of these were working as one engine. So, we started to make changes.
Rejection: Uncertainty, fear, and doubt in adopting changes is in human nature and this creates an environment of rejection. One of the most difficult tasks was to convince the team to implement new changes, transformation, and drive towards cultural shift.
To solve this, we started with constant education, process changes, and asked teams to make these changes slowly. This process has helped teams to see the value in DevOps adoption. Additionally, we were lucky to have the support of our management team. Without their support and alignment, our DevOps change would have been impossible.
Feature delivery: Another learning we experienced was feature delivery. Teams were under constant pressure to deliver new features with very little time in hand. They were focused on delivery of working code but were not responsible for quality. Quality was the responsibility of a separate quality engineering (QE) team. Developers were working day and night, delivering code over and above team’s capacity but QE were still finding lots of defects.
While making the adjustments in the team processes to adopt the changes, we did not want to make a big bang shift to disrupt the new feature releases which provide value to our customers.
So, what were our options? For the first step, we reduced the team’s velocity to only 50% capacity of their regular capacity, to ensure that the time needed for a higher quality focus was available to them. Second, we provided time for teams to re-approach and change the definition of “done”, we began to focus on root cause analysis, and focused on automation to avoid similar issues to reduce future issues.
Technical Debt: Technical debt is part and parcel of software development. Most teams with legacy code have some kind of technical debt and we were no different. Most of our mobile products have some amount of technical debt.
Since we slowed down our velocity to 50% of our previous capacity, we had some breathing room to take smaller chunks of technical debt as a part of planning within a team’s given velocity. Deciding to reduce our feature velocity by half is a difficult decision for any team to make. We worked very closely with product management teams to prioritize tackling technical debt and engineering improvements along with new features to ensure the success of our products in the long run.
Team Structure: When we started our DevOps transformation journey, QE teams operated independently of developers. It was the responsibility of quality engineers to test the product. But this arrangement does not fit well within the DevOps structure.
Management realized this problem and changed the team structure; QE teams were merged with development teams. Everyone was focused on delivering a quality product and quality was owned by everyone in the team. QE paired with developers, reported to same managers, and engaged in quality at the beginning as the design and development started. We created DevOps-style teams. DevOps-Style teams are full featured teams, able to build, test, have infrastructure skills, and manage services. If you are a large organization like VMware, you don’t need to create skills down to bare-metal for each team and can continue to have a platform team to serve for key infrastructure and common needs across multiple projects. However, ensure each team is enabled to manage applications and services by themselves.
Skills & Knowledge: It was easier said than done when management changed the structure of teams to make everyone responsible for product quality. There were gaps in skills and product knowledge. E.g. Developers did not know how to create test cases; The QE team was unaware about product code to contribute in code development.
To address this problem, we started to up skill and cross skill the team; To train QE on code development and developers to think about quality, test planning, test strategy, and overall adopting quality mindset. The purpose of this was not to have QE start developing code or developers start to test full time but get the skills needed to get more familiarity with the product and code. Additionally, we wanted to have the programming expertise to start building the tools and systems needed by their teams. Knowledge sharing, pair programming, cross team skill development eased the journey of transformation.
Automation: DevOps is all about early feedback in the entire SDLC lifecycle and automation plays a very important role to provide early and consistent feedback. DevOps evolution cannot be achieved without automation. When we started, automation was in place, however, it was not used effectively. Test results were ignored due to lack of trust in automation scripts, and developers were not authoring any test scripts for automation (except for a few unit tests).
What did we do? We looked back at our automation test strategy. For shifting left, we chose the tools and technology close to what developers were using, so they could write test scripts in their favorite language, IDE, and tools. This helped us in a big way, slowly developers were able to start contributing to write test scripts and quality engineers started to fix product defects. It also provided us the stability in test automation. Better design and architecture of our test frameworks made it possible that each team member was able to contribute towards quality goals.
Environment: One of the bottlenecks for automation was the availability of on-demand test environments, as Workspace ONE is a very complex product and has many inter-dependencies. It was not easy for individual teams to create such environments on demand for every test run.
This is where our infrastructure team stepped in and started to work on a project to provide on-demand test environments foreach team. The entire solution is based on a self-service portal and REST APIs; it was easy for all of us to adopt and use the APIs to integrate with automation and create test environments.
Cross team collaboration: As stated, Workspace ONE is a very complex product with hundreds of inter-dependencies and divided into hundreds of modules. Teams often worked in silos and were focused on their own deliverables without any consideration to sharing best practices or creating reusable code. This was not an easy problem to fix and required a cultural mind shift.
What was the solution? We formed a small team of people to start coordination among teams, align release cycles, reuse code across teams, and improve code documentation. One of the examples was recreating our test framework which was based on micro-service architecture so each team can share their own code scripts to avoid duplication. A cross platform team was formed to decouple and write reusable code which could be used across product lines.
Conclusion:
We were able to make many positive changes by applying the above solutions. Last year when we looked back and evaluated results for our mobile SDK, we found that we were able to release at 50% faster on iOS and 25% faster on Android. Unplanned patches, minor releases reduced by 58% on iOS and 29% on Android. We reduced our escalation counts, improved productivity, increased collaboration across teams and quality along with many other intangible benefits.
Don’t be afraid of the changes. As Martin Fowler once said
If you’re afraid to change something, it is clearly poorly designed.
Start your DevOps transformation with proper planning, ensure management is on your side and all stakeholders understand it is a journey and not a destination. Be mindful with some of the above impediments during your expedition, but enjoy and continue to learn.
I would like to extend my gratitude to William Pinner in co-authoring this article.
