5 Observations From Adopting Agile

What We Learned at Pivotal Greenplum

Ivan Novick (on right) and his colleague, Ed Espino.

While most enterprises are pretty conservative, more and more of them are encouraging teams to adopt Agile development and extreme programming methodologies in order to spur innovation.

That’s because they understand the importance of treating software as a product (not a project) to continuously improve the customer experience and deliver new, often game-changing services. It’s a message we at Pivotal have been preaching since our inception just over three years ago.

As anyone who has worked with us knows, we practice what we preach–and teach — around Agile. But even for Pivotal, a company which brought together platform, microservices, big data and Agile domain experts, teams had to adapt and learn along the way.

Over the past couple years, the Pivotal Greenplum engineering group has steadily transitioned from a more traditional software development approach to Agile practices that stress shipping code early and often. The goal has been the same for us as it is for our customers — to increase the speed of innovation and get valuable new Greenplum features into the hands of users significantly faster than previously possible.

Agile is still, for many enterprise software vendors, a fairly significant departure from the way they develop and ship product. Enterprise software products typically get updated with new features once or twice a year. That means users wait months — sometimes a year or more — for valuable new features. In contrast, with Agile and extreme programming methodologies in place, the Pivotal Greenplum team releases new features, on average, once every 25 days.

That’s close to 15 software releases per year compared to one or two for more conventional release cycles.

The result is ongoing, steady improvements in functionality that enable Pivotal Greenplum customers themselves to innovate faster than their competition.

The Greenplum team learned from the best: our colleagues in Pivotal Labs. Before it became part of Pivotal in 2013, Labs played a central role in pioneering Agile development and extreme programming practices over the last twenty years, working with some of the world’s most disruptive software titans along the way. Since then, Labs has worked with hundreds of large enterprises, not just building applications using Agile methodology, but teaching enterprise developers how to implement Agile development practices on their own.

I spent some time last week chatting with Ivan Novick, a Pivotal Greenplum product manager with close to 15 years of experience in the enterprise software industry — as both as a provider and customer — about the impact of Agile on his team. I thought sharing some of his best practices could help others who are thinking about how they can apply Agile methodology in their enterprises. The full video of our conversation is embedded at the end of the post, but here are the key principles behind the successful adoption of Agile at Pivotal Greenplum.

• Small, focused teams. The Pivotal Greenplum engineering group is organized into small teams of no more than eight people. Each team is dedicated to a single component or feature, allowing it to stay focused, and to work independently from other teams. This way, if one team is struggling for whatever reason, it won’t hold back other teams’ progress. There’s also a practical reason for focused teams — it “minimizes the conflicts between people checking into the same directories, the same files,” said Ivan.

Pair programming in action.

• Pair programming. Each focused team practices pair programming in which two engineers/developers work together at one computer. Each member of the pair brings his or her own experience to the project, and that diversity of viewpoints results in more creative problem solving. It also enhances the knowledge transfer among team members, Ivan said. “We avoid single points of knowledge,” he added. “So if you have eight people in a team and one person is an expert in something and they pair with another person, and the next day they pair with a different person, at the end of a short period of time all eight people know that component.”
• Weekly sprints. Some software teams that use Agile operate in two-week, three-week, or even longer sprints. The Pivotal Greenplum team does one-week sprints, which is where the ‘extreme’ in extreme programming comes into play. “If you’re not used to it, you might think, ‘Hey, how can you get anything done in a one week sprint?’,” said Ivan. “The answer is, it’s actually much more efficient and it allows us to change course faster, and we’re able to break stuff down into smaller pieces.” It also forces teams to constantly strive to be more efficient and find more productive ways to develop their piece of the code, he said. The one week sprint approach wasn’t an original idea, Ivan admits. “We completely copied the process from Pivotal Labs.”
• Automated QA testing. Prior to adopting Agile, Pivotal Greenplum QA cycles took as long as six weeks to release a new feature. While the team had automated much of the testing already, the remaining 25% of manual tests proved to be a major bottleneck to faster releases. As part of the transition to Agile, the Greenplum team completely automated its QA testing and “invests in test suites as part of the product code,” Ivan said. In other words, QA testing is just as important as the product itself from a development perspective. Further, Greenplum QA testing now runs in virtualized environments, meaning tests can be run in parallel rather than linearly. As a result, QA testing now takes as little as 24 hours instead of six weeks.
• Release every 25 days. With QA testing down to just 24 hours, the Greenplum team could theoretically release new code every day. But most enterprises, even those who themselves have adopted Agile development practices, aren’t looking for daily updates to their infrastructure software. After analyzing user feedback, the Greenplum team determined that about three-and-a-half weeks was the right release cadence. “What we found is that 25 days is roughly the time period that will basically not scare and freak out users that a release is coming too fast, but quick enough that at any given point in time, no one has to wait too long if there’s something critical or important coming,” said Ivan. “25 days is the sweet spot.”

That last point is really important.

Agile development and extreme programming methodology enable developers to ship code early and often, but you should never lose sight sight of the people consuming the code and their needs.

Speed for speed’s sake isn’t the goal here. The goal is velocity — or developing a steady drumbeat of feature releases that help users do their jobs better, faster, and more efficiently. If that is the end result, you know you’re getting the hang of it.

Of course, every software group is different and there is no one-size-fits-all approach to Agile development. But hopefully Ivan’s insights on how the process is playing out at Pivotal Greenplum gives you some ideas to consider as your own enterprise make the transition.

Change is the only constant, so individuals, institutions, and businesses must be Built to Adapt. At Pivotal, we believe change should be expected, embraced and incorporated continuously through development and innovation, because good software is never finished.