Modern Software Delivery Maturity Matrix

Karyn is a Colorado Digital Service alum and one of its founding members. This post represents the personal views of the author and not their current or former employers.

In the summer of 2020, I was a few months into my tour of service with the first iteration of the Colorado Digital Service (I’ve since then rolled off). At the time barely half a year old, our team conducted a discovery sprint on the state’s child welfare case management system. As we reviewed documentation, conducted and synthesized dozens of interviews, we struggled for a way to codify and effectively convey our observations about the team’s practice maturity in areas like product management, understanding of end users, development cycles, and even team culture. Our insights made sense to us, but felt fragmented and untethered as we cobbled together a report intended for an audience with very mixed levels of expertise in modern software delivery practices.

We were serendipitously inspired by David Eaves and Ben McGuire’s Medium post proposing a Maturity Model for Digital Services, from their 2018 State of Digital Transformation Report. Their framework, which codified learnings into five high-level themes and ranked each from low/medium/high/future state, was a lightbulb for us.

Could this be an effective way to ground and contextualize both successes and shortcomings within a project ecosystem? If the team has a low maturity state in one dimension, the matrix can serve to show them ways to improve. If they have high maturity in other areas, those successes can be celebrated and reinforced, along with clarity on what needs to be prioritized.

We decided to flesh out the idea, and pretty quickly realized this framework could work. To round it out better for projects and product teams, we also pulled in insights from 18F (in particular the concept of a Quality Assurance Plan, or “QASP”) and from GDS’s service assessments as additional inspiration.

The dimensions we ultimately came up with were:

• Team culture
• Purchasing vs. procurement
• Modular contracting
• User centered approach
• Product ownership
• Agile software development
• DevSecOps
• Building with loosely coupled parts

For each of these eight dimensions, we outlined what practice maturity looks like when it is low, medium, or high according to best practices from high functioning product teams.

Let’s take team culture for example:

Low

• Lack of accountability
• Confusion over roles
• Lack of transparency
• Working in silos
• Culture is described with words such as fear, apathy, toxic
• Finger-pointing and scapegoating are common

Medium

• Silos may still exist
• Honest conversations are happening but still feel difficult or combative
• The team is willing to acknowledge dysfunctions and working to improve dynamics

High

• Cohesive team dynamic
• High levels of psychological safety and trust
• Team dynamics marked by collaboration and transparency, with effective retros spurring continuous team improvement
• Low levels of turnover/burnout

We did the same for the other seven more tactical dimensions, but chose to lead with culture because the human layer is the foundation that needs to be in place before a team can operate effectively. Remember, it’s almost never about the technology.

We were fortunate enough to get lots of input from the digital HKS and Public Digital community once we had a draft. What started as a shared Google Doc has now been moved to Github, and we hope the community will continue to help us improve it!

Crafting the maturity matrix helped us anchor our findings and effectively convey to Colorado Department of Human Services (CDHS) leadership that certain areas were working relatively well (e.g., product ownership), while other areas needed their attention. It resonated deeply with team members, some of whom felt empowered to say aloud what they had been feeling, and also provided leadership with a barometer for what better looks like.

Aaron Snow, the first CEO of the Canadian Digital Service, was so right when he remarked that teams like ours are “a change management team disguised as a digital services team.”

We thought our purpose was to evaluate the technology, but the deeper impact we had was almost certainly empowering the team to instigate their own change.

In addition to catalyzing team growth, another highlight of our engagement is certainly that the dimension of team culture really resonated with the agency team. Along with the concept of psychological safety, we shared an assessment to quantify and measure it over time. The team has adopted the practice and has committed to measuring it quarterly.

As of July 2021, our team is now nearly two years old and we have evolved our application of the maturity matrix in interesting ways. In the first half of 2021, our team completed a 13-week consulting engagement with the Colorado Department of Public Health and Environment (CDPHE), to help with a reorganization to better enable technology efforts in support of public health. As we were wrapping up our engagement and stitching insights together for next steps, an idea came up from one of the task force’s public health experts (who had seen our maturity matrix) to turn the attributes into a self-assessment. The idea was to ask folks to reflect on various aspects of their practice in order to underscore our assessment of how mature technology practices are at CDPHE, and where we can go from there.

We translated applicable dimensions of the maturity matrix into survey questions and sent the questionnaire out division-wide. Here are some examples of how we translated attributes from the matrix into questions:

Maturity matrix attribute: After work is complete, it must be re-done because people are not engaging with it as expected → Questions: Reflecting on technology projects you are a part of, how would you rate the following processes? Understanding who our end users are; regular engagement with our end users; ability to meet end user needs [Likert scale]

And here’s how we translated the team culture dimensions:

The evolution into a self-assessment was unexpected, and while it made a lot of sense to our team, the actual execution was met with mixed reception and was ultimately not successful. Despite our best efforts to avoid using tech jargon, many of the questions we asked made no sense to public health folks completing the questionnaire. As a result, many of them couldn’t even get through the form.

With reflection, we believe the self-assessment flopped in this particular case because CDPHE is not organized around teams that build and support technology. Rather, technology is seen as a support function in service of public health (and rightly so). However, because of this fundamental difference, there is no concept of “team” when it comes to talking about technology projects. Without a concept of team, this entire maturity matrix falls apart.

Even if the concept of a team exists, if practice maturity is low, it’s likely that folks would struggle to assess their own baseline. Over and over during our engagements at the state, we’ve seen struggles with unfamiliar language, with the concept of a “technology team,” and with what scope and boundaries to think about when answering such questions.

Whether a self-assessment derived from the maturity matrix can become a useful tool will likely vary from agency to agency. For teams that do exist around the support of a major technology project, the self-assessment would likely yield much more reliable results. We are looking for an opportunity to test out this hypothesis in the near future.

We’re excited to see how this framework might continue to evolve. I hope that as the founding members rotate off our still-young digital service team, the next cohort of folks will contribute to it, keep it alive, and find new ways to make it a useful tool in Colorado and beyond.

Have you used something similar with your team and/or agency partners? My former CDS colleagues and I are gathering stories to include in a collection of learnings for new digital service teams. We’d love to hear from you! Email us at digital-service-start-ups@googlegroups.com or leave a comment below if you have a similar story to share.