Why I Care About Software Process Improvement

A quick Google search will tell you that at least 25% of all software projects fail. Only 67% of projects are completed within budget, and a similar number of projects are completed on time. This makes me consider how mature the development process is at MANTA. What happens when we neglect the technical debt or simply stop evolving the development process? How can I raise the bar as an individual tester, developer, analyst, team leader, or project manager? How can we successfully suggest, implement, and adopt an improvement?

When I joined MANTA as a delivery manager, the current level of engineering was already mature enough to satisfy the requirements of our global customers. At that point, MANTA’s major version was already 24 releases — 24 times, the team delivered a working solution to the production.

My boss told me: “Everything kind of works here.” He was right in many aspects. The development process was settled and generally followed. Highly modular architecture allowed the team to work in parallel without unnecessary conflicts when pushing the code. The continuous integration (CI) system made the best use of the unit tests, static code analysis tools, precisely configured build configuration, and dependency management.

The team had automated major parts of the release process and employed various development tools to make their work more effective (validator of Maven POM configurations, OWASP vulnerability check in external libraries). An issue tracking system had been in place in connection with defined configuration management.

Having said all of that, my boss concluded: “…and it can be infinitely fine-tuned”. I was lucky enough to get this empowerment and trust to work on it. Long-term development projects and our product development environment allow us to observe the trends, measure, compare recurring moments, improve, and evaluate the impact of the changes introduced.

Software Development is More than Programming

Since then, the engineering department has grown and matured a lot and many experienced developers, testers, UX professionals, and DevOps engineers have enriched our 13 teams. But initially, the team was a diverse mixture of motivated young engineers with strong connections to technical universities. Thus, it was quite natural that the focus was strongly programming-centric, while other disciplines of software engineering were not getting the attention they deserved. We decided to balance this.

I know that saying “SW development is not just programming” sounds provocative, and that’s absolutely intentional. I am in no way diminishing the importance of implementation. I’m a software developer at heart, probably like most of you. But let’s have a look at the development as a whole, with a broader perspective. In my experience, the usual share of implementation effort is between 30–50% out of all commercial project activities, depending on the project size. The bigger the project, the less programming is usually involved and the more share is taken by project management, architecture, analysis, documentation, integrations, and testing.

With an implementation share of 35% at MANTA, the importance of the other slices of the cake becomes obvious. Of course, one can still deliver a top-notch product with an inferior production line, but having a top-notch production line increases the probability of producing much better output.

Major Risks of SW Projects

To support the above-mentioned claim, let’s look at the importance of a holistic view from another perspective. What represents the main problems in the development of SW systems?

A few years spent in the industry or a few minutes of Googling could probably compile a similar list of risky areas to be cautious of. Although implementation-related risks are very important, they are not the only risks and aren’t prevalent. The causes for failure can also be “political” (e.g., reorgs, budget cuts, competition taking advantage, and game-changing decisions by the department of architecture). For now, let’s stay with the engineering practices we can more or less influence.

Another perspective. My favorite first step after entering a new company or team is to ask developers about the biggest problems they’re facing on their projects. Let’s interview our fictitious developers Adam, Ben, and Caroline to see the common classes of answers I frequently get.

• Adam: “I don’t like the duplication in CSS. Oh, and the DoItAllServiceImpl class has 3,000 lines and is a total mess.” Alright, this is a bit too detailed of a problem. Let’s see if we can reveal more severe impediments.
• Ben: “Let’s plug in a new check-style rule. If correct code formatting is not observed, the build will fail.” Well, in cases when the release is at stake, this improvement actually makes things rather worse.
• Caroline: “What do you mean by design? Everything’s OK.” She doesn’t know what she doesn’t know. That’s probably the most serious case.

As we can see, typically little is said about analysis, documentation, testing, CI, releasing, organization of work, or the number of regression errors in production. If you ask the developers, you’ll get bits and pieces, but you’ll have to stitch together the big picture on your own to have a complete understanding of and identify the areas of improvement that could bring the most value.

Why Improve?

Why should we care about the development process that much? Because we have to succeed. Let me give an example regarding project management.

Source: The Knowledge Academy

At MANTA, project management is only one-tenth of the whole pie to eat. And even this small part can ruin 32% of all projects. That’s very alarming. Generally, projects with malformed or totally lacking a well-shaped development process, or projects with growing undetected technical debt, are sooner or later deemed to fail.

It’s even harder to spot those at first glance. Such projects just sort of keep working, until they get stuck slowly or fail fast. I always like to remind myself of this metaphorical whip behind my back. Luckily, there is always a carrot hanging in front of our noses making me strive for a better development process. Besides that, there are many pragmatic reasons to kickstart the improvements:

• I’ll get to know the project from the perspective of processes or architecture
• I can learn the good and bad of the libraries, frameworks, and tools
• I will get in touch with other people
• I will explore the dead ends. Hopefully, I will be able to show it can be done
• I’ll help my colleagues simplify their work or level up the results of their work
• I will make my boss happy and maybe even boost my career

“We’ve Always Been Doing it That Way Here”

Let’s imagine you finally rolled up your sleeves and presented your ideas to the team. Suddenly, your colleagues come up with many counter-arguments as to why your proposals wouldn’t work in practice:

• We’ll do it when there’s more time.
• Code review? What for? You can program, right?
• You know this is a very specific project…
• We can’t have a project plan because of the frequently changing requirements.
• What do you mean? Everything is working fine!
• It has been broken for a long time.
• You can’t do anything with it.
• We’ve always been doing it that way here.
• What are you reading those thick books for? You won’t use it here anyway.
• OOP would be nice, but there are already so many IFs in the code that it has no meaning.
• We do not have checklists. Everyone does it their own way here
• The integration department is to blame.

Do these points sound familiar? Such never-adopters can be recognized by their excuses. On the other hand, we don’t want to throw the baby out with the bathwater. Some counter-proposals can be perfectly reasonable. Take an example. The formal code review will be widely adopted once we have migrated to Git and feature branches will be the only way to integrate incremental changes into the main development branch. Fortunately, MANTA cherishes an open and friendly culture where feedback and striving for improvement are always welcome.

Antipatterns

On the other hand, I recall several moments when I myself was overly enthusiastic about innovation, which made my colleagues mad. See some of the anti-patterns I followed by mistake:

• “Improvements are what is most important. Development can wait.” I did not know or respect the priorities of other people.
• “I understand that bugs get fixed before the release, but I also want you to start doing continuous code reviews.” Ouch, bad timing.
• “Why are you still on Java 8? We’re upgrading to Java 11” I had no clue about the actual reason for Java 8.

To Be Continued…

In this part of our two-part blog series, I talked about why I care about software process improvement. In part two, I’ll talk about the “how” — that is, the steps you can take to actually accomplish that.