Transforming the productization of software — we the people

10 min readOct 16, 2020

This is an article in two parts. In the first part, we show how including developers in an iterative development process can improve the technical solution. In part two of this article, we discuss how the input from the developers impacts the development of the technical solution.

Introduction

What separates those who build software professionally at scale from those who don’t? There are many factors involved, and many have nothing to do with technology:

Does there exist a culture of innovation and willingness to take risks?
Is there a sufficient number of employees with the required skills?
How does the company organize its people?
Does there exist a technical foundation that allows teams to excel?

Each of these parts is equally important. If the organization we work for, Oslo’s municipality, didn’t already have some of these bases covered, we probably wouldn’t be having this conversation. By technologically young, we mean that the digital transformation the municipality of Oslo is undergoing involves building up in-house software product teams. These product teams build and deliver products to the municipality’s citizens. Currently, there is no standard way of turning source code into products used by end-users. This article will discuss why standardizing such a set up can improve an organization’s technical foundation, and we define a holistic approach to achieving that goal. A significant part of undergoing such a shift is getting the developers aboard. We will be dedicating more time to discuss everything that builds up to a solution then what we ended up building.

What part of the technical foundation are we talking about?

The software engineering discipline we are concerned with is the productization of software for end-users. Some of the critical aspects of this domain are:

Identification of the parts that make up a release.
Ability to guarantee operational stability.
Provide a stable framework for development, deployment, audit, and accountability.
Ensure productivity in the software development cycle.

Quite a lot, in other words. We will discuss how an organization undergoing a digital transformation can use the principles above to improve its technical maturity. We will do so by presenting one way of setting up a production environment and how that environment can integrate with the essential components that facilitate getting the source code to the end-users as a product. Where setting up the production environment includes:

Creation of the infrastructure on which the software runs.
Assist the developers with managing application assets and dependencies.
Integration of version control.

We also include elements of release engineering. Including tasks before getting the product into production:

Test the functionality and security of the source code.
Build artifacts or containers.
Wrap the artifacts in a unit of deployment.
Release or transition the deployment into production.

And after the product is in production:

Collect metrics for alerts and trends.
Gather logs and traces for debugging performance issues or bugs.
Track usage for business insights

The system we are describing above has numerous moving parts and is complicated to set up. Many details aren’t apparent at first glance. One of the critical aspects of the tool we have built is automating these processes’ setup. Most importantly, we make them replicable. Pertinent questions to the above statement might then be:

Why is automation required?
In a world of DevSecOps, shouldn’t the individual teams be responsible for this?

These are valid questions, and depending on your business’s type, size, or technological maturity, setting up a production environment might not be a problem. Nonetheless, we would argue that automation is critical. By automating, we minimize the chance of introducing an error. Automation is also fast. Finally, just because we set up a system on a team’s behalf, it doesn’t preclude that team from having a working relationship with it. Not if we build it correctly, at least. At any rate, there are several good arguments for a standard solution.

Why standardizing might be sensible.

If the company isn’t already using a cloud operator, it is common to start moving to the cloud in a digitalization process. Moving to the cloud makes sense; the cloud operators offer managed services, an endless amount of storage and compute, and the ability to do what you want when you want. If you’re going to be an agile and modern company that builds or maintains software products, it is hard to avoid using the cloud.

The downside is that many cloud operators exist, and each operator offers several ways to implement a cloud production environment. Early in a digital transformation, most teams need to build their production environments from scratch, and they frequently do so with varying degrees of coordination. From a business perspective, this is not a viable solution for several reasons:

It takes time to build a production environment.
Time is money; I know, I know, but it is still valid.
Knowledge is limited to a few individuals, which might leave or change teams.
Security teams will have to audit a larger number of solutions.
Do the teams have sufficient experience building such environments?

From a developer’s perspective, this might not be viable either, for their own reasons:

I prefer to focus on developing the product.
I am not interested in being the infrastructure person again.
I do not want to learn a new field.
I want something that works, alright!

There are then viable reasons for centralizing production environments’ configuration and maintenance from a business and developer perspective. Deciding to consolidate on a standard solution is tough if no such solution existed before. There is a perception of loss of control for many developers, and the decision will meet resistance. There is a silver lining, though. The developers’ fears are opportunities to improve the way we think and work with infrastructure and deployment.

Whatever we do, we must not disregard the concerns, but rather address them. We can achieve this by making the developers an active part of the development process:

Interview the developers to understand their relationship with production environments, concerns, fears, desires, and more.
Make the source code available for all to read.
Be open about the road map and welcome contributions.
Communicate frequently and clearly.

Finally, and most importantly, don’t try to convince everyone or force anyone. As with the adoption of any new technology, there will always be early adopters and laggards. Focus on assisting the early adopters and listen to the laggards. You will frequently learn more from the latter group.

Not forcing anyone is pertinent. We will not require anyone to adopt the solution we are making. Instead, we devise strategies to ensure adoption. We need to provide value and let the results do the convincing.

How do we go about building a replicable setup?

Where do we start? Selecting a combination of components and landing on a technological platform is demanding, but that isn’t too difficult. The most challenging aspects have nothing to do with technology, but people:

What do our users want?
Are we building the right thing?
How can we ensure users adopt the solution?

As it turns out, that is a good thing. If we do not consider these questions, we are merely selecting a technology stack based on our own bias’ and interests, which we want to avoid.

We do not want to build a solution looking for a problem.
We are not making a solution for ourselves.
We need to solve actual problems, our users’ problems.

In essence, the technology stack is not the hero. It is merely a means to an end. Our motivation for building a solution is to ensure that the developers can deliver software to their end-users with minimal friction.

How do we find out what our users want?

We can understand what our users want by:

Creating an interview guide and talk to the developers.
Understand the motivation for the developers’ current solutions, if such exist.
Review similar solutions they have used previously.

By completing these activities, we can uncover new insights or verify our assumptions. When completing this process ourselves, we learned that many developers are skeptical about being provided a solution because:

They fear a loss of control.
What happens if they have use for functionality that doesn’t fit the mold.

At first glance, the fear of losing control seems like quite a severe concern, and what is the implication for us? Does this mean we will never provide a satisfactory solution because it wasn’t built by themselves? When unraveling this statement, we came to understand that this means developers are afraid they will not be able to:

Introspect or understand what the solution is doing on their behalf
Be able to debug issues that might arise on their own

In essence, do not build a black box. The developers do not want something that magically works. That is great because we can work with that. It is possible to write good documentation that describes the solution’s composition and why we arrived at that solution. We can also make any artifacts we build, compile, or generate on behalf of the user available to them. We can even create detailed reports that describe all changes that are applied by the tool. No guarantee this will satisfy every developer, but it is a step in the right direction.

Similarly, we can ensure that we minimize the use of custom-built functionality. Use what already exists within the space as much as possible, but help the user configure and use it correctly. What does this mean? Do not build a run-time environment from scratch, unless nothing exists for the problem your company is facing. The rest of us can probably get away with using Lambdas, Cloud Functions, Kubernetes, or some similar technology. When landing on a run-time environment, be compatible with the tooling that the system offers. Now our users can search google if they come across an error message before reaching out to us. They can use a familiar CLI and make use of their one-liner magic. The users are in control.

How do we know we are building the right thing?

To ensure that we are building the right thing, we need to:

Verify, verify, verify.
Build as little as possible and then verify.
We want to mention the word verify again, but we won’t.

To verify, we need to put our solution in the hands of the users. We need to organize user tests to use the tool we have built to create an environment with its integrations. Ideally, that environment is not a black box. We use standard tools, document what we are doing, and report what is going on.

Continuous user-testing is essential to ensure that we are always on the right track. By frequently testing our product on users, we can uncover our biases’. We are not building this for ourselves, but we are still the ones making it, and user testing will reveal:

Things that are intuitive to an expert may not be intuitive to others.
The functionality we think is excellent provides no real value to the users.
Opportunities we haven’t considered yet.

It encourages a symbiotic relationship and puts the users in control. It shows that we are listening to them and take their concerns seriously. User testing also uncovers opportunities we might not have thought of; we tap into all the developers’ collective knowledge.

How do we ensure user adoption?

We are well on the way when adhering to the insights we have discussed so far, from functionality, documentation, feedback, and introspection using familiar tools. There are still some key points that can help push us over the edge:

Make it easy for the developer, even if that means it is difficult for us.
Be faster and more accurate than the competition.
Build something developers can operate, not us.
The building is simple; maintaining is difficult.

Imagine a situation where you come into a team with an existing infrastructure setup and modify it. Where do you begin? Probably by working through a list of prerequisites in a readme. Maybe there exists a Makefile that automates some of the processes. Invariably, the user has to jump through several hoops before they can get started. We want to avoid a situation where the user is responsible for tasks that we can complete on their behalf. It will most likely make our lives more difficult, but that is far preferable. The more obstacles we remove before the user gets value out of the tool, the more likely they will adopt our solution.

Being fast and accurate will also aid in adoption. Ask members of different teams how long time they take to set up a new environment, and you will likely hear numbers ranging from two weeks to three months. The former seems optimistic; insert joke about developers and estimates here, but this might be too long even if the two-week estimate is correct.

For Oslo’s municipality, during the outbreak of COVID-19, it was critical to quickly and efficiently build digital solutions to relieve pressure on personnel, such as those operating the phone lines for booking a COVID-19 test. You do not want to be in a situation where you need to wait to put your product into production when lives are at stake. Even when lives don’t depend on it, focusing on product development and having an efficient and fast production environment also facilitates experimentation and rapid prototyping. The bottleneck should be the cloud operators, e.g., the time it takes to complete a request for some cloud resource, not the developers on a team.

Developer first, developer first, developer first. We have said this before but bears repeating. Developers are users just like end-product users. Building something that suits their needs is essential. We are not making a production environment that we can operate. We are building a production environment that our users can run. We often create infrastructure for the infrastructure team without considering the software’s customer: the developers. If you provide a Kubernetes cluster, don’t start with the Custom Resource Definition until you know that it solves a problem.

We can assist with maintenance as well as set up. Generally, setting up something the first time around is the easy part. Moving with the times, simplifying, taking advantage of new models is the hard part. Doing this in a product team, while trying to reduce the backlog can be challenging. By offering an easy way of keeping up to date, we can increase our product’s value. It also shows that we are in it for the long run.

In part two of this article, we discuss how the input from the developers impacts the development of the technical solution.