Design Thinking & Software Craftsmanship — Part 1

I think building most software is a mix of engineering and art, and in some cases maybe even more art than engineering. I think most of the industry is heading in this direction too. By most, I mean developers building applications that do not operate in life or death situations. So developers working on self driving cars, airplane software, and surgery robotics your code still probably requires more engineering than art.

What is art?
It is the expression or application of human creative skill and imagination, typically in a visual form such as painting or sculpture, producing works to be appreciated primarily for their beauty or emotional power.

Last week I wrapped up 5 weeks of Design Thinking at Design Cofounders. Over the next few months I’m going to explore applying the tools I learned and apply them to how software is created.

Today’s Symptoms

Software should be more like other engineering fields, get more rigour in design, building and testing.

In university, a lot of time was spent on hammering this idea into our heads. After working in the field for 13 years, I have ironically learned that the above idea could be the reason why still most software projects fail. These failures are also not due to a lack of funds, talent, or technology. Millions and even billions of dollars have been invested before these projects failed. The issue isn’t the idea, it’s how the industry executed on this idea.

Software Development Life Cycles

Regardless of what methodology you use, software goes through different steps before eventually reaching the customer. We have:

1) organization objectives that push in certain directions

2) sales and account reps gathering market and client feedback

3) analysts pulling it all together and creating requirements

4) software architects influencing the design and implementation of these requirements,

5) QA analysts creating test plans and finding all the weird edge cases that the software needs to handle

6) project managers that won’t budge on deadlines or scope, but are okay with cutting corners

7) developers that want to use some cool tech and lose focus of why this software is getting built in the first place

8) the “swoop and poop” project stakeholders that aren’t involved in the day to day work and will pop in once every couple of months and crap over everything.

I could go on. Finally, all these silos focus on protecting their silo and making sure the project does not fail because of their silo’s work.

These steps were deemed important so that 1) we define what we want to build, 2) we build it properly and 3) we deliver working and useful software to the end user. Awesome. So why do software projects still fail? It’s not due to a lack of process or rigour. We have so many methodologies, yet anywhere between 50% and 80% of large software projects fail depending on what statistics you read. Why is that? Why is it that 80% of manufactured cars do not get recalled? Or why is it that 65% of building projects don’t get abandoned or worse collapse? Or why is it that 77% of airplanes don’t drop from the sky?

I attended a presentation once around software methodologies. The presenter talked about Waterfall, Agile and Lean. He made an interesting point on where each methodology works best (but again your mileage may differ):
Waterfall. Great for when the problem is well defined, and the solution is well defined.
Agile. Great for when the problem is well defined, but the solution is not as well defined.
Lean. Great for when neither the problem nor the solution is well defined.

Scientific approaches to problem solving work in domains governed by constant laws such as nature. Gravity is constant at 9.80655 meters / second squared everywhere. However in the abstract software world, you are truly only limited by your imagination. Even most hardware limitations that used to exist 50 years ago are negligible today.

The building blocks of imagination are experience, so if you haven’t experienced something, you can’t use that as the basis of imagination. Imagination is also constrained by experience, so the more you know the more limited imagination may become — Paul King

But still, most software projects still fail.

Where do Design Thinking tools fit in Software Craftsmanship?

In the 5 weeks at Design Cofounders I learned that Design Thinking is about divergent problem solving. This means we are not trying to converge into a solution from the start but allow ourselves to explore different solutions in parallel. By doing so we try to build up the solution that will create the ideal future state of the system.

As humans, we’re not great at planning stuff. We don’t do very well at engineering a pre-defined outcome right from the start. When that works, we just got lucky, but we think it’s because we are great planners. On the other hand we are good at experimenting, learning, analyzing and finally making informed decisions to achieve the best possible outcome given our current knowledge. — side note, get the free PDF of 37Signal’s Getting Real book. I have re-read it at least once a year in the past 10 years.

All of today’s software development methodologies assume that the ideal future state is well defined up front; even Agile. These software methodologies that are based on a scientific approach to problem solving ignore all the hidden variables in a system. These hidden variables:

1- cannot be uncovered in a requirements document delivered up front

2- can only be uncovered by the people living in the problem space daily

The client signed off on X and we delivered X so we’re good.

These hidden variables can make or break a project. Yet, we still deliver these requirements documents, we still write these user stories and we still create these perfect mock ups. It gives us a fake sense of security about the outcome. It also gives us an out when faced with failure — but you signed off on this…

Does it bring us closer to that ideal future state?

There was a great talk at #DevTO by Jason Theodor titled “Fail to Succeed” (starts at about 46min). There is one slide in there that describes how “failure” originally used to mean “not starting” and everything else past the start point was varying degrees of “success”. Today, achieving anything that is not the outcome you wanted is usually considered a failure.

My goal for applying Design Thinking to Software Craftsmanship is to create a framework where it is safe and even encouraged to explore the problem through code. To actually build stuff and use that as your compass towards a solution that will bring you closer to that ideal future state. To replace feature requirements and user stories with context and goals. To help us remember it’s not about idea A or B, nor is it about technology C or D, but instead it is about answering this simple question: does this version of our software bring us closer to that ideal future state?