Nine factors that shape software evolution

In 2012 I started moonlighting on a startup with my former PhD student Parmit Chilana and my UW iSchool faculty colleague Jake Wobbrock. We wanted to take some of Parmit’s dissertation research on a system called LemonAid and find a way to sell it to businesses. We had some good evidence that people loved using it to get help; we thought businesses might love it to, since people who get instant answers to their questions contact businesses less for help and are more likely to buy.

Zoom forward more to 2017, and now that little commercialization project has turned into AnswerDash, a venture-backed startup with almost two dozen employees, a diverse collection of small, medium, and enterprise customers, and a slick, simple, best-in-class solution to contextual Q&A.

But this isn’t a post about AnswerDash. This is a post about what it was like to build AnswerDash, as a co-founder, an engineer, a designer, and a researcher.

Now, because I’m a researcher, I didn’t want to start a software startup and then just reflect on my experiences subjectively. I wanted to rigorously study my experiences, as any good scientist studies anything. To do so, I wrote about a thousand words a day across the three years I was at the company about what was hard about engineering our product. Not just technically hard, but socially and emotionally hard. After those thousand days of startup rollercoaster, I had hundreds of thousands of words about the challenges I faced as an individual.

After a long process of reading and analyzing my diary, I derived nine claims about what I believed explained the challenges I faced. Then, like any good academic, I wrote about them and shared them with my academic software engineering research community, and then presented them (today!) to an eager crowd at ICSE 2017.

Here are the nine claims:

1. Software engineering is structured by individual and social decision-making under uncertainty. This is the idea that progress on product is impeded primarily by two big forces: the need for consensus and the lack of evidence. When you lack either, decision making becomes a primary bottleneck both for decisions and for quality decisions.
2. Product design is a power struggle between diffuse sources of domain insights. This is the observation that regardless of what you want to build, your company isn’t the only entity with power to decide. Customers, investors, and end-users also decide, and they often decide based on biased, incomplete information about exactly what benefits of your product offers. This changes what you make in ways that better fit a market, but often leads to less superior products.
3. Translating a value proposition into code requires both planning and persuasion. Think of software development like a big telephone game. Every single beautiful idea that comes out of product design has to survive the translation from idea to implementation, and that translation is lossy not only because of the medium of code, but also the personalities and opinions of the people writing the code, who might resist, refine, or revise that beautiful idea. Fred Brooks called this conceptual integrity.
4. Debt is both technical and cognitive. The idea of technical debt is essentially that the ideas in code often lag the vision of a product: debt are all of the things you know are inconsistent with your vision. But I found that technical debt wasn’t the only kind: there was also cognitive debt in my engineers’ understanding of both the code and the vision. I suspect that developers can’t even identify technical debt until they understand the product’s vision. Managing a whole team of heads out of sync with a vision is a grand challenge of product management.
5. Effective developers are patient teachers, rapid learners, and prosocial communicators. I hired some great engineers. After reflecting for many years on what made them great, it was ultimately their skills at acquiring and conveying knowledge, which derived from their ability to be open to learning and teaching. This is consistent with some of my lab’s other work on software engineering expertise.
6. Quality-driven management requires trust. Deadlines are the easiest form of project management to enforce, but I found that they were terrible at producing software quality. On the other hand, shipping based on quality required an immense amount of trust because of the anxiety it generated outside of the engineering team. Making sales, marketing, and support wait for the right amount of verification and polish requires an incredible amount of faith in an engineering team.
7. Schemas structure work. More than anything else, how we structured and managed our data appeared to determine the code we wrote, the processes in our company, the customer education we did, and the product and market opportunities we had. If content is king, your schema is god.
8. The actual software development work involved in evolving software was mostly trivial—unless dependencies are hidden. The only time that coding was hard was when things were invisible: missing reproduction steps, unexpected inputs, undocumented behavior, external infrastructure changes, vulnerabilities.
9. Research impact requires perfect timing and minimal risk. Researchers (like myself) often wonder why industry doesn’t pick up our ideas. Once I was in a position of actually having invented something in research that was appropriate for our business, the reasons why became crystal clear: there’s a massive gap between a research prototype and a product, and most companies do not have the capacity to risk bridging that gap. It’s the job of researchers to take their work that last mile and make it a worthy risk.

This is theory I came up with to explain these nine claims:

Software evolution is structured by the constant, collective communication and agreement of a coherent proposition of a product’s value across design, engineering, product, marketing, sales, support, and customers.

What does this mean?

Software design isn’t about technical choices alone, but rather 1) the collective development of a theory of a product’s value, and 2) the collective effort to communicate that theory to the employees in a business and and the customers outside it. If this theory is true, software engineers should be focused just as much on the ideas in their code as the code itself and organizations should be focused on ways of explicitly supporting this idea development and communication.

What to learn more about the specific experiences that led to these ideas? Read the ICSE 2017 paper I published on these ideas:

Andrew J. Ko (2017). A Three-Year Participant Observation of Software Startup Software Evolution. International Conference on Software Engineering (ICSE), Software Engineering in Practice.