Unlearning software engineering

Things I will not bother with next time.

My startup journey began a little over one and a half years ago when I joined Narrative (formerly Memoto). I knew that we were building a wearable camera and accompanying software that would let people remember every moment.

However. There was no roadmap. No specs. No investors or other external stakeholders. Not a single line of code written. I had longed for this freedom since a few months after I took my first programming job.

I wanted to do everything right. I had ideas on how to make this place better than my earlier workplaces. I had studied the literature. Well tested and documented code, continuous integration, service oriented architectures, lean methodology—you name it.

Wrong. Most development best-practices do not apply when you start from nothing and do not know what your future product looks like.

Architecture and infrastructure

I started out building a service oriented architecture. I liked the idea of having clear system boundaries and to have small and easy to debug systems that were easy to scale out on their own, that separate teams could work on, should the need present itself.

I soon learned that the costs to operate and to make many systems communicate with each other is high. Dealing with and reasoning about failures becomes way more difficult. Today our architecture is mostly monolithic and taking things apart is not a goal in itself.

Before I started out, the operations part did not occupy my mind too much. I had a little experience being a sysadmin from university. Today one of the first criteria I consider before adopting a new technology is how easy it will be to run. This often rules out building (especially stateful) stuff in-house.

Tests and documentation

Those 273 unit tests and succinct documentation? No good when what you are building may no longer exist or look very different in a few weeks’ time.

Frameworks and libraries

You need to build that part in-house because no existing piece of software fit? You may feel like you are advancing along a good trajectory at first but after a while you wish that you had gone for a popular framework with its thriving community and ecosystem, instead of only looking at its technical merits. (This is why we adopted Django a few months in.)

Further, when you build things in-house you increase how long it takes for new coworkers to get up to speed because they can not reuse as much knowledge from earlier projects. Think of how much this costs before you deem a popular library unfit for your purposes.

Methodology, processes and planning

Kanban? Scrum? I am sure that the ideas in these and other methodologies are sound given the right circumstances. At my first gig we did Scrum by the book and I thought it was great. However, that place had a product, customers and had been operating for a long time. I have come to believe that things like work-in-progress limits and two-week sprints do not fit companies without a mature, stabilized product.

If your CEO gets a chance to speak with investors tomorrow and you can increase the likelihood of your company being funded in three months’ time by finishing that highly demo-able feature or configuring that server until then, you should ignore WIP limits.

If you do sprints, you may discover halfway into one that there are better ways of doing things that renders your current plans useless. See this as an opportunity not given to established corporations with more people and larger codebases to deal with. You may have no costs associated with backing off and changing directions, and you are more likely to encounter these surprises than companies working on tried and tested products.

A high bus factor? If you have as many people on board as the number of platforms and devices you need to build software for, you are in luck when people share crucial skills. Your runway may be shorter than how long it takes for somebody to get proficient developing for a new part of the system, not to mention how big the difference between proficiency and mastery is.

Going forward

Compared to last year, I think more about how the state of projects, systems and teams affect how to approach them. It often boils down to asking myself what should be done now and what should be postponed. Reasons to postpone include too little knowledge (sometimes things become clear just from time passing) or too small benefits until we grow larger (especially when it comes to time-saving activities and when to automate).

Most of the literature and discussions seem to focus on the end game: how you approach building and maintaining systems in a stable environment (known customer, funded, established team). Few articles mention how technical decisions affect other areas. Most articles deal with the long-term, but the short-term is sometimes more important to focus on.

We have gotten to a point where the development of some parts have cooled off while other parts still change rapidly. I find the mix exciting but gravitate towards new stuff and the next big thing.

If you are looking for something to read, try Antifragile by Nassim Nicholas Taleb. It deals with uncertainty.

Discussions on Reddit and on HN. Some reactions and my responses:

There is a huge difference between following sound engineering principles and just mimicking whatever you find on random blog posts. — grauenwolf

I do think you should read a lot and try to stay current, if for no other reason than to know what is out there and to draw inspiration from it.

Eventually you will end up embracing something on weak grounds. It is easy to get carried away, especially when you are dealing with something that you are having problems with and you are presented with a possible solution. But you should still be reading random blog posts, I think.

And of course you need to be actually doing things and trying things out.

There are two extremes. I think you risk making worse mistakes by copying too little and following too few trends, trying to come up with too many original solutions and ideas.

How do you learn sound engineering principles? How do you know when to mimic another solution and when not to? I think it helps both to have copied others in the past that should not have been copied, as well as having built things from the ground up where you should have just used some existing solution.

Yikes.

It’s one thing to be pragmatic, it’s another to toss out the history book and repeat mistakes millions of others have made.

This stuff will come and bite them in the ass eventually. Likewise it’s not exactly the best way to recruit top engineering talent (which seems to be the point of this posting). — nazbot

As for sound engineering principles in general, I think one important trait is to know when the established best-practices apply and when they do not.

You may knowingly accept technical debt because of short-term benefits. Similarly, I think you should allow yourself to take on debt in other areas when needed. And you should recognize that things outside of engineering and writing code needs much more attention and thought when you are just beginning and the business is volatile.

I am not arguing that you should abandon all of history forever. We have many tests today. The parts that need documentation are documented. We do plan ahead. We have some money in the bank.

My key lesson is that when you start up and build a new product, you should not set out to immediately incorporate ideas that you liked in earlier workplaces, projects and products to make something even better. Of course you want to do better than you did before, and you should have long-term goals, but everything you do should be based on your current needs and you should not assume that something works for you just because it worked elsewhere or at your last workplace.

Pre-order your Narrative Clip today. Shipping and rollout start in November. We are hiring.