Let me tell you a story about programming.
This is a story of mental labor that is something people not design to do. Computers are.
We stare at lines of code and trying to execute the code in our heads. Some of us are the best in the world in such operations and are proud of that. There are competitions where you can show your ability to solve complex problems by decomposing them into computer operations. We do all this because this is the way it has been since the first computer was designed by mathematician. They are usually pretty good at doing such operations and that is obviously affected how computers and programming languages are designed.
We stare at text files, that have hundreds and hundreds of lines of a language that somebody made up. Quirks of syntax and style guide that gives grief to beginners doesn’t bother us anymore. But the real problem lies in the complexity of the systems. When you start a new project, everything is great — you can keep the whole system and all the code in your mind. Problems arise when many people work on a rapidly growing system. You spend hours debugging some simple thing, just because some internal assumptions of the function you used were different than yours.
The core problem is assumptions. They are everywhere. Making assumptions is what we do everyday to write code. Any line of code you write starts with an assumption. Is this data going to be on disk? How many users will access this service at any moment? etc etc
Beginners usually oversimplify assumptions and then pay for it. They didn’t account for many future questions. In result, they will need to hack across the system or spend very long time refactoring to change an assumption.
Medium experienced people start to over-engineer everything. These are beginners who got burned repeatedly on oversimplification. They realized that assumptions change as projects progress. To prevent rewriting and hacking they start building “spaceships”. Software which has configuration file that allows to change it’s functionality from flying into space to drilling to the earth core. Usually such projects takes a lot longer and they many times fail because of time/money/complexity of design.
Experienced programmers find the balance of making right assumptions when they know what will be the need in future year and leave enough flexibility in the places that may change in close future. This comes with better understanding of business needs and reasons why requirements change. The skills to design such systems also come after building over-engineered systems that didn’t deliver value and were abandoned or dismantled.
But even an experienced programmers can’t predict future and in a year or two their code needs to be rewritten. Why? Because if business grows or shifts, so do the needs for the software. It may be customer requested features that nobody thought about before, regulations, new algorithms, new frameworks, new data became available, better machine learning, acquisition or partnership with other products, just simply not enough design originally to support growth (of traffic or features).
This leads to weeks and months of refactoring or rebuilding from scratch. Refactoring takes a toll of having working state every time you change something. Rebuilding may never end because either over-engineering kicks in (oh, our design couldn’t handle this, let’s rebuild everything so we can handle anything next time) or just while development was frozen the product competitors overcame with better offering. Sometimes people get tired of working on dying code and they leave, and now you have new people trying to refactor previous solution while not knowing all the details and assumptions that are encoded in millions of lines of code. This leads to degradation of the product and customer churn.
This happens all the time and everywhere. Any software company big or small have these problems. The code churns all the time, systems get build, refactored and rewritten. Business get born out of hacks, gets traction, hires number of engineers to rewrite it and sometimes dies before they manage to rebuild the first or second version.
Is there a way out of this?
I argue that yes. We need to lift our understanding of software from lines of code to problems it solves. We need to change our perception of engineering from bottoms up, where you need come up with an algorithm that matches current assumptions and only define assumptions and let computer to find appropriate way to handle them. Move from text files to UX-driven development, where UX is aligned with human thought process.
What are assumptions that costed you dearly? Did you ever spent weeks or month reworking something that would’ve been fairly straightforward to change earlier in development process?
(inspired by this article: http://mashable.com/2014/04/30/programming-sucks/)