The software engineering cult
When I joined a company back in 2010 they were going through something of a transformation. They had just redesigned, rewritten and released their customer facing website with great success. They were making more money than they could ever have anticipated and it was growing exponentially. They had also (like many other companies) just discovered agile.
While many companies were more cautious in jumping on this bandwagon, the company I worked at (which comprised almost entirely of consultants, or ex-consultant who had become permanent employees) quickly prioritized becoming more “agile” at any cost. If any software got built as a consequence of being agile then that really was a happy co-incidence.
While the engineering teams were the first to get their card-walls, standups, retrospectives, showcases, sprints, etc; it wasn’t long before every team in the company was religiously meeting each morning to tell each other what they did yesterday and what they were planning to do for the rest of the day. We were even paid visit from one of the founding members of the agile movement (who’s recent blog post happened to inspire this post) who delivered a sermon on the next stages of enlightenment: more REST was needed.
Seemingly now; so agile we couldn’t fail at anything; we decided to turn our attention to the neglected back-office systems. The CRM, ERP, payment processing systems had not been touched since the company was first conceived and needed the same treatment we’d recently given to the customer facing website. We spent about a year and a half working on a replacement. We picked an arbitrary deadline by which it needed to be done and were then forced to work evenings and weekends on a death-march to finish it.
About 2 weeks before it was supposed to be released the project was finally cancelled entirely. There was no hope of it replacing what we already had. It was slow, unreliable and missing many of the necessary features of the software it was intended to replace.
Much soul searching followed. We had retrospectives that lasted multiple days. Team members were individually interviewed and thankfully all of the root causes for the failure were discovered.
We had not used micro-services. This was the first problem. No-one was pair-programming either; another problem. Test driven development had not been effectively enforced leading to less than 100% test coverage and acceptance criteria had not been written using the words: “GIVEN”, “WHEN” or “THEN”.
We started a new project from scratch and would not be making these mistakes again.
This time we built hundreds of services. Some of them were only a few lines in total. They all had their own database so we invented new (unscalable) ways to sync data between them.
We were deploying code into production hundreds of times a day; it was an amazing feeling (unfortunately no-one was actually using this system at the time).
We couldn’t test anything before deploying it because it was impossible to keep a consistent test or staging environment running with all these services at the same version they were in production so we invented new tools for mocking services. We told everyone about them. Conferences, meetups, blogs. We had no evidence they effectively prevented bugs but that didn’t bother us at all. The github stars were all that mattered
No-one programmed alone. We plugged in 2 mice and 2 keyboards to our laptops. One person wrote a test, the other had to guess what feature they were working on and write the implementation. Just when you got to know the person a little, we all swapped partners. This game was not meant to be easy.
After about 9 months with a pretty huge team on this all we’d come up with was an ipad app that allowed sales people to calculate the price of a contract for the products we sold. We’d failed again at replacing our CRM and I decided it was time to move to another company.
The place I work at today (which is incidentally is not the next company I moved to) is a big company, so I’m sure not all teams work the same way, but the two teams I have worked in keep all their code in the same repo. We enjoy having atomic commits. We can refactor the whole code base either manually or with codemods in one go. Deployments go out a few times per day and we don’t have to manage thousands of different deployment scripts to get a simple app out the door. We can re-use code without having to package it up into a dependency that then needs to be kept up-to-date in multiple services. Data mostly lives in the same place, but when it doesn’t it is because it needs to be in a data store with different characteristics, not because a consultant told us it should. We do have some challenges with this approach but we’ve learned (from experience) that these problem are more desirable than the alternative set of problems we’d get with a different architecture. We are lucky to have the resources to do this, but we often implement things more than one way when faced with different options to understand the pros and cons of each; particularly if the cost of getting it wrong outweighs the cost of testing.
There are a lot of ideas out there and some of them have merit and some of them are just popular because they sound convincing. In my experience, the ones that originate from companies that have already demonstrated their value internally (e.g. graphql, or kubernetes) are usually the better bets
