The Tug Of War

It was 2017 and we, the Weengs tech team, found ourselves in a tug of war. The business had grown 700% in that year, vigorously pulling us forward. On the other end of the rope, our legacy system was firmly pulling us backward. The situation was unsustainable and brought us to a halt — we had reached the chronic stage of patch-only development.

By that time we had pivoted twice, changed the profile of our customers and changed how we do business. Our data schema was so stretched out by these pivots it was about to snap. In addition, indiscriminate usage of our beloved ORM coupled all layers of our architecture to the data layer. Meanwhile the company was creating new departments and they were all using the same admin system.

It might sound calamitous but situations like these are quite common to startups. On the first stages of a business - let’s call it 0 to 1 - building new stuff and getting things done takes precedence over quality or doing things the “right way”.

But 0 to 1 had passed, we were now at 1 to 2 stage and Series A funding round was on the horizon. Right at this critical junction, the company took the brave decision to plant the foot on the break and give the tech team the chance to do proper foundation work in order to sustain the growth. So around January 2018 we all gathered together to decide where and how to start, the aim being to improve some important parts of our system. These were:

Data

Objective: create the conditions to evolve our data schema

The way frameworks are presented makes people believe they are the silver bullet of software development, even jobs are advertised with frameworks on their title, but it’s not like that. Frameworks can indeed help with some heavy lifting but systems can’t be developed in autopilot mode by following framework documentations. In our case, in the rush to build new stuff, Eloquent ORM was indiscriminately used, pretty much like it is presented on Laravel’s documentation: in routes, views, controllers, jobs etc. The consequence was quickly felt: nothing could be changed on the database. Our data layer was coupled to every other layer of our then undefined architecture.

Action: make our apps Eloquent free

We decided we would remove Eloquent from all our apps, period.

Code

Objective: make our code more readable, remove code duplication and increase test coverage

The business was running on untested spaghetti code, duplicated over different apps and sharing the same database. To bring some order to this chaos we decided to adopt a service-based approach to interact with our domain.

Regarding tests, we consciously decided to go for minimum test coverage of our to-be-created application services. By minimum tests I mean testing the “happy path” of the service (where the action was successful) plus any exception the service throws. We had to be pragmatic about it as we knew we couldn’t push for full coverage, this refactoring had a six-month deadline with an investment from the company of 20% of the tech team time, meaning we could only use Fridays to refactor. But moving from no tests to minimum tests was good enough. The best is the enemy of the good.

Action: create an application service class for each business action

These services are ignorant of infrastructure, relying on good abstractions instead. They have a strong contract, receiving a request object and returning another response object. The naming would be decided by the users of the system (not the tech team). The aim was to make the whole domain more expressive and easier to understand. Our apps and APIs would only interact with the domain through these services, and Eloquent models could only be used inside the them (we added a tool in our pipelines called The Snitch to enforce it). By doing this we would disentangle the application and data layers, an essential step toward enabling us to evolve our data schema.

Contexts

Objective: stop different departments sharing the same admin system

Everybody being able to do everything in the same admin system was causing an array of problems to the business, it had to stop. We could either create different apps for each department or create a complex permission system in the existing admin system. We chose the first option because we wanted to control the users’ actions and most importantly to give each of the company’s department a unique and focused experience from their area of interest/perspective. Given the number of departments we judged we would be adding too much complexity by using a permission system so we decided to go for contexts. Each department would be treated as a different context, in DDD jargon they are some sort of bounded contexts.

Action: create different apps for each of the company’s department

For the operation (warehouse floor) department we’d create Electron apps to be installed on tablets and laptops. Customer facing apps would be Vuejs SPAs and our drivers would use React Native mobile apps, all these apps backed by Laravel APIs. And for the rest of the company’s departments we’d create classic Laravel MVC apps with HTML views.

Architectural Style

Objective: identify and master an architecture style that accommodates our existing system

This refactoring was about our 0 to 1 PHP Laravel codebase, our legacy system. We decided we’d do an effort not to grow it even more, new features would come in as new independent systems with the most appropriate technologies for their use case.

Action: move from “no style” to monolithic microkernel style(ish)

Different apps were sharing the same database and code was just “the framework way” duplicated across the apps. Something like this:

When you share a database you’re basically left with two options: have a monolith or a monolithic deploy. We chose the later in the form of the monolithic microkernel architectural style where you have a stable core (representing the high coupling on data level in our case) that gets extended by plug-in components (in our case our Laravel apps or APIs).

This core package is composed exclusively by database models, migrations and abstract application services classes.

• Database models and migrations because they represent the data and the data is shared.
• Abstract application services classes are needed to DRY out any logic shared across contexts.

Thus our architecture continues to be monolithic because all those apps and core live together in a single Git repository and share the same pipeline. But on the pipeline deploy step the different apps are fanned out into separate serveless apps on AWS Lambda.

Also very important, our architecture is transitional and the adoption of this style was needed as a first step to fix the distributed spaghetti that was causing coordination overhead on our deploys. The way it was, any changes on the database would leave us guessing which app was being affected. It was obviously an unsustainable situation, in normal conditions of temperature and pressure we wouldn’t adopt this style.

Conclusion

Now that we have an overview of the situation I can’t wait to dive into the nitty-gritty and show how we pulled this off — more posts are on the way! As you can see, the problems overlap and can’t be solved in isolation. Needless to say, a tight-knit team who fully understands the problems and participates in the solution is essential to tackle such a mammoth task. During this refactoring journey friendships were forged, tears were shed and lots of coffee was drunk. But we are now reaping what we have sown, instead of putting out fires we can now implement features with impact. We feel safer to change the system because of the increased test coverage. Onboarding new developers is easier, as now we have clear development guidelines. There’s a strong desire to protect what has been done, no one wants to go back on how things were. Now new features come in orderly fashion and things naturally fall into the right place. And last but not least we are proud of what we have achieved ❤.

Next: The Nitty-Gritty