Refactoring Legacy Application — Part I

TLDR: This series posts cover close to 4 years of history for Commodity & Allocation Tracking System (CATS) developed for automating the commodity management pipeline of Humanitarian Supply Chain. I try to cover important events and milestones for the project and how it came to be in its current form.

I’ve always wanted to write this post about a story of how I managed to refactor a legacy web application to address shortcomings in the first iteration of the development with the goal of addressing issues related to performance, maintainability, DevOps and overall introduce fast iteration to the application. The goal is to document the process I took in refactoring a decently sized application to address above mentioned issues.

This is a multi part blog post where I will try to cover first historical background of the project, next I will point out architectural changes and code refactoring I’ve done to address challenges faced and finally I will outline things that are lined up to be implemented to complete the evolution of the app to v2.

The application — background

This is a story of an application whose development started in mid 2013 addressing commodity tracking for humanitarian supply chain requirements. The app was aimed at addressing challenges related to lack of visibility into movement of food and non-food items through a supply chain pipeline creating delays in delivering assistance from source to beneficiaries.

Development process and team structure

The project followed Scrum development process with a two week sprint cycle and a major release cycle of three sprints or six weeks. One thing to note here is, even if the development followed a strict Scrum process most of the user stories and a significant part of the product backlog was identified before the development began. One could say this is somehow this is anti agile because of the extensive “upfront” requirement gathering and analysis but during the development process the team picked items from the backlog and validated it’s value with users during sprint planning sessions.

Team was composed of one tech lead with a double role of scrum master and 10 developers. The team followed Scrum process to manage it’s development process adopting daily standups, spring planning, sprint demo and retrospective by setting the length of it’s sprint iterations to two weeks. The team also committed for a major release of the product every three sprints (two months) with a project completion time-frame of 1 year.

Tech stack

Development was based on .NET stack (management decision) and because most of team members were already familiar with the platform as well as had years of experience working on it. Framework and toolset used were:

• .NET Framework 4.0
• Visual Studio 2012
• ASP.NET MVC 3
• AngularJS 1.x
• Kendo UI from Telerik
• Entity Framework 5
• SQL Server 2012
• SQL Server Reporting Services 2012
• Windows Server 2012

In addition the following tools were also used for collaboration and code management.

• Git and Github for code management
• Atlassina JIRA and Confluence for issue tracking and documentations. Shout-out to Atlassian team for providing free licenses for the tools since the project was licensed under Apache 2 license. Project wiki maintained in confluence can be found here.
• Atlassian Bamboo as build and CI server

Version 1

So now onto the main subject for the post! There were a number of assumptions that were considered in coming up with first iteration of the application architecture:

• Use the default project structure generated by Visual Studio for ASP.NET MVC application.
• Introduce dependency injection tool to assist in creation of dependencies for controller classes. Ninject was chosen due to popularity and active community at that time.
• Entity framework code first approach was selected to implement the data layer. ASP.NET MVC leaves out the model layer for the implementer to choose which meant that the team had to pickup an appropriate model layer for the application. It’s not like there are a lot of options for implementing the model layer in the ASP.NET world such as found in other frameworks. At the time the clear winner was Entity Framework but the issue was the fact that it was very difficult to use EF DbContext instances in unit tests so the team decided to implement unit of work and repository pattern inspired implementation on top of what EF provided. In addition the choice was to go with code first approach rather than the database first one.

Even though the decision was to use code first approach of EF, team members mostly opted for designing database tables first and create the corresponding model classes (POCOs) latter. This of course created a myriad of issues latter in the project due to inconsistent naming of database objects, duplicated in database tables and above all domain objects which look, sound and feel like database tables.

• There was also a service layer whereby all business logic resides for the application. The goal of separating logic into its own logical layer was the need for sharing capabilities within different modules (more on modularization below)
• SQL Server was used mostly for storage with very minimal logic in the database itself. Usage of stored procedures was as much as discouraged in-favor of moving logic to the service layer with the goal of making the application database agonistic and not dependent on any specific database.

High level logical architecture for CATS v1

From the above diagram one can infer that the first iteration of the application was one giant monolith with it’s capabilities tightly coupled with each other.

Beta release of v1

Development of the first version of the application almost took 1 year with an average team size of 7–10 members. Most of the team members were developers while there were 2 -3 team members engaged in exploratory testing and documentation. Once active development was completed, the application was released into beta testing.

After the completion of the beta testing (pilot testing as it was called in management circles) the app was endorsed to go into production. The app immediately faced a number of challenges, both from fitness and technical point of views. To summarize these challenges:

1. Usability and user experience: Even if the application was developed in a very close collaboration and contribution from users having frequent meetings as prescribed by Scrum framework; requirement gathered and validated from users was to some extent different from what’s really happening on ground. Stories provided by users and validated by the product owner indicated a need for the system to enforce strict business rules and rigid workflow processes which proved to be extremely restrictive and rendered the app completely unusable to a point no data entry activity was taking place by users. This was a shocking outcome considering the process the team took to develop the app through frequent consultation and approval from real users of the app.
2. Performance and speed: Performance of the application was degrading as more data was captured into the system and more users become onboard and start using it actively. There was so much data binding on most pages in the app, some pages started to take close to a minute to load — actually there were two pages which took almost forever to load that the server started to timeout before response was composed and sent to the client. Increasing the server time out (IIS) duration on one of the debugging sessions showed that one page took 4+ minutes (without exaggeration) which was quite a surprise for everyone.
3. Boilerplate code: The architecture the team followed and the nature of the framework used dictated massive amount of boilerplate code to be written for implementing a simple CRUD operation. As anyone can guess, the more code someone has to write increases the surface area for bugs to creep in. To mitigate this problem the team used T4 templates to generate a significant portion of repetitive code but still the amount of code that needed to be written by each developer was still massive.
4. Maintainability: The team followed Entity Framework (EF) code-first approach to avoid the need for extensive up-front database design opting for building the database schema from the domain model. The shortcoming of this approach was the lack of automatic database generation from existing models in EF (version 4) in the form of migrations. First of all it was difficult to have a consistent database model since each developer makes changes to the database — difficulty in enforcing naming conventions. Second the approach allowed creation of duplicated tables/fields in the database. A number of tables and fields remained unused in the database or are simply contain null values and finally limitation to follow industry accepted database modeling and design practices means the current database model could suffer from issue related with data duplication and and its effect on performance.
5. Test coverage: Lack of good test coverage means that making the smallest trivial change into the code base could result in breaking one of the most significant parts of the app — fear of making changes!
6. Automation and devops: Lack of automated build and deployment pipeline. Deployment was primarily a manual process involving copying files, running database scripts, stopping and starting services.

Evolution of the application continued in it’s current form with minor changes and additions to it’s over all architecture. The biggest challenge here is the proliferation of regression errors whenever new features are added — downtime increased due to unforeseen bugs creeping in.

It’s at this time that I decided to consider extensive refactoring or rewriting some portion of the app as a viable solution to the challenges being faced.

Version 2

The first thing I did was trying to identify major pain points in the current implementation faced by developers — both functional and non functional; which includes:

Convention based development: This is mainly for removing any decision that need to be made when implementing stories. Questions like where should I put this piece of capability? in the model? in a controller? etc. Having a clear definition of what to put where means we’re going to have highly maintainable and readable codebase that any new team member can come in and understand what it’s supposed to do.

Out-of-the box capabilities for non-functional requirements: One thing that increased maintenance cost of Version 1 codebase is that pretty much almost every non-functional requirements for the application are custom implementation rather than using existing libraries to do the same thing. Starting with security, localization, breadcrumbs, input methods for different languages, workflow engines and state management. You can imagine how much effort and energy is required to maintain all these capabilities. From the get go in our Version 2 effort, we decided that we’re not going to implement any of the above NFRs but rather find an approach where by we will be using existing libraries to implement them in an effort to reduce the code base that we have to maintain and evolve.

Model based: we wanted to simplify how the database should evolve. With that regard; having the most in-depth knowledge of the domain at the time; went ahead and designed database schema for the app by reverse engineering the existing database and removing unused fields, correcting naming of database objects, enforcing referential integrity, removing un necessary foreign key relationships among tables whose relationship could be enforce at the business layer whereby solving the issue of strict sequential workflow constraints.

Database evolution and migration: from the very beginning I made sure that we have to use an approach for managing database updates and migration. No one wants to manage database updates manually through running scripts on a production database.

Automated deployment and update: again by having an automated deployment pipeline, I anticipated that the team will feel confident to push changes as quickly as possible hence the hustle of going through lengthy and error prone manual processes for deploying changes is removed.

Other than these technical concerns, the first version of the app addressed more than 90% of user requirements. The only complaints were unfriendly UI, frustrating performance, very strict workflow steps to enforce business constraints etc. So getting domain requirements right was not an issue because the first version of the app addressed everything requested by user and all we have to do is keep those capabilities while resolving shortcomings mentioned above — it was not what it did that caused issues but rather how it did stuff that was the problem.

Evaluating candidate approaches

I did a preliminary assessment of potential approaches to address the above concerns. Having worked on a number of web frameworks using different languages, I did a proof-of-concept implementation of basic user stories from the existing application with the goal of evaluating potential web frameworks to select a good one (including the existing codebase) to address most of the non-functional requirements:

• ASP.NET MVC Core
• Ruby on Rails
• Django
• Spring Boot
• Grails

In addition to proof-of-concept implementation, I also conducted an extensive evaluation of the frameworks and produced a detailed report for the the team to decide which one to go with. In the end I decided to conduct a proof-of-concept implementation in Ruby On Rails and Spring Boot. The main reason I choose to try the above two frameworks was because my familiarity in previous projects.

From the very beginning I knew that I might end up using more than one framework to address the different aspects of the application. One of the biggest technical challenges in the first version was both the data maintenance (CRUD) and reporting aspects were combined in one huge monolith application making it a maintenance nightmare. To come across this challenge I set out to separate the two parts of the application early on and I managed to strike a comfortable balance of productivity and pressing project deadline. On one part my team had to get ready a revised version of the application with at least 80% of existing features in less than 3 months, and on the other hand balancing speed of development for the team. From my experience I know nothing beats Rails when it comes to creating a web front end for a database and in my case that’s exactly what I needed for the data maintenance aspect of the application. At first the team was very reluctant about my choice of framework but once we started putting together gems to address most of the non-functional requirements of the application allowing us to focus on the main business requirements.

In Part 2, I will discuss more on the architecture side of version 2 of the app in detail focusing on choices and decisions I made early on in the refactoring process that paid off greatly.