Dealing with Legacy Systems: The Inevitable Challenge
Disclaimer: No fingers were broken for countless snapping
The problems that are associated with legacy systems are commonly known; obsolete technologies, the exorbitant cost of maintenance, and many more. Given these challenges, organizations tend to find themselves in a difficult position to decide if they should keep them or completely replace the systems. A thorough analysis is required to determine which approach is the best.
The recent legacy system that my team and I, Team Thanos, had the opportunity to work on was sitting on a mainframe. According to a recent survey conducted by Stack Overflow on the most popular programming languages, mainframe languages are less than 50%.
Because the revamp also had a tight deadline, we had to brainstorm the best approach for the successful delivery of the new system. Here are our top key decisions.
System Migration Key Decisions
1) Big Bang Approach vs Agile Method (MVP + Incremental Releases)
From the beginning, it was decided that the claims processing system would do a big-bang approach. The nature of the claims is that they are interdependent on one another. If we had done incremental releases in an agile way, eg. Gradual feature by feature releases, one of the biggest issues with this is that it is inefficient to rely on two systems to handle different claim submissions. This could lead to many potential problems and wasted efforts, such as creating temporary custom integrations with the legacy system, troubleshooting bugs on both systems, and many more.
Despite that, doing a big bang approach also had its risks. Team Thanos worked under a tight deadline to deliver all features, performed user tests extensively, conducted comprehensive planning and preparation to reduce potential issues that could surface before and after the launch.
It was not an easy decision. After weighing the cost and risks of both approaches, we believed that it was the best one. A big-bang system migration was definitely out of our comfort zone as it was never done before in our department, Agile Consulting & Engineering (ACE). But hey, our main Marvel character, Thanos, has put it aptly.
2) The Need for Data Migration: Reuse or Revamp?
The next task on the list is inevitable, the data migration module. The nature of the legacy claim data is that future claim applications depend on past submissions. Hence, it makes sense for the data in the old system to be migrated over to the new system.
You might wonder, why didn’t we just use the old database. Well, it’s simply because the new system that comes in with new and streamlined processes is reflected in how the data is structured too. Additionally, there were considerations that also helped to facilitate this decision.
Change of data structure due to new/improved processes: The UX Designer team worked hard to understand the former processes to ensure the new ones are efficient and effective. If you’ve heard of the PPT (People, Process & Technology) framework, the relationship between the three elements should be balanced as it significantly contributes to the operational efficiency of a process. As a result, naturally, the new system and its database structure would reflect the changes too.
Change of database technology: We looked at key considerations such as system compatibility, ease of use, availability of support from the developers’ community, and expertise within the team. As the new system is hosted on Government Commercial Cloud (GCC), we cannot simply continue with the mainframe database given that it is not compatible with the cloud services. In addition to the key considerations mentioned above, benefitting from cloud services is one of the core aims of the overall system replacement.
3) Using a Combination of BPM Software and Bespoke Web Development
The legacy system had a lot of business rules and processes.
As such, we leveraged on Business Process Management (BPM) software, a COTS proprietary product. The core benefit of such a BPM is the workflow engine that helps to manage and track the life cycle of a claim. Other benefits include built-in functions and features such as a form builder, integration module, authentication/authorization module, and many more.
However, we only used it for the internal-facing system. For the applicant-facing portal, we chose to build a bespoke web application instead because the portal did not require extensive business logic and we wanted to customize the look and feel of the portal. The COTS product did not allow us to customize the web design as per our UX team’s user research data.
Using a combination of bespoke web and a COTS product comes with its challenges such as integration issues where the COTS product couldn’t handle high traffic requests well. As a workaround, a custom queue was implemented to throttle traffic coming from the applicant-facing portal. Another issue was that testing the entire system comprehensively was more difficult due to the lack of testing capabilities in the COTS product. We had to develop custom scripts to fill these gaps.
Despite these challenges, it was still worth it to use the COTS product. From our experience and track record of building claims processing systems eg. Business Grants Portal and OurSG Grants Portal, using it has greatly helped to simplify the overall development. The above-mentioned benefits also helped the team to develop business requirements fast while working with a tight deadline.
Ending The Journey With a Smile :)
Looking back, the experience of revamping a legacy system has been enlightening. When we undertook this project, we could only anticipate the challenges that come with it. The project is not only the first of its kind in ACE to do a big-bang implementation but also do a full data migration. We believed in ourselves and worked hard to deliver our best.
After the launch in early November 2021, it was a rewarding feeling to realize that although there have been bugs being reported, no critical issues had happened. That spoke a lot about the amount of planning and preparations done by us. Furthermore, it was wonderful to know that the end-users like the new system.
The journey wasn’t as easy as how Thanos would do it by snapping his fingers. I guess, naming the team as Thanos inspired us to get to the closest outcome of transforming the challenging task into reality. And we did it!
Rabiah Khairy, from Team Thanos signing off *snaps fingers*
Thank you to my *Team Thanos and stakeholders for this incredible experience. I have learnt a lot. To Gabriel (Tech Lead), Hock Kay (Delivery Manager), and Kian Seng (Senior Software Engineer) who guided me in our Data Migration module, a million thanks for your contributions to this post.
*Members of Team Thanos
Hock Kay (Delivery Manager), Gabriel (Tech Lead), Chang Chao (Quality Engineer), Patricia (UX Designer), Software Engineers: Rabiah Khairy (Me), Kian Seng, Hanwen, Yong Xiu, Hao Eng, Charles, Eric, Danial, Jun Yong, Justin
Our Cognizant friends: Karthik, Raghul, Teja, Prakash, Sampath, Guo Xian, Shiau Hui, Ameline, Shriya, Chin Siong, Gopal, Venkat, Pratiksha, Ravi, Saikrishna
