What is your Data Migration Plan?
It’s been a while since my last post. Thank you to those a few who read, clapped, responded, and saved my post in your list. It’s been a busy time with relocation from New Zealand to Melbourne to pursue data governance specialisation in my new role. I am drafting more content on data governance and data management topics I want to cover in the future under the Data View House™ series — simplifying the language of data.
Meanwhile, I started to curate and conclude my data migration writings drafted over the last two years about the data migration plan, including tips, techniques, and lessons learned about data migration during decommissioning legacy applications in the public healthcare sector. My last two articles on data migration from legacy applications only describe challenges and best practices from the technology perspective for the ETL developers and data analyst tribe. With the extended thoughts and ideas on the data migration plan in this article, I want to elaborate people and process components for project leads and program managers involved with decommissioning a critical legacy system.
Motivation, Structure and the Audience
In my early days with data migration from legacy systems, I believed it would be one off-job and a particular case. However, I soon discovered that tech-debt challenges from critical business systems operational from the late 80s to the early 2000s are widespread across business and industry verticals, making legacy data migration projects a common phenomenon that kept me busy with multiple clients for almost for a decade.
A few years ago, in early 2021, I was fortunate to be part of a discussion where a senior business executive wanted to know what he could do proactively to decommission a significant asset management system (Water Management) planned for the next three years. This discussion highlighted a need for a framework or guiding principles on data migration while decommissioning legacy systems, which can be understood by semi or non-technical business executives and stakeholders, including the project/program/portfolio managers, and also guide technical teams.
This series of blogs is the outcome of my contemplation and research over the last three years on the possible solution to the problem. Inspired by the DAMA wheel for data governance, I formed a Data Migration framework with ten components to encapsulate all activities from discovery to execution for successful data migration from a business-critical legacy system. Each component clubs a set of questions for stakeholders to ferret answers and compile a data migration plan that facilitates a structured and informed execution of data migration and minimises the risk of change stemming from the decommissioning.
Why we need a data migration plan?
A plan is a proactive artifact containing a set of informed decisions individuals and organisations made to move from Point A to Point B with minimised risks stemming from the change. Retiring a legacy business system can be complex and risky, demanding a thoughtful plan to clear the technology debts with minimal impact on the business continuity.
We need a plan to address, assess and mitigate risks with thought through and informed decisions. Crafting and following a data migration plan when decommissioning a critical legacy system would minimise the risk stemming from the system-wide change. ISO 31000:2018 standard for risk management that defines risk as “the effect of uncertainty on objectives”. The core reason for making a plan for data migration is the same as making a plan for your holiday, which is the assurance of a good experience and conscious avoidance of unpleasant surprises while doing the actual activity (i.e. migration and holiday).
Data migration activity is overlooked and undercooked in most cases and treated as a technology problem only. DAMA DMBoK also explicitly refers to this ignorance under the data integration and interoperability functional area of data governance. “Data migration projects are frequently under-estimated or under-designed because programmers are told to simply move data; they do not engage in the analysis and design activities required for data integration” [1]. Such ignorance of data migration poses huge risk which leads to unexpected results in new business applications, reporting issues, compliance violations, and eventually increasing project costs and delays. Therefore, I recommend crafting a strategy based on the proposed framework and drafting a plan out of it that everyone can agree to follow.
While drafting a plan for legacy data migration early in the project, one could anticipate possible risks and challenging situations beforehand. This proactive measure onboards potential stakeholders from various business units and systems to be impacted by the change. I gained first-hand experience creating and executing a data migration plan as a Data Migration Lead in the public healthcare sector. After the successful plan-based data migration project, I worked as an external data migration consultant introduced late in two similar legacy system decommissioning projects in the health sector. Therefore, I was fortunate to witness the chaos and delays in the legacy data migration projects that commenced without plans.
The data migration plan sets the foundation for well-informed and timely decision-making, leading to a coherent solution approach with no surprises on time and cost with the project progression. A legacy data migration attempt without a plan makes decisions on an ad-hoc basis on similar items covered in a planned migration. However, the unplanned approach could face limited time and cost flexibility constraints, leading to chaos, stress and reputation risk for the executives and the organisation. Unplanned data migration example — Have you ever encountered a situation where a legacy system retirement is postponed because some dependency was never known until a few days before the retirement date?
Components of the Data Migration Framework
Data migration projects retiring business-critical legacy systems usually touch several business units and require decisions from key stakeholders from business leadership. In some cases, defining the scope of historical data under the migration and building consensus on that scope could become daunting. The complex challenges of legacy data migrations can also highlight gaps and tribal knowledge in your existing data management practices. Higher tech debts with low IT and data maturity could prevent business executives from making informed decisions, introducing significant delays in executing data migration activities.
Considering all the above risk-augmenting scenarios and counting many more from the hands burnt on data migration, I compiled a questionnaire that requests information gathering and decision-making to execute data migration from a legacy system critical for business continuity. These questions with their answers were then re-factored into a plan with following ten components to effectively minimise the risk of change.
- Data Migration Scope
- Data Segments and Compliances
- People and Technology
- Business Rules
- Data Quality
- Iteration Goals
- Downstream Data Integration
- Testing Strategy
- Decommission RACI (Responsible, Accountable, Consulted, Informed)
- Post Decommission Support
The data migration plan is a living document.
You cannot expect the entire plan to be ready with all ten components as a prerequisite for commencing data migration activities such as data discovery and developing data transformation logic/code for data migration. However, before you write a single piece of code or buy expensive software licenses for data migration, a first approved version (Data Migration Plan 1.0) should be ready with the first three foundational components covered in detail.
The first version of the plan should include a high-level summary of decisions and choices for the remaining seven components. Data Migration Plan 1.0 is a strategic document for the stakeholders to review and endorse. Amend the remaining seven components with business rules and decisions made with the project progression. Multiple templates for the data migration plan are one Google search away. However, I found them overwhelming as a one-off document covering all facets of the data migration before the project initiation. My struggle in the early days (as data migration lead) in drafting a data migration plan using templates from Google search is the fundamental motivation behind this blog series.
During legacy decommissioning projects, the data migration work stream manages an unknown source (legacy system with limited support and documentation) and a moving target (ongoing customisations of the target system). Therefore, waiting for correct business rules for mapping or finalising a robust downstream integration design for the data migration plan could cause significant delays. On the other hand, data migration test runs could introduce new mapping or data quality gaps after the stakeholders endorsed the data migration plan. Such gaps would require forming business rules and risk mitigation analysis, subject to documentation (record management) in a regulatory-sensitive environment.
The first version of the plan sets the foundation for the initial test run or data migration iteration. As the project progresses, decisions made in the remaining components, each decision a valuable contribution, should be documented and agreed upon. The final version of the plan, which should be endorsed by all stakeholders before decommissioning the legacy system, serves as the organisational knowledge base. It provides retrospective information about how business data were migrated from the legacy system, underscoring the strategic value of the Data Migration Plan as more than just a courtesy document.
Next
In the following blog, I will cover the first component of the data migration plan with a set of questions it should answer for the stakeholders. I am also preparing Excel and Word templates for my readers who want to kick-start drafting a data migration plan based on my questionnaire across ten key components I listed above.
I hope decision makers (tech or business) who are directly or indirectly associated with legacy system migrations will find my legacy data migration plan series helpful. I highly appreciate constructive comments/feedback and questions. Feel free to share your thoughts and reach out for early access to data migration plan templates.
Reference:
- DMBoK — Data Management Body of Knowledge (2nd Edition) Chapter 8 — Data Integration and Interoperability — section 2.3.3
- DMBoK — Data Management Body of Knowledge (2nd Edition) Chapter 9— Document and Content Management
