Data Migration Plan – Business Rules
In this series of blogs, I have concluded the significance of the data migration plan and how to draft one using the data migration framework with ten components, with firth three components covered in detail so far. While the first three components of the data migration framework set the foundation, from here the focus is on the actual operational nuances of moving the data into the new target system.
Data migration is about moving data from one structure to another, which requires mapping individual data entities in the target system with the data entities (tables) of the source (legacy) system. The data structure of the target system could be tables and columns in an actual SQL database, a bunch of Excel files (as templates) or REST API endpoints to push legacy data into the new system. Whatever the means on the target side, each attribute/field/column of the target table/Excel/APIs must be mapped with the legacy data structure.
Mapping Scenarios
DMBoK define mapping as a synonym for transformation that addresses both the process of developing the lookup matrix from source to target structure and the result of that process. “A mapping defines the source to be extracted, the rules for identifying data for extraction, targets to be loaded, rules for identifying target rows for update, and any transformation rules or calculations to be applied.” [1]
In some cases (especially master data), this mapping could be simple with a one-to-one match between the source and target system (for example, customer first name). However, mapping could get complex when the target data structure demands the value to be derived or transformed from one or multiple fields (of one or multiple tables) of the source system. On the other hand, reference data (such as Gender and product Type) would differ between legacy and target systems, requiring a lookup structure to derive the correct target value from the source value (i.e. M in the source is MALE in the target). It is also expected to find mandatory fields/columns in the target system that are unavailable in the years-old legacy system, leading to a hardcoded default value for the migrated data.
The screenshot below is a highly simplified example of capturing mapping between source and target data structures for a few target fields/columns when migrating patient data. SMEs should review and validate this mapping before database developers can implement it in the ETL code.
Mapping = Business Rules
Whether complex or simple, the mapping between source and target data structure is a bunch of business rules that control data transformation from source to target system. A business could need these business rules retrospectively post-decommission for various analytical or compliance needs. Thus, it must be captured, curated, and approved by SMEs or stakeholders and made available in a human-readable form. Poorly executed data migration where the mapping between source and target structure is seen as purely a technical job would burry all such business rules in the data transformation code without any scrutiny for transform logic or default values for the mandatory fields, causing significant risk in meeting analytical needs and disrupting downstream data feeds to integrated systems.
The rules defined to map the target data structure with the source data structure (i.e., legacy system), to transform or cleanse the data during migration, de-duplication rules, etc. must be explicitly validated and documented as business knowledge. Documenting business rules for such a major transition would not only add value as metadata generated from the data migration but also serve as a reliable reference for future analytical or regulatory needs that demand data aggregation from the legacy and current systems.
When buried in data transformation code (and in the developer’s head who no longer works here), aggregation of legacy and current data would depend on a massive reverse-engineering exercise, causing doubts about the integrity of the combined dataset. Therefore, the value of your work in documenting these business rules cannot be overstated.
DMBoK identifies collecting business rules as a critical activity for the data integration stream, including data migration. According to DMBoK, a business rule is a statement that defines or constrains an aspect of business processing while asserting the business structure and influencing the business’s behaviour [1]. Hence, I considered “Business Rules” as a standalone component of the plan that informs stakeholders about how business rules are validated and how business knowledge is retained for future needs.
Please refrain from attempting to list all business rules in the plan, as they will emerge and develop as the data migration project progresses. Instead, focus on including information about how business rules are validated, who (SME’s) is responsible for the validation, and provide a reference to the location where business rules are documented.
Depending on the culture, budget, and preferences, one can capture business rules in a simple Excel file saved in Sharepoint or opt for a sophisticated business rule engine that allows non-technical users to manage business rules.
In Summary
Leverage the fourth component of the data migration framework to create a process for curating and capturing such critical business knowledge generated due to the decommissioning of the legacy system. This component addresses the following questions of stakeholders:
- Are we migrating all critical data elements in the legacy system into the new system for regulatory, financial and operational needs?
- How are we handling mandatory data in the target system that is unavailable in the source system?
- What is the process for defining and validating mapping between source and target as business rules?
- What is the process for finding information about how we migrated legacy data after the decommissioning?
- What are the criteria to select correct customer /product/ patient (Master data) when duplicates are found within one legacy system or between multiple instances of the legacy systems?
Next:
In the next blog, I will cover the fifth component — Data Quality. Please reach out in comments if you have any question, feedback or suggestion. I am also preparing Excel and Word templates for my readers who want to kick-start drafting a data migration plan based on my questionnaire across ten key components.
I hope decision makers (tech or business) who are directly or indirectly associated with legacy system migrations will find my legacy data migration plan series helpful. I highly appreciate constructive comments/feedback and questions. Feel free to share your thoughts and reach out for early access to data migration plan templates.
Reference:
- DMBoK — Data Management Body of Knowledge (2nd Edition) Chapter 8 — Data Integration and Interoperability
