Data Migration Plan — Data Segments and Compliances

Drafting a data migration plan during the project initiation phase of decommissioning a legacy system requires ferreting out the data migration scope and data segments with relative compliances coupled with identifying in-house subject-matter-experts/stewards. While the scope component proposes the amount of data for migration, the segment component proposes logical partitions of the data with the order of migration and documents the due diligence activity to comply with regulations and policies while migrating the data.

Data Segment and Classifications

Data Segments

Critical business applications such as ERPs and CRMs from different vendors within the same domain (e.g., healthcare, FMGC, logistics, etc.) can vary in functionalities and features. However, the underlying data segments remain the same: Master, Referential, and Transactional. Transactional data cannot be captured in a business application until master and referential data are available [1]. Business rules and data transformation code generated during the data migration exercise express how data structures of these three types are mapped between the legacy source system and the new target system [2].

1. Master data, for instance, includes customer, patient, and employee demographics. These are the foundational details that form the basis of any business operation.
2. Referential — Lookup values such as gender codes and product category lists are referential data.
3. Transactional — An actual business transaction, such as a sales order, quote, or medical appointment, generates transactional data.

Data migration between applications is executed as multiple test runs (iterations) before the decommissioning date. The plan should propose migrating master data first during the initial test runs. For example, migrate the product catalogue and customers first, followed by historical sales orders. Aiming to migrate master data first will also touch upon the necessary referential data to be mapped between the legacy system and the target business application. Achieving master data migration early on in the initial test runs can facilitate data migration testing activities at an earlier stage of the transformation.

With the context provided above, the plan articulates the logical partition of the business data and proposes the data migration order to the stakeholders. Consensus on migrating master first, followed by referential and transactional, will steer the planning and execution of mapping rules, test runs and migration testing strategy, along with identifying key measures to monitor data migration progress.

Data Compliances

This section of the component is for the review and endorsement by stakeholders from the legal, compliance or data security teams. Once you identify the data segments , classify the segment further in applicable regulatory compliance. Usual data classification are below. Note that there may be more industry specific classifications and regulatory needs.

• Personal Identification Information (PII)
• Financially Sensitive Data
• Medically Sensitive Data/Personal Health Information (PHI)
• Educational Records

Note — I recommend to clarify the stakeholder expectations on risks and compliance before commencing any ETL development activities.

The plan addresses the regulation of the limits on retaining personal information in the first component of defining the scope. Here, the plan should describe adherence to regulations related to the use of personal information, disclosure of personal information, disclosure outside national and state legislative boundaries, and unique identifiers.

During a data migration project, business data could move out of the organisation and be subject to auditable adherence to confidentiality and regulatory compliances. Even if you are hosting the application in-house within the private network for the production, during the data migration test-runs, data may move to a sandbox environment hosted by the vendor. Thus, we investigate and address data regulations and policies early on in the plan as we identify data segments. The review process of the draft plan by the legal or risk and compliance team, a crucial step, could recommend data encryption or obfuscation step in the Extract-Transform-Load (ETL) process before the data is transmitted out of the organisation’s network boundaries, providing a comprehensive solution to potential issues.

In an environment with regulatory sensitive data and applied data governance practices, leverage existing data classifications and relative policies in the data migration plan by adding a concise tabular view of a list with relevant data regulations, data segment affected by the regulation, reference to policies followed and control implemented [3]. One example below while migrating a legacy hospital management system.

Index | Regulation          | Data Segment Affected               | Links to Policies followed             | Control Implemented in ETL 
1     | Privacy Act 2020 NZ | Master Data - Patient Demographics  | Link to published privacy policy       | Data and identifiers are obuscated before the final load
2     | ...
...

In Summary

Along with data retention adherence in the migration scope, defining the data segments with their relative compliance adherence methods in the first version of the data migration plan will address all regulatory compliance risks with respective mitigation strategies in the project’s discovery phase and instil confidence in ETL development and data transformation activities.

The second component of the plan, namely data segments and compliances, should answer the following questions of business and technical stakeholders:

• Can we identify and differentiate business data in the migration scope in logical segments and regulatory classifications?
• How will we measure data migration progress during the decommissioning project’s life cycle?
• Shall we wait for the data migration testing until all data is migrated completely to the new application, or shall we start the data migration testing in phases early on?
• How do we ensure the data migration test runs meet all applicable data compliances?
• Can we use actual data for the data migration testing?
• How can we ensure robust and extensive data migration testing to build confidence in business continuity after legacy decommissioning?

Next

In the next blog, I will cover the third component of the plan — People and Technology. Please reach out in comments if you have any question, feedback or suggestion. I am also preparing Excel and Word templates for my readers who want to kick-start drafting a data migration plan based on my questionnaire across ten key components.

I hope decision makers (tech or business) who are directly or indirectly associated with legacy system migrations will find my legacy data migration plan series helpful. I highly appreciate constructive comments/feedback and questions. Feel free to share your thoughts and reach out for early access to data migration plan templates.

Reference:

1. DMBoK — Data Management Body of Knowledge (2nd Edition) Chapter 10 — Reference and Master Data
2. DMBoK — Data Management Body of Knowledge (2nd Edition) Chapter 8— Data Integration and Interoperability
3. DMBoK — Data Management Body of Knowledge (2nd Edition) Chapter 7 — Data Security