Data Migration Engine for Power Apps
Last year the Capgemini Microsoft team released, as NuGet packages, a number of DevOps accelerators and tools that allow us to deliver high-quality Power Platform and Dynamics 365 projects. Currently we have 19 packages, with a combined total of nearly 40,000 downloads for one of these packages (Capgemini.Xrm.DataMigration.Engine).
We have now made this package Open Source, by releasing the Source Code.
One of the key things about this and all our accelerators is that they have been field-tested on numerous complex and large scale projects and have enabled Capgemini to improve development and deployment processes through the introduction of automation. The greater level of control the accelerators provide, especially around consistency of deployment, has improved reliability and quality when delivering software to production.
What is the Data Migration Engine and what can it be used for?
The Data Migration Engine is a flexible powerful engine, based on the XRM SDK which allows Dynamics 365 Configuration, Reference and Seed data to be extracted, stored in version control and loaded to target instances. The engine supports two file formats: JSON and CSV, supporting the migration of simple reference data entities (e.g. Titles, Countries) to more complex scenarios around Security Roles and Teams. Using this tool allows data to be managed in the same way as code and a release can be created that can load the required data to support the released functionality.
We created the Data Migration Engine to resolve issues encountered by the team on different projects. For example:
- Why can’t we store all Power Platform and Dynamics 365 configurations and customisation's in source control (not just code)?
- What’s stopping us deploying configurations, customisation's and data into an environment?
Consequently, the Data Migration Engine has been field tested and enhanced by our experiences implementing it on on-premise and cloud enterprise scale delivery projects for over 5 years.
Some of the features and use cases are:
- Ability to substitute GUID’s on data load
- Supports composite key lookup on data load
- Multi-pass loading for dependency management
- Mass data migration via a built-in configurable batch mechanism
- Ability to manage applications state and status — e.g. can be used to disable or enable required records
- Configurable UPSERT (one operation) or Update/Insert (at least two operations) ways of creating/updating records — as not all entities support UPSERT!
- Ability to use XML file with schema Microsoft Data Migration Format, or just bunch of Fetch XM Queries with filters to get data.
- Full support for moving business calendars, knowledge articles, duplication rules
- Can be used as a NuGet in any application to embed data export/import functionality in custom application
- Easy to extend by implementing one of the predefined interfaces
- Can be used to inject/set environment specific authentication details into Service End Points for Azure Service Bus integration during a deployment For more details and examples see the GitHub home project: xrm-datamigration
How have we used the Data Migration Engine?
Developers use the Data Migration Engine to extract Reference and Configuration data from the development environment and check it into source control. The Export JSON files holds the configuration for the engine defining the file prefix, filters, run parameters etc. The Schema XML defines the entities and attributes to be extracted. Please see GitHub home project for more details.
As part of a release pipeline, the Reference and Configuration data are loaded into the target environment using an Azure DevOps Extension of the Data Migration Engine.
This ensures that all the required Reference and Configuration data are loaded with the functionality they support, reducing the risk of deployment discrepancies as the release progresses through the delivery life-cycle.
In addition, by extracting and storing sets of test data we can set up a pipeline to create a new instance, release the system and associated Configuration and Reference Data, load a predefined set of test data and execute a suite of automated tests to verify the release.
Summary
At Capgemini we’re passionate about delivering high quality Microsoft solutions and this accelerator is an important part of our approach.
We now have a Data Migration Engine. XrmToolBox plugin that is now available on the XrmToolBox store.
Feedback welcome, please leave a comment!
Do you want to work collaboratively on interesting, re-usable solutions? Join The Capgemini Microsoft Team.
