The pains of a Microsoft Dynamics migration

Captured by myself — tdashworth

Any data migration is difficult and many errors are inevitable no matter the size or complexity of it. There are also many different resolutions for these and understanding them early on will help you along your own journey. Make sure you allocate a rational amount of time within estimates for errors because they will occur.

In a recent project, I developed a migration package to move data from an on-premise Dynamics CRM 2011 to an online Dynamics 365 (Version 9). To perform the migration, we selected SSIS paired with KingswaySoft’s Dynamics connectors due to its flexibility and ease of use.

I will explore some of the errors I received while loading the transformed records into Dynamics 365. These were thrown by Dynamics’ 365 API which Kingsway Soft implements so., I will explain the reason I believe the error was thrown and solutions to resolve the issue.

The record does not exist

<Entity Name> With Id = <GUID> Does Not Exist

This occurs when referencing a record of a specific type that doesn’t exist in the destination environment which could be because of incorrect ordering of entities or a record failed while migrating in a previous step.

It is important to understand the relationships between different entities being migrated. For one-to-many relationships between two different entities, ensure the ‘many’ entity is migrated before the ‘one’ entity. Where this isn’t possible due to circular relationships or one-to-many relationships between the same entity then the process should be split into two stages.

Create the necessary entities ignoring the field(s) regarding the erroring relationship(s) then, once those entities have successfully loaded, perform an update to populate the previously ignored fields. Dynamics doesn’t allow updates to ‘disabled’ records, so ignore the Status and Status Reason fields (which are active by default) then populated those within the update.

Attachment file size is too big

Attachment file size is too big.

Dynamics has a configurable file size limit to prevent large attachment (stored within the Annotation and Email entities) to be uploaded. This can be modified from the default of 5MB to 128MB via Settings > Administration > System Settings > Email and adjusting the ‘Maximum file size (in kilobytes)’ setting.

The user does not have send-as privilege

User does not have send-as privilege.

When loading email records with the status ‘Pending’, Dynamics tries to send them, but as the executing user could be different to the owner/sender an error is thrown which fails the record. A simple solution would be to give all user that have pending emails send-as privilege although you might not want to open that security up.

Another solution is when transforming those record statuses to ‘Draft’ and deal with them post-migration, but we decided to delete those few emails from the source because they were considered junk.

Expected non-empty GUID

Expected non-empty Guid.

This error was discovered in activities, specifically the party lists including a missing contact. The format of a Party List record includes the Entity Type Code (ETC) and GUID of the participant in a JSON object when using Kingsway Soft. By ‘missing contact’ I mean record with an ETC of 2 (Contact) and a GUID of ‘00000000–0000–0000–0000–000000000000’ which Dynamics can’t associate upon loading. Our resolution involved transforming the ETC to 0 for these records which are rendered as a red link in Dynamics. The Party List object also stores the email address used, this is what will be displayed as the red link text.

Internal implementation error

Internal Implementation error id: <Record GUID>, contact IT support

This error occurred within plugins during creating or updating of records. You can identify the root cause by viewing the trace logs or performing the action manually where you can apply a fix. Another solution is to disable all plugins and workflows before executing the migration then enable them after completion which is the route we choose.

A validation error occurred

A validation error occurred. The length of the ‘<Field Name>’ attribute of the ‘<Entity Name>’ entity exceeded the maximum allowed length of ‘<Max Length>’.

This is regarding system validation (types, length, etc). While developing the migration package, we ensured the staging table met the same requirements as the destination system which puzzled us when receiving this error. The first check examined the specific failed data to ensure it was of the correct length.

I later discovered that the plugins were left enabled and one modified the record title adding additional length which failed the validation. We resolved this by turning off all our plugins but there is no one solution to this but to check the affected data being migrated and the processes that could modify that data.

Creating an entity with an invalid parent

Creating Entity with an invalid parent. Entity: <Entity Name>, ReferencingAttribute:regardingobjectid

Like ‘Record does not exist’ above, this occurs when you are referencing a record that simply doesn’t exist and can be solved using the same methods. Is the order of entities being migrated valid? Was it a previously failing record? Is it the same entity type requiring a two-stage load to ensure all records exist before referencing them?

Timeout

The request channel timed out while waiting for a reply after <Some amount of time>. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. (Error Type / Reason: Timeout)

The Kingsway Soft Dynamics connectors use the Dynamics 365 web API making the appropriate calls and waiting for a response to verify the transaction. This error is related to this process and has been the most temperamental due to inconsistencies with our network, the data and Dynamics’ servers.

Ensure you have a stable internet connection during the migration but the biggest variable which you can control is the rate of flow to Dynamics. Kingsway Soft provides a few options around this which you should experiment with, ideal with real data or at least real volumes of data. Those options are:

• the batch size of each transaction,
• the number of threads processing data, and
• the timeout the connection is willing to wait for dynamics to respond.

The batch size and thread count can be found in each Dynamics task and we used a batch size of 200 as our baseline, reducing where necessary and one or two threads. The timeout we left as the default, but this can be found in the connection settings for Dynamics. Your own mileage may vary with these, but I suggest the reading Kingsway Soft’s recommendations and experimenting yourself.

Generic SQL error

Generic SQL error.

Contact Microsoft Support and good luck 😊. Seriously though, you can’t get more generic than this and after many hours head scratching we gave in to contact Dynamics’ support who provided us with the crucial SQL log which only they have access to. To note, if you are running on-premise then you can view those logs yourself.

Missing user privilege

Principal user (Id=<User’s GUID>, type=8, accessMode=<User’s Access Mode>, roleCount=<User’s Role Count>) is missing <Privilege Name> privilege (Id=<Record GUID>)

I encountered this error when I referenced users that weren’t set up correctly. Identify the user with the GUID stated and check the roles they have applied to them. Then check the security permissions for those roles to validate they meet the requirements for the migration. This might raise some questions if the security model requires a change but, like in our situation, a temporary role could be assigned during migration to allow the old data to be imported.

Summary

Dynamics 365 throws a mix of helpful, specific errors and vague, generic ones. Most are due to your data or process and that should be the first avenue to investigate, some are related to the destination system and the connection to it. This post doesn’t cover this but ensuring the migration package is well designed and structure is fundamental to easily investigating why you could be seeing these errors.

I hope you found help and or guidance within this post to solve any issues you’re facing in your own migration of data to Dynamics 365 Online. This is not a comprehensive list of solutions but some pointers to help. Therefore, I encourage you to comment below any solutions I haven’t mentioned regarding these errors.

Join the Capgemini Microsoft team, open roles here.