10 Tips to Speed Up your Data Migration

There are a number of tricks that can help improve the speed of data migrations. I have bullet pointed them below and included links and references at the bottom of the page.

  1. Turn off all workflows (including business rules).

  2. Turn off Plugins

  3. Custom plugins can be turned off.

  4. Microsoft plugins need to stay turned on however in my experience there are a number of Microsoft plugins that can be turned off that will help speed up the migration. These plugins are mostly related to Marketing Segments, Marketing Customer Journey and GDPR. I am sure there are other plugins that can be turned off, especially depending on which first-party apps your environment has installed.

  5. Microsoft.Dynamics.Crm.MarketingPlugins.Plugins.Segmentation.SegmentMembersPlugin

  6. Microsoft.Dynamics.Gdpr.Consent.Plugins.GdprConsentChangeLoggingPlugin

  7. Microsoft.Dynamics.Cxp.OrchestrationEngine.Plugins

  8. Microsoft.Dynamics.EventManagement.CrmPlugins.EventMgmtPlugin.Plugins

  9. Use the BypassCustomPluginExecution option (see link below for more detail).

  10. Turn off wall posts

  11. Request Microsoft to Scale the front end servers. Yes, this can be done and I have done it for a number of clients.

  12. Batch and multithread your data migration to the edge of the API Limits. Use the Task Parallel Library (Parrallel.Foreach).

  13. Reduce latency by hosting your migration at the same Azure Data Centre as the Dataverse tenant.

  14. Use multiple users to perform the data migration to exceed API limits. The API limits are applied per user. (6000 requests per user per 5-minute window)

  15. Turn off cookie affinity (to optimise BAU requests the same affinity cookie is used on all requests from the same source. This causes all requests to go to the one front end server, which will have cached data. This is great for BAU performance but bad for data migrations as the load will be on one front end server INSTEAD of balanced overall front end servers.

  16. Ask Microsoft to re-index the SQL Server for your Dataverse environment. Microsoft Prem Support can do this but it might be a battle to get the request approved. If you have a Microsoft Fast Track engagement request it through Fast Track.

I have personally done all the above performance tuning options with great results. The above performance tips and additional performance tips are discussed in detail at https://docs.microsoft.com/en-us/power-apps/developer/data-platform/api-limits#how-to-maximize-throughput.


Feel free to contact me if you need help optimising your migration or integration.

 

Terminology and Links

  1. BypassCustomPluginExecution Option - https://docs.microsoft.com/en-us/power-apps/developer/data-platform/bypass-custom-business-logic?tabs=webapi

  2. Dataverse API Limits - https://docs.microsoft.com/en-us/power-apps/developer/data-platform/api-limits

  3. Sample: Task Parallel Library with CrmServiceClient - https://docs.microsoft.com/en-us/power-apps/developer/data-platform/xrm-tooling/sample-tpl-crmserviceclient

  4. API Limit Workaround - https://docs.microsoft.com/en-us/power-apps/developer/data-platform/api-limits#how-to-maximize-throughput


15 views0 comments

Recent Posts

See All

Power Platform storage seems to get consumed very quickly. I have had to come up with capacity management strategies for numerous clients over the years. I really do wish Microsoft would increase the