There are a number of tricks that can help improve the speed of data migrations. I have bullet pointed them below and included links and references at the bottom of the page.
Turn off all workflows (including business rules).
Turn off Plugins
Custom plugins can be turned off.
Microsoft plugins need to stay turned on however in my experience there are a number of Microsoft plugins that can be turned off that will help speed up the migration. These plugins are mostly related to Marketing Segments, Marketing Customer Journey and GDPR. I am sure there are other plugins that can be turned off, especially depending on which first-party apps your environment has installed.
Microsoft.Dynamics.Crm.MarketingPlugins.Plugins.Segmentation.SegmentMembersPlugin
Microsoft.Dynamics.Gdpr.Consent.Plugins.GdprConsentChangeLoggingPlugin
Microsoft.Dynamics.Cxp.OrchestrationEngine.Plugins
Microsoft.Dynamics.EventManagement.CrmPlugins.EventMgmtPlugin.Plugins
Use the BypassCustomPluginExecution option (see link below for more detail).
Turn off wall posts
Request Microsoft to Scale the front end servers. Yes, this can be done and I have done it for a number of clients.
Batch and multithread your data migration to the edge of the API Limits. Use the Task Parallel Library (Parrallel.Foreach).
Reduce latency by hosting your migration at the same Azure Data Centre as the Dataverse tenant.
Use multiple users to perform the data migration to exceed API limits. The API limits are applied per user. (6000 requests per user per 5-minute window)
Turn off cookie affinity (to optimise BAU requests the same affinity cookie is used on all requests from the same source. This causes all requests to go to the one front end server, which will have cached data. This is great for BAU performance but bad for data migrations as the load will be on one front end server INSTEAD of balanced overall front end servers.
Ask Microsoft to re-index the SQL Server for your Dataverse environment. Microsoft Prem Support can do this but it might be a battle to get the request approved. If you have a Microsoft Fast Track engagement request it through Fast Track.
I have personally done all the above performance tuning options with great results. The above performance tips and additional performance tips are discussed in detail at https://docs.microsoft.com/en-us/power-apps/developer/data-platform/api-limits#how-to-maximize-throughput.
Feel free to contact me if you need help optimising your migration or integration.
Terminology and Links
BypassCustomPluginExecution Option - https://docs.microsoft.com/en-us/power-apps/developer/data-platform/bypass-custom-business-logic?tabs=webapi
Dataverse API Limits - https://docs.microsoft.com/en-us/power-apps/developer/data-platform/api-limits
Sample: Task Parallel Library with CrmServiceClient - https://docs.microsoft.com/en-us/power-apps/developer/data-platform/xrm-tooling/sample-tpl-crmserviceclient
API Limit Workaround - https://docs.microsoft.com/en-us/power-apps/developer/data-platform/api-limits#how-to-maximize-throughput
Comments