Upgrading large Microsoft Dynamics 365 Customer Engagement on-prem deployments to online

March 21 2019

Despite the fact that most of the communication that Microsoft does regarding Dynamics 365 Customer Engagement focuses on the Online version, there is still quite a large share of customers still using on-premises versions of Dynamics CRM or D365CE. The reasons for this may vary. In some cases, it may be tied to regulatory compliance. Others perceive the security of the online service as less than that of a system hosted in your own data center (a notion that definitely could and should be challenged). However, something also holding many customers back is the fact that they have large complex implementations and they fear that upgrading will be too complicated, take too much time, cost too much money, and be too risky.

I was the solution architect behind an upgrade for a large CRM 2011 system directly to D365CE Online. It had a database of over 300 GB, more than 50 custom entities, some tables containing more 30 million records, more than 10 integrations, and over 100 reports. We started on May 1, 2017 and the organization was up and running in the online version by December 15, 2017. This article will briefly describe some of the takeaways from this project.

Do not assume – know

We started this project by being handed a pre-study recommending an upgrade to on-prem for three reasons:

  • The database was very large—over 300 GB
  • Over 100 reports
  • Many integrations

I do not scare easily, so I started by asking some technically savvy people at Microsoft about the 300 GB database. It turned out that there were quite a few customers in D365CE Online with databases of this size. Yes, you must pay for the extra size, and yes, it does cost more than what you pay for a 1 TB personal OneDrive, but on the other hand, it is as SSD level SQL database stored in at least two geographical locations plus short-term and long-term backup features and calculation costs included. So, in order to be fair, you must compare apples with apples. I also talked to the customer and they did not want to migrate all of the data due to GDPR considerations.

As for the reports, this can really be a problem. When I presented on this subject in a webinar once, I asked what the main reason was for people not upgrading to the cloud, and the majority answered that it was due to complex reports. However, I asked the customer about this and they answered that they only used a maximum of four reports. The reporting functionality has extended a lot over time and now includes Power BI and more. I thought it relevant to ask about what these four reports were. It turned out that they didn't have to be reports at all. When we understood the real business requirement, we saw that one of them turned out to be solvable by just creating a view and putting this view and entity in a Modular App for a specific user group. The customer was excited and they said, "It is much better! Now we can see it real time!". Previously, someone had emailed the users the PDF export of the report. While this may not be a general solution, do try to understand what the business really needs and drill into this and solve it at its root. Often it doesn't have to be a report. Common solutions to replacing a complex SQL report include:

  • Precalculate in Dynamics
    • Report with SSRS and FetchXML data source
    • Use a view
    • Use a chart
  • Export data to Azure SQL with Data Export Service, aggregate, view with Power BI or similar
  • Use document templates for Word or Excel or even Document Core Pack from mscrm-addons.com to create more document-based reports

The last issue that was put forward as a main reason for not going to the cloud was that there were many integrations. I challenged this by saying that there generally was no problem integrating with D365CE Online compared to on-prem. There are some particulars like throttling limitations that need to be considered, but these are quite steep (60,000 API calls every five minutes). Using the Data Export Service and doing parts of the reading from the Azure SQL database can also be both faster and a good way to offload the API a bit. In this case, we were upgrading from CRM 2011 so we had to revise all the code anyway and hence the added overhead to move it online was not too large.

Approaching the challenge

As for the general strategy used, I have adopted a strategy I call 1:1 upgrade. I picked it up from the gurus Joel Lindstrom and Scott Sewell during one of their sessions on upgrades at the CRM User Group Summit. Essentially, it means when upgrading you should not try to fix things, just try to upgrade. I like to make the analogy with refurbishing a house. If you are redoing the kitchen, focus on that, don't start doing the bathroom and bedrooms too.

In concrete steps, the 1:1 method was realized in the following ways:

  1. Export the solution from CRM 2011
  2. Strip out code and reports
  3. Upgrade the solution to Dynamics 365 (8.2) via Dynamics CRM 2013 OnPrem (latest version)
  4. Revise forms and other UI logic for 8.2
  5. Upgrade code, component by component, and add back into the solution
  6. Upgrade integrations, installed in a new hosting environment
  7. Migration – as soon as 8.2 solution became fixated from a data model perspective, the migration could be started
  8. Test, test, test – and of course fix based on testing

What is important to know is that since the time of this implementation, Microsoft has launched the Dynamics FastTrack program to assist moving a database, by moving the entire database, including customizations to the online system. This is an entirely different procedure, by no means a "next-next-finish" solution, especially for a complex deployment such as this.

Key project decisions

We were using an agile project methodology based on Kanban when implementing this and it worked well, in hindsight. But there were some interesting issues we had. One was that my customers business has seasonal ups and downs. This meant that we either had to be done before Christmas or wait until April for another window which would fit the business for changing the system. This type of scheduling requirement is the case for most businesses, which cannot simply adjust to the "we'll be done when we'll be done" kind of mindset that sometimes comes with an Agile team. I will probably be hit with a ton of bricks by many Agile experts and I would love to have a discussion around how to handle that.

Another lesson we learnt was that the consultant we had utilized for doing the migrations was a seasoned SSIS expert who I taught the Kingswaysoft adapters and how the D365CE data model works. The advantages of this was that he was really skilled in setting up very advanced SSIS flows and utilizing SSIS features to a large extent as well as optimizing performance so that the actual downtime was reduced. However, not knowing the peculiarities of Dynamics is a distinct handicap for any developer, be they an SSIS developer or C# developer.

One bug that took me some time to find and that he couldn't understand was that objecttypecode for custom entities in the CRM 2011 and the new system were different, and we hence had to have a mapping for these to make sure that any polymorphic relationship, like for instance the regarding field, connected to the right entity.

His contract was also a bit short, as we expected the migration do be done after go-live. However, real life turned out to be a bit trickier than that and we still had some non-critical data left, for which he and I had to spend some late nights "burning the SSIS oil."

SSIS and Kingswaysoft as tools have, however, been a real hit with this customer, even after moving to the cloud. Previously, they sought some tools for bulk management of their data, which they now have.

Was cloud the right choice?

As I mentioned in the beginning, the customer had many ideas of other things they wanted to do while we upgraded, and we were pragmatically firm on not adding any new functionality that we didn't have to add. After the move to the online deployment, many new features and capabilities opened up for the customer, which put some of these old requests into perspective. As a result, some plans have been put on the backburner and many new ideas have been implemented during the last year since the move to online. Examples include new features such as:

  • 100+ surveys using Voice of the Customer – many surveys used as customer facing forms
  • Sentiment analysis of surveys and other data using Azure Cognitive Services
  • Profanity checking of survey responses using Flow and Azure Content Moderation
  • D365CE gamification
  • Change of the marketing automation system (DotMailer)
  • Evaluation of Dynamics Marketing
  • Evaluation of Microsoft Social Engagement
  • GDPR features
  • BI integrations

A small team simultaneously maintaining and supporting the system helped the business achieve its goals. This could not have been achieved without the move to the cloud because the cloud services enable high modularization and decoupling, making it easy to connect very powerful features using often no or low amounts of code. Independent modules and low amounts of code also make for a system that is easy to maintain with fewer dependencies, as long as that is an identified goal of the implementation.

In conclusion, I stress my key questions that I believe should be asked when discussing on-prem versus online: If D365CE Online offers so much value, what are the reasons for not choosing it? Then, challenge those reasons.


Photo by Viviana Rishe on Unsplash

FREE Membership Required to View Full Content:

Become a MemberLogin
Joining MSDynamicsWorld.com gives you free, unlimited access to news, analysis, white papers, case studies, product brochures, and more, and it’s all FREE. You’ll also have the option to receive periodic email newsletters with the latest relevant articles and content updates. Learn more about us here
About Gustaf Westerlund

Gustaf Westerlund is a 7xMVP and MSc EE working with Dynamics 365 since version 1.2. He primarly works as solution architect and helps many customers bridge the gap between business and technology. With his very broad and deep technical knowledge centered around Dynamics 365 he is often called in to solve issues not being solved by others. He is a public speaker and often speaks at the Dynamics 365 events around the world. He is the founder of the consultancy where he works, CRM-Konsulterna, located in Stockholm, Sweden.

More about Gustaf Westerlund


Srikanth Paladugula's picture

Thank you Gustaf Westerlund for sharing your experience, recently I did the same kind of migration(>100GB) using fasttrack process which solves most of the issues we had during data migration and makes the life simpler for Ground to cloud migrations.