New bulk workflow execution tool for Microsoft Dynamics CRM 2011 overcomes common data management hurdle

With a little help from his friends, Andy Popkin, technical consultant at Zero2Ten has released a new Microsoft Dynamics CRM 2011 Bulk Workflow Execution Tool aimed at overcoming limitations of other methods of applying workflows to manipulate CRM data.

"Ever since I started working with Dynamics CRM, I have always been searching for ways to become more efficient when it comes to those repeatable tasks we all love doing every day," Popkin said in a blog post.

Popkin admits that as much as he loves CRM, it still has its limitations - limitations that users do their best to find ways around.   

Using CRM workflows for data manipulation is a common practice.  The problem is that running the same workflow for 25,000 records takes forever to do, page by page, 250 records at a time.

"I would rather do the bug eating challenge on Fear Factor," he said.

So like most innovators, Popkin figured that there had to be a better way to get the job done.

"I came up with my first solution which was 100% CRM based, but the solution did not work well with due to plug-in timeouts," he said. "So I decided to make things run on the client side and I created the Bulk Workflow Execution Tool."

Here's how it works: The CRM 2011 Bulk Workflow Execution Tool lets you to run any On Demand workflow in CRM 2011 (on-premise and online) for any view (system or personal) for the entire record set returned by that view.

"Need to send an email out to all leads created in the last seven days, but send a different email based on region? Create your workflow, create a view, and within five clicks you are done," Popkin said. "The Bulk Workflow Execution Tool will use the record set returned by your view's FetchXML, and execute the On Demand workflow for each record, saving you the hassle of going page by page for who knows how long. You may also use a custom FetchXML query if needed."

Gus Gonzalez, Solution Architect and Microsoft Dynamics CRM MVP at Zero2Ten, explained a little more about ways this tool can save time.

“A couple of weeks ago I had to process 200,000 records with a workflow that would connect to the parent record, review the value of a field and update the child with data depending on the parent’s value,” he said. “This is a perfect task for a workflow but the problem is that until now we had to sit on a computer and run 250 at a time, it takes about 20-30 seconds to submit 250 workflows manually to the database so if you take 200,000 and divide that by 250, we are talking about hitting those three to four steps 800 times, multiply that for 30 seconds and it comes to more than 6 and a half hours of work. Let’s just say, a whole day sitting in front of a computer to get that done. There were other ways but they all took a lot of time, even if it reduces the wait from 6 to 2 hours.”

This tool gives the CRM administrators, consultants and solution architects the ability to execute thousands and thousands of workflows automatically and without the need to sit in front of a computer and submit 250 workflows at a time, Gonzalez said.

“The tool submits the workflows via system call and we have seen that it submits about 1,000 workflows every 20 seconds,” he said. ‘The best part is that you don’t have to pay attention to the tool running in the background, which means that I can start the tool and submit those 200,000 workflows while I can focus on something else. It is very rare to work in a project where you will not need to run workflows in mass so this tool will save a lot of time to CRM professionals around the world and reduce the cost to implement CRM for a lot of customers.”

Popkin said if you're currently running Update Rollup 12 or higher, the Bulk Data API will be enabled - allowing the workflows to be created in batches of 200. He said this is an average of 5x faster than "pre-UR12" environments.

Popkin posted a video to prove just how fast the tool can help you run workflows for 2,500 records in less than one minute.

There are plans to add some features to the tool in the near future to make the process of running workflows even simpler, such as a workflow scheduler. And as soon as CRM 2013 arrives, this tool will be updated to be compatible, he added.

Popkin gives a shout out to Jennifer Ford at Sonoma Partners for helping with the QA on this tool as well as his Zero2Ten colleague Gus Gonzalez, for pushing him to finish the tool. Gonzalez has been doing his own tests on the performance of the tool and was able to run through 20,000 records which took 38 minutes to submit and 1 hour and 52 minutes to run within Dynamics CRM Online.