Increasing number of items increases execution time exponentially

[batk0]

For the given workflow definition: https://gist.github.com/batk0/32e03d360903cdfcdb8075267419b892 execution time increases exponentially with increasing number of items.

For example:
14 items = 84s
32 items = 181s
76 items = 813s
101 items = 1566s

Also CPU usage of st2workflowengine is sustained at 100% while WF is running

[bigmstone]

@batk0 Thanks for the report. I'm actively working on a fix for this issue though it will be a little longer before it is out as it will require a refactor of some pretty involved parts of Orquesta. I will keep you posted and post against this issue as we resolve it.

[m4dcoder]

@bigmstone Just trying to help out here. I did reproduce the problem. It's caused by MongoDB write conflict in workflow service handle_action_execution_completion method. When the action executions for each item completes, they compete with each other to update the task and workflow execution. If we use coordination service to do a lock (test using file driver), then the run time increasing from 10 items to 100 items is linear. Here's the change I made to test. We can talk more on Slack this Monday. Let's think about it. If we are using coordination service to fix this, we may as well use coordination service everywhere in the workflow service instead of the write conflict and retry mechanism currently.

   720	    if task_ex_db.itemized:
   721	        from st2common.services import coordination as coordination_service
   722	        coordinator = coordination_service.get_coordinator()
   723	
   724	        with coordinator.get_lock(str(task_ex_db.id)):
   725	            # Update task execution if completed.
   726	            update_task_execution(task_ex_id, ac_ex_db.status, ac_ex_db.result, ac_ex_db.context)
   727	
   728	            # Update task flow in the workflow execution.
   729	            update_task_flow(
   730	                task_ex_id,
   731	                ac_ex_db.status,
   732	                ac_ex_result=ac_ex_db.result,
   733	                ac_ex_ctx=ac_ex_db.context.get('orquesta')
   734	            )
   735	
   736	            # Request the next set of tasks if workflow execution is not complete.
   737	            request_next_tasks(wf_ex_db, task_ex_id=task_ex_id)
   738	
   739	            # Update workflow execution if completed.
   740	            update_workflow_execution(wf_ex_id)
   741	    else:
   742	        # Update task execution if completed.
   743	        update_task_execution(task_ex_id, ac_ex_db.status, ac_ex_db.result, ac_ex_db.context)
   744	
   745	        # Update task flow in the workflow execution.
   746	        update_task_flow(
   747	            task_ex_id,
   748	            ac_ex_db.status,
   749	            ac_ex_result=ac_ex_db.result,
   750	            ac_ex_ctx=ac_ex_db.context.get('orquesta')
   751	        )
   752	
   753	        # Request the next set of tasks if workflow execution is not complete.
   754	        request_next_tasks(wf_ex_db, task_ex_id=task_ex_id)
   755	
   756	        # Update workflow execution if completed.
   757	        update_workflow_execution(wf_ex_id)

[m4dcoder]

Fix at StackStorm/st2#4595. The solution requires a distributed lock and users will have to configure the coordination service in st2 for the solution to take effect.