Implement with items in task of native spec #94
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add zip function for use in expressions. The zip function is similar to python zip functions which takes x number of list and return a list of tuples where each tuple contains elements from each lists.
Add a function in utility to unescape escaped characters in string.
Add a property named `with` to the native task spec in support of with items. The `with` property in the json schema is type of string with a regular expression to match the patterns "x in <expression>" or "x, y, ... in <expression>". Add unit tests to cover validation of various patterns.
Add support for fully modeled with items with additional attributes such as concurrency in task spec. Continue to support the shorthand inline version of with items.
Add an item function for expressions to retrieve current item from the context.
Add support of with items in the conducting when the task spec is evaluated and being rendered.
Make sure that once a task has already progressed, repeated calls to get next tasks, specifically by task name, will not return duplicates.
Add context to the ExecutionEvent class. Information related to the task item that is associated with the action execution will be passed into the context for processing.
The method get_next_tasks in the conductor already serves similar role as get_start_tasks. Remove get_start_tasks to minimize confusion on use and duplication of code.
Add concurrency to the task spec, ensure conductor returns appropriate number of items in get_next_tasks, and that action execution event related to an item is processed correctly with appropriate task state change.
Add examples for single and multiple items list with and without concurrency.
…rocessed Move the execution of the engine commands until the current task is processed. Currently the engine commands identified in the task transition will be immediately executed and this may causes problems where the current task needs to be completed processing.
Add more comments, shuffle, and remove duplicate codes for getting task result, removing staged task, and evaluating task transitions.
Move the rendering logic to the task spec so it's more specific to the specific workflow definition language.
Ensure that pausing and pending an action executions of task with items are handled properly.
Ensure that canceling an action executions of task with items are handled properly.
Ensure the task with items is in running state if any related action executions is still active. This ensures that action executions related to the task in a previous cycle is not going to overwrite the item state and result for the latest cycle.
…ancel On various use cases for task with items that is active or dormant and with or without concurrency, ensure that the task state is handled properly on request to change workflow state to pause, resume, or cancel.
Codecov Report
@@ Coverage Diff @@
## master #94 +/- ##
==========================================
- Coverage 94.24% 94.11% -0.14%
==========================================
Files 37 38 +1
Lines 2102 2309 +207
Branches 400 453 +53
==========================================
+ Hits 1981 2173 +192
- Misses 72 79 +7
- Partials 49 57 +8
Continue to review full report at Codecov.
|
Add a unit test to cover basic performance on the conductor runtime as a function of the number of items in a task with items.
bigmstone
reviewed
Oct 15, 2018
orquesta/conducting.py
Outdated
| @@ -333,19 +333,20 @@ def _render_task_spec(self, task_name, ctx_value): | |||
| else items_spec.items[:items_spec.items.index('in')].replace(' ', '').split(',') | |||
| ) | |||
|
|
|||
| for item in items: | |||
| current_item = item | |||
| for i in range(0, len(items)): | |||
There was a problem hiding this comment.
Could swap for enumerate
bigmstone
approved these changes
Oct 15, 2018
orquesta/specs/native/v1/models.py
Outdated
| else items_spec.items[:items_spec.items.index('in')].replace(' ', '').split(',') | ||
| ) | ||
|
|
||
| for i in range(0, len(items)): |
There was a problem hiding this comment.
Could simplify to enumerate. Not a blocker.
There was a problem hiding this comment.
Thanks. I will refactor.
If the with items concurrency is an expression, the expression is not evaluated. Fix the issue and add unit tests coverage.
The conductor currently handles pausing and canceling states but not paused and canceled states on request workflow state. This patch adds the appropriate list of events and mappings to the task state machine.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implement the following variations for with items in task of native spec. Add support in the conductor to handle various use cases for concurrency and other state changes such as pause, pending, resume, and cancel. If no concurrency is defined, than action executions for each items will be launched immediately. Otherwise, if concurrency is defined, the conductor will direct the execution as items are completed. Also, if not concurrency is required, workflow author can use short hand shown in examples below.