Recursive generation of YAQL objects

Bug #1680109 reported by Xavier Hardy
18
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Mistral
Triaged
Medium
Unassigned

Bug Description

As a workflow/action designer, I should be able to define variables that I can re-use in other expressions in the same task.

For instance, with "publish":

```yaml
version: 2.0

my_workflow:
  tasks:
    start:
      action: std.noop
      publish:
        my_first_variable: 1
        child_variable: <% $.my_first_variable + 1 %>
        gd_child_variable: <% $.child_variable + 2 %>
```

This task should publish:
```yaml
my_first_variable: 1
child_variable: 2
gd_child_variable: 4
```

This can be done using a dependency graph/topological sort. During the dependency graph generation, cycles can be detected and reported as an error. The variables have to be generated according the result of the topological sort.

Tags: feature
Revision history for this message
Nick Maludy (nmaludy) wrote :

So, i've run into this same issue myself.

From my investigation i believe it's caused by mistral not updating the context used for expression evaluation after each publish statement is evaluated. Instead the context is created at the beginning of the publish block and then merged into the global context upon completion of the task.

The task begins publishing variables here:
https://github.com/openstack/mistral/blob/master/mistral/engine/tasks.py#L164

Then a context is created and evaluation begins:
https://github.com/openstack/mistral/blob/master/mistral/workflow/data_flow.py#L187-L217

Evaluation is triggered by this line:
https://github.com/openstack/mistral/blob/master/mistral/workflow/data_flow.py#L209

Evaluation is done by the following:
https://github.com/openstack/mistral/blob/master/mistral/expressions/__init__.py#L88-L103

It's in this final code snippet where you can see that the evaluation simply creates a new data structure (dict) and does not merge the result back into the current context being used for evaluation.

This is my first time looking at the code base so i may be off in left field.

Any feedback is appreciated.

Dougal Matthews (d0ugal)
Changed in mistral:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Xavier Hardy (xavier-hardy) wrote :

I have already done this in another project with variables generated from Jinja2. Basically, what we could do is:
- extract all of the variables inside the YAQL code
- generate a dependency graph using the variable names, it is possible to detect dependency loops at the same time. Raise an error if there's a loop (it could be possible to support loops, but I don't think it's good idea).
- compute the variables in the order of the dependency graph, recursively

I just need to spend more time studying the code that interprets the "publish" sections in the DSL.

tags: added: feature
Revision history for this message
Xavier Hardy (xavier-hardy) wrote :

nmaludy:

https://github.com/openstack/mistral/blob/master/mistral/expressions/__init__.py#L88-L103

'evaluate_recursively' does not recursively evaluates the YAQL code as we would like. It just goes through the data dictionary recursively.

It allows you to put YAQL in non-string types:

```yaml
version: 2.0

my_workflow:
  tasks:
    start:
      action: std.noop
      publish:
        my_first_variable:
          child_variable: <% $.stuff + 1 %>
```
instead of only being able to use YAML strings in published variables like this:
```yaml
version: 2.0

my_workflow:
  tasks:
    start:
      action: std.noop
      publish:
        my_first_variable: <% dict(child_variable=>$.stuff + 1) %>
```

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.