networking-midonet

Invalid scenario argument: ''dict' object has no attribute 'services''` in rally jobs

Bug #1670577 reported by YAMAMOTO Takashi
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
networking-midonet
Fix Released
High
YAMAMOTO Takashi
networking-midonet 6.0.0

Bug Description

eg. http://logs.openstack.org/69/442169/6/check/gate-networking-midonet-rally-dsvm-ml2-ubuntu-xenial-nv/4ea2c4d/console.html

2017-03-07 04:11:07.296484 | 2017-03-07 04:11:07.296 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine [-] Input task is invalid!
2017-03-07 04:11:07.297677 | 2017-03-07 04:11:07.297 |
2017-03-07 04:11:07.298965 | 2017-03-07 04:11:07.298 | Subtask NetworkPlugin.create_networks[0] has wrong configuration\Subtask configuration:
2017-03-07 04:11:07.300274 | 2017-03-07 04:11:07.299 | {'runner': {"type": "rps", "times": 100, "rps": 1}, 'args': {"network_create_args": {}}, 'sla': {"max_seconds_per_iteration": 5, "failure_rate": {"max": 0}}, 'context': {"users": {"tenants": 2, "users_per_tenant": 2}, "quotas": {"neutron": {"network": -1}}}}
2017-03-07 04:11:07.301392 | 2017-03-07 04:11:07.301 |
2017-03-07 04:11:07.302766 | 2017-03-07 04:11:07.302 | Reason:
2017-03-07 04:11:07.304097 | 2017-03-07 04:11:07.303 | Invalid scenario argument: ''dict' object has no attribute 'services''
2017-03-07 04:11:07.305276 | 2017-03-07 04:11:07.305 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine Traceback (most recent call last):
2017-03-07 04:11:07.306597 | 2017-03-07 04:11:07.306 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine File "/opt/stack/new/rally/rally/task/engine.py", line 376, in validate
2017-03-07 04:11:07.307983 | 2017-03-07 04:11:07.307 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine self._validate_config_semantic(self.config)
2017-03-07 04:11:07.309421 | 2017-03-07 04:11:07.309 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine File "/opt/stack/new/rally/rally/common/logging.py", line 197, in wrapper
2017-03-07 04:11:07.311202 | 2017-03-07 04:11:07.310 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine result = f(self, *args, **kwargs)
2017-03-07 04:11:07.312461 | 2017-03-07 04:11:07.312 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine File "/opt/stack/new/rally/rally/task/engine.py", line 354, in _validate_config_semantic
2017-03-07 04:11:07.313853 | 2017-03-07 04:11:07.313 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine workloads_with_users, self.deployment)
2017-03-07 04:11:07.315143 | 2017-03-07 04:11:07.314 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine File "/opt/stack/new/rally/rally/task/engine.py", line 305, in _validate_config_semantic_helper
2017-03-07 04:11:07.316287 | 2017-03-07 04:11:07.316 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine raise exceptions.InvalidTaskConfig(**kw)
2017-03-07 04:11:07.317539 | 2017-03-07 04:11:07.317 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine InvalidTaskConfig: Input task is invalid!
2017-03-07 04:11:07.318906 | 2017-03-07 04:11:07.318 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine
2017-03-07 04:11:07.320103 | 2017-03-07 04:11:07.319 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine Subtask NetworkPlugin.create_networks[0] has wrong configuration\Subtask configuration:
2017-03-07 04:11:07.321324 | 2017-03-07 04:11:07.321 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine {'runner': {"type": "rps", "times": 100, "rps": 1}, 'args': {"network_create_args": {}}, 'sla': {"max_seconds_per_iteration": 5, "failure_rate": {"max": 0}}, 'context': {"users": {"tenants": 2, "users_per_tenant": 2}, "quotas": {"neutron": {"network": -1}}}}
2017-03-07 04:11:07.322799 | 2017-03-07 04:11:07.322 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine
2017-03-07 04:11:07.323889 | 2017-03-07 04:11:07.323 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine Reason:
2017-03-07 04:11:07.325077 | 2017-03-07 04:11:07.324 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine Invalid scenario argument: ''dict' object has no attribute 'services''
2017-03-07 04:11:07.326276 | 2017-03-07 04:11:07.326 | 2017-03-07 04:11:07.294 27601 ERROR rally.task.engine
2017-03-07 04:11:07.327607 | 2017-03-07 04:11:07.327 | Task config is invalid: `Input task is invalid!
2017-03-07 04:11:07.328773 | 2017-03-07 04:11:07.328 |
2017-03-07 04:11:07.330123 | 2017-03-07 04:11:07.329 | Subtask NetworkPlugin.create_networks[0] has wrong configuration\Subtask configuration:
2017-03-07 04:11:07.331312 | 2017-03-07 04:11:07.331 | {'runner': {"type": "rps", "times": 100, "rps": 1}, 'args': {"network_create_args": {}}, 'sla': {"max_seconds_per_iteration": 5, "failure_rate": {"max": 0}}, 'context': {"users": {"tenants": 2, "users_per_tenant": 2}, "quotas": {"neutron": {"network": -1}}}}
2017-03-07 04:11:07.332419 | 2017-03-07 04:11:07.332 |
2017-03-07 04:11:07.333634 | 2017-03-07 04:11:07.333 | Reason:
2017-03-07 04:11:07.335119 | 2017-03-07 04:11:07.334 | Invalid scenario argument: ''dict' object has no attribute 'services''`

Revision history for this message
Antonio Ojea (aojea) wrote :

The rally job was working fine on May 1st, https://review.openstack.org/#/c/199387/, and the jobs that we run internally with rally stable branch are working.

This commit seems to affect the way that rally validates tasks
https://github.com/openstack/rally/commit/4dcb597949bebaaaad3f2eacdce2f6bb9b480cb3

anyway, we need to investigate more, but most likely is because of an incompatible change in rally.

Revision history for this message
Antonio Ojea (aojea) wrote :

Thanks Takashi for show me the logstash tool, it's very useful.

I could check with the following query http://logstash.openstack.org/#dashboard/file/logstash.json?query=(message%3A%20%5C%22%5BZuul%5D%20Job%20complete%2C%20result%3A%5C%22%20OR%20message%3A%20%5C%22To%20get%20raw%20JSON%20output%20of%20task%20results%2C%20run%5C%22%20OR%20message%3A%20%5C%22Input%20task%20is%20invalid%5C%22)%20AND%20build_name%3A%20gate-networking-midonet-rally-dsvm-ml2-ubuntu-xenial-nv

that the last rally job that worked was with this commit

9e3ccfa [CI] Update hacking and fix pep8 errors

and the first job failing has this rally commit

732bbc1 Merge "[Core] Make task engine more abstract”

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to networking-midonet (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/443058

Revision history for this message
Antonio Ojea (aojea) wrote :

Workaround https://review.openstack.org/443058 until we fix our custom plugins

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to networking-midonet (master)

Reviewed: https://review.openstack.org/443058
Committed: https://git.openstack.org/cgit/openstack/networking-midonet/commit/?id=a487c8439d058004e5c0cf72c18ec348d658f8fa
Submitter: Jenkins
Branch: master

commit a487c8439d058004e5c0cf72c18ec348d658f8fa
Author: Antonio Ojea <email address hidden>
Date: Wed Mar 8 11:23:31 2017 +0100

    Disable custom plugin scenarios in rally

    The rally jobs are failing because they fail to validate
    the subtasks that use our custom plugins. It seems that
    something has changed with rally upstream and we need to
    fix our plugins in order to met the new requirements.

    This patchs remove the scenarios that are using our custom
    plugins.

    Related-Bug: #1670577

    Change-Id: I7d8430dc988a68bc6f465f7e6035237da5fb261b
    Signed-off-by: Antonio Ojea <email address hidden>

Lucas Eznarriaga (lezbar)
Changed in networking-midonet:
assignee: nobody → Lucas Eznarriaga (lezbar)
YAMAMOTO Takashi (yamamoto)
Changed in networking-midonet:
status: New → In Progress
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to networking-midonet (stable/ocata)

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/467563

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to networking-midonet (stable/ocata)

Reviewed: https://review.openstack.org/467563
Committed: https://git.openstack.org/cgit/openstack/networking-midonet/commit/?id=76e932cda6f575bcd0de723fef09d781f155798d
Submitter: Jenkins
Branch: stable/ocata

commit 76e932cda6f575bcd0de723fef09d781f155798d
Author: Antonio Ojea <email address hidden>
Date: Wed Mar 8 11:23:31 2017 +0100

    Disable custom plugin scenarios in rally

    The rally jobs are failing because they fail to validate
    the subtasks that use our custom plugins. It seems that
    something has changed with rally upstream and we need to
    fix our plugins in order to met the new requirements.

    This patchs remove the scenarios that are using our custom
    plugins.

    Related-Bug: #1670577

    Change-Id: I7d8430dc988a68bc6f465f7e6035237da5fb261b
    Signed-off-by: Antonio Ojea <email address hidden>
    (cherry picked from commit a487c8439d058004e5c0cf72c18ec348d658f8fa)

tags: added: in-stable-ocata
YAMAMOTO Takashi (yamamoto)
Changed in networking-midonet:
assignee: Lucas Eznarriaga (lezbar) → YAMAMOTO Takashi (yamamoto)
Revision history for this message
YAMAMOTO Takashi (yamamoto) wrote :

https://review.openstack.org/#/c/492376/

Changed in networking-midonet:
milestone: none → 6.0.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-midonet (master)

Reviewed: https://review.openstack.org/492376
Committed: https://git.openstack.org/cgit/openstack/networking-midonet/commit/?id=fb024e587dc073b3bc0cdb80b79edfad25dd4bf4
Submitter: Jenkins
Branch: master

commit fb024e587dc073b3bc0cdb80b79edfad25dd4bf4
Author: YAMAMOTO Takashi <email address hidden>
Date: Thu Aug 10 13:51:22 2017 +0900

    rally: Adapt to the new API

    See the following rally changes for details.

        Ife436d2167d0768a7e7fcc83460338fba2ebf236
        I539838186d2e809143adb5b4c25dfedecbf2764b
        Ib05cc0a04f9874904e9f42f4ca982bbe4e798c0b
        Ia6841068eb1d5a39a8f64f160160f1fcc1e05ac5

    Closes-Bug: #1670577
    Change-Id: I1be253557adb1857ddf870045345cef5e4871b1b

Changed in networking-midonet:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to networking-midonet (master)

Reviewed: https://review.openstack.org/492374
Committed: https://git.openstack.org/cgit/openstack/networking-midonet/commit/?id=0eba6c8bb436970307651f5b19358cd9e0627cbd
Submitter: Jenkins
Branch: master

commit 0eba6c8bb436970307651f5b19358cd9e0627cbd
Author: YAMAMOTO Takashi <email address hidden>
Date: Thu Aug 10 04:31:22 2017 +0000

    Revert "Disable custom plugin scenarios in rally"

    This reverts commit a487c8439d058004e5c0cf72c18ec348d658f8fa.

    Also,

    - Remove unused context from midonet scenarios.

    - Enlarge the subnet for for NetworkPlugin.create_routers to avoid
      "Unexpected error: increment exceeds address boundary!"

    - Comment out sla sections for some failing scenarios because their
      failure ratios seem too high for a voting job.

    Related-Bug: #1670577
    Related-Bug: #1710589
    Change-Id: I10f49177e64acb64820831f3f32eb26eca05f153

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-midonet 6.0.0.0b1

This issue was fixed in the openstack/networking-midonet 6.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on networking-midonet (master)

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/447054
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.