OpenStack Compute (nova)

AttributeError: 'HostState' object has no attribute 'failed_builds'

  1. Series stein
  2. Bug #1834691
Bug #1834691 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Matt Riedemann
Pike
Fix Released
Medium
Lee Yarwood
Queens
Fix Committed
Medium
Matt Riedemann
Rocky
Fix Committed
Medium
Matt Riedemann
Stein
Fix Committed
Medium
Matt Riedemann

Bug Description

I'm not sure how this happens, but I hit this in a devstack environment with 35 nova-compute services using the fake virt driver, one scheduler worker and then trying to schedule a server:

Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server [None req-91eb7095-ab0e-4c51-ba65-669863c08cbf admin admin] Exception during message handling
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 165, in _p
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, i
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, i
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 228, in in
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server return func(*args, **kwargs)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/manager.py", line 192, in select_destinations
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server allocation_request_version, return_alternates)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 96, in select_destinations
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server allocation_request_version, return_alternates)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 210, in _schedule
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server hosts = self._get_sorted_hosts(spec_obj, hosts, num)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 449, in _get_sorted_hosts
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server spec_obj)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/host_manager.py", line 600, in get_weighed_hosts
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server hosts, spec_obj)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/weights.py", line 137, in get_weighed_objects
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server weights = weigher.weigh_objects(weighed_objs, weighing_properties)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/weights.py", line 107, in weigh_objects
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server weight = self._weigh_object(obj.obj, weight_properties)
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/scheduler/weights/compute.py", line 36, in _weigh_object
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server return host_state.failed_builds
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server AttributeError: 'HostState' object has no attribute 'failed_builds'
Jun 28 19:49:31 devstack nova-scheduler[23128]: ERROR oslo_messaging.rpc.server

The HostState.failed_builds attribute gets set outside of __init__ in _update_from_compute_node here:

https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L261

And I'd expect the _update_from_compute_node to get called from update() which is called from here:

https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L787

Ah bingo if the compute node hasn't had resource usage updated yet we don't set the failed_builds attribute:

https://github.com/openstack/nova/blob/231908a7f42b8329a58e62c8a89204fdeaa16dca/nova/scheduler/host_manager.py#L191

Tags: scheduler
Revision history for this message
Matt Riedemann (mriedem) wrote :

The HostState.failed_builds change was introduced in Rocky and backported through to Pike:

https://review.opendev.org/#/q/I71c56fe770f8c3f66db97fa542fdfdf2b9865fb8

Changed in nova:
status: New → Triaged
importance: Undecided → Medium
assignee: nobody → Matt Riedemann (mriedem)
tags: added: scheduler
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/668243

Changed in nova:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.opendev.org/668252

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/668243
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d540903463aa9b0cf69cefac7cc60e5b70e40a1c
Submitter: Zuul
Branch: master

commit d540903463aa9b0cf69cefac7cc60e5b70e40a1c
Author: Matt Riedemann <email address hidden>
Date: Fri Jun 28 16:19:49 2019 -0400

    Init HostState.failed_builds

    If _update_from_compute_node returns early and the HostState is
    not filtered out we can hit an AttributeError in the
    BuildFailureWeigher because the failed_builds attribute is not
    set. This simply initializes the attribute like the other stats
    fields.

    Change-Id: I5f8e4d32c6a1d6b61396b4fa11c5d776f432df0c
    Closes-Bug: #1834691

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/668282

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/stein)

Reviewed: https://review.opendev.org/668282
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=725b37f515e4ad01e1f4491b6d8137ce1416f6d6
Submitter: Zuul
Branch: stable/stein

commit 725b37f515e4ad01e1f4491b6d8137ce1416f6d6
Author: Matt Riedemann <email address hidden>
Date: Fri Jun 28 16:19:49 2019 -0400

    Init HostState.failed_builds

    If _update_from_compute_node returns early and the HostState is
    not filtered out we can hit an AttributeError in the
    BuildFailureWeigher because the failed_builds attribute is not
    set. This simply initializes the attribute like the other stats
    fields.

    Change-Id: I5f8e4d32c6a1d6b61396b4fa11c5d776f432df0c
    Closes-Bug: #1834691
    (cherry picked from commit d540903463aa9b0cf69cefac7cc60e5b70e40a1c)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/668520

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/rocky)

Reviewed: https://review.opendev.org/668520
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5acbea506a137492511a762e454d785810365bd8
Submitter: Zuul
Branch: stable/rocky

commit 5acbea506a137492511a762e454d785810365bd8
Author: Matt Riedemann <email address hidden>
Date: Fri Jun 28 16:19:49 2019 -0400

    Init HostState.failed_builds

    If _update_from_compute_node returns early and the HostState is
    not filtered out we can hit an AttributeError in the
    BuildFailureWeigher because the failed_builds attribute is not
    set. This simply initializes the attribute like the other stats
    fields.

    Change-Id: I5f8e4d32c6a1d6b61396b4fa11c5d776f432df0c
    Closes-Bug: #1834691
    (cherry picked from commit d540903463aa9b0cf69cefac7cc60e5b70e40a1c)
    (cherry picked from commit 725b37f515e4ad01e1f4491b6d8137ce1416f6d6)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/668911

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/queens)

Reviewed: https://review.opendev.org/668911
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9bc95675325498a7d30b67089e19b5b953d77e75
Submitter: Zuul
Branch: stable/queens

commit 9bc95675325498a7d30b67089e19b5b953d77e75
Author: Matt Riedemann <email address hidden>
Date: Fri Jun 28 16:19:49 2019 -0400

    Init HostState.failed_builds

    If _update_from_compute_node returns early and the HostState is
    not filtered out we can hit an AttributeError in the
    BuildFailureWeigher because the failed_builds attribute is not
    set. This simply initializes the attribute like the other stats
    fields.

    Change-Id: I5f8e4d32c6a1d6b61396b4fa11c5d776f432df0c
    Closes-Bug: #1834691
    (cherry picked from commit d540903463aa9b0cf69cefac7cc60e5b70e40a1c)
    (cherry picked from commit 725b37f515e4ad01e1f4491b6d8137ce1416f6d6)
    (cherry picked from commit 5acbea506a137492511a762e454d785810365bd8)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/669538

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 19.0.2

This issue was fixed in the openstack/nova 19.0.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.2.2

This issue was fixed in the openstack/nova 18.2.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.12

This issue was fixed in the openstack/nova 17.0.12 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 20.0.0.0rc1

This issue was fixed in the openstack/nova 20.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.opendev.org/669538
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=61104f3974c61bffb500c268fb4e1d6e65c6b3e6
Submitter: Zuul
Branch: stable/pike

commit 61104f3974c61bffb500c268fb4e1d6e65c6b3e6
Author: Matt Riedemann <email address hidden>
Date: Fri Jun 28 16:19:49 2019 -0400

    Init HostState.failed_builds

    If _update_from_compute_node returns early and the HostState is
    not filtered out we can hit an AttributeError in the
    BuildFailureWeigher because the failed_builds attribute is not
    set. This simply initializes the attribute like the other stats
    fields.

    Change-Id: I5f8e4d32c6a1d6b61396b4fa11c5d776f432df0c
    Closes-Bug: #1834691
    (cherry picked from commit d540903463aa9b0cf69cefac7cc60e5b70e40a1c)
    (cherry picked from commit 725b37f515e4ad01e1f4491b6d8137ce1416f6d6)
    (cherry picked from commit 5acbea506a137492511a762e454d785810365bd8)
    (cherry picked from commit 9bc95675325498a7d30b67089e19b5b953d77e75)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova pike-eol

This issue was fixed in the openstack/nova pike-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.