tempest test failure due to incorrect provisioned_capacity_gb

Bug #1943829 reported by Ashley Rodriguez
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Triaged
Medium
haixin

Bug Description

Description:

Encountered this bug within zuul logs (https://zuul.opendev.org/t/openstack/build/002f38456eae4c7c988cf0eb9f5bc60a/log/controller/logs/screen-m-sch.txt)
Involves line 97 in /opt/stack/manila/manila/scheduler/filters/capacity.py:
provisioned_ratio = ((host_state.provisioned_capacity_gb +
                                  share_size) / total)
host_state.provisioned_capacity_gb is None type.

Logs:
Sep 15 16:18:41.295015 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-4ee65e6b-5874-474f-9c62-945e165b5d95 tempest-MigrationCancelNFSTest-1544969794 None] Starting with 3 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:80}}
Sep 15 16:18:41.295323 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-4ee65e6b-5874-474f-9c62-945e165b5d95 tempest-MigrationCancelNFSTest-1544969794 None] Filter AvailabilityZoneFilter returned 3 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:97}}
Sep 15 16:18:41.295595 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.capacity [None req-4ee65e6b-5874-474f-9c62-945e165b5d95 tempest-MigrationCancelNFSTest-1544969794 None] Space information for share creation on host ubuntu-focal-rax-dfw-0026509888@paris#gamma (requested / avail): 1/28 {{(pid=108015) host_passes /opt/stack/manila/manila/scheduler/filters/capacity.py:79}}
Sep 15 16:18:41.295859 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.capacity [None req-4ee65e6b-5874-474f-9c62-945e165b5d95 tempest-MigrationCancelNFSTest-1544969794 None] Space information for share creation on host ubuntu-focal-rax-dfw-0026509888@london#alpha (requested / avail): 1/29 {{(pid=108015) host_passes /opt/stack/manila/manila/scheduler/filters/capacity.py:79}}
Sep 15 16:18:41.296165 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR manila.scheduler.manager [None req-4ee65e6b-5874-474f-9c62-945e165b5d95 tempest-MigrationCancelNFSTest-1544969794 None] Failed to schedule create_share: unsupported operand type(s) for +: 'NoneType' and 'int'
Sep 15 16:18:41.298603 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Starting with 3 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:80}}
Sep 15 16:18:41.298909 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:97}}
Sep 15 16:18:41.299192 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.capacity [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Space information for share creation on host ubuntu-focal-rax-dfw-0026509888@paris#gamma (requested / avail): 1/28 {{(pid=108015) host_passes /opt/stack/manila/manila/scheduler/filters/capacity.py:79}}
Sep 15 16:18:41.299476 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Filter CapacityFilter returned 1 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:97}}
Sep 15 16:18:41.299801 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:97}}
Sep 15 16:18:41.300077 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.driver [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Driver Filter: Checking host 'ubuntu-focal-rax-dfw-0026509888@paris#gamma' {{(pid=108015) host_passes /opt/stack/manila/manila/scheduler/filters/driver.py:37}}
Sep 15 16:18:41.300336 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.driver [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Filter function not set :: passing host. {{(pid=108015) _check_filter_function /opt/stack/manila/manila/scheduler/filters/driver.py:53}}
Sep 15 16:18:41.300586 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.driver [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Result: True {{(pid=108015) host_passes /opt/stack/manila/manila/scheduler/filters/driver.py:40}}
Sep 15 16:18:41.300838 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.driver [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Done checking host 'ubuntu-focal-rax-dfw-0026509888@paris#gamma' {{(pid=108015) host_passes /opt/stack/manila/manila/scheduler/filters/driver.py:41}}
Sep 15 16:18:41.301105 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Filter DriverFilter returned 1 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:97}}
Sep 15 16:18:41.301373 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Filter ShareReplicationFilter returned 1 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:97}}
Sep 15 16:18:41.301632 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Filter CreateFromSnapshotFilter returned 1 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:97}}
Sep 15 16:18:41.302480 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Filter AffinityFilter returned 1 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:97}}
Sep 15 16:18:41.303199 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Filter AntiAffinityFilter returned 1 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:97}}
Sep 15 16:18:41.303649 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.weighers.goodness [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Checking host 'ubuntu-focal-rax-dfw-0026509888@paris#gamma' {{(pid=108015) _weigh_object /opt/stack/manila/manila/scheduler/weighers/goodness.py:46}}
Sep 15 16:18:41.303921 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: WARNING manila.scheduler.weighers.goodness [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Goodness function not set :: defaulting to minimal goodness rating of 0.
Sep 15 16:18:41.304174 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: INFO manila.scheduler.weighers.goodness [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Goodness function result for host ubuntu-focal-rax-dfw-0026509888@paris#gamma: 0.
Sep 15 16:18:41.304442 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.weighers.goodness [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Goodness: 0 {{(pid=108015) _weigh_object /opt/stack/manila/manila/scheduler/weighers/goodness.py:48}}
Sep 15 16:18:41.304697 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.weighers.goodness [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Done checking host 'ubuntu-focal-rax-dfw-0026509888@paris#gamma' {{(pid=108015) _weigh_object /opt/stack/manila/manila/scheduler/weighers/goodness.py:49}}
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server [None req-4ee65e6b-5874-474f-9c62-945e165b5d95 tempest-MigrationCancelNFSTest-1544969794 None] Exception during message handling: TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/scheduler/manager.py", line 127, in create_share_instance
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server self._set_share_state_and_notify(
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server self.force_reraise()
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server raise self.value
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/scheduler/manager.py", line 118, in create_share_instance
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server self.driver.schedule_create_share(context, request_spec,
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/scheduler/drivers/filter.py", line 91, in schedule_create_share
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server weighed_host = self._schedule_share(context,
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/scheduler/drivers/filter.py", line 247, in _schedule_share
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server hosts, last_filter = self.host_manager.get_filtered_hosts(
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/scheduler/host_manager.py", line 561, in get_filtered_hosts
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server return self.filter_handler.get_filtered_objects(filter_classes,
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/scheduler/filters/base.py", line 91, in get_filtered_objects
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server list_objs = list(objs)
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/scheduler/filters/base.py", line 44, in filter_all
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server if self._filter_one(obj, filter_properties):
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/scheduler/filters/base_host.py", line 27, in _filter_one
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server return self.host_passes(obj, filter_properties)
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/scheduler/filters/capacity.py", line 97, in host_passes
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server provisioned_ratio = ((host_state.provisioned_capacity_gb +
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server

Vida Haririan (vhariria)
tags: added: migration tempest
Revision history for this message
Vida Haririan (vhariria) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/810842

Changed in manila:
status: New → In Progress
summary: - tempest test failure when migrating
+ tempest test failure due to incorrect provisioned_capacity_gb
Changed in manila:
assignee: nobody → haixin (haixin77)
importance: Undecided → Medium
milestone: none → yoga-1
Changed in manila:
milestone: yoga-1 → yoga-2
Changed in manila:
milestone: yoga-2 → yoga-3
Changed in manila:
milestone: yoga-3 → zed-1
Changed in manila:
milestone: zed-1 → zed-3
Changed in manila:
milestone: zed-3 → antelope-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on manila (master)

Change abandoned by "haixin <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/manila/+/810842

Changed in manila:
milestone: antelope-3 → antelope-rc1
Changed in manila:
milestone: antelope-rc1 → bobcat-1
Changed in manila:
milestone: bobcat-1 → bobcat-2
Changed in manila:
milestone: bobcat-2 → bobcat-3
Changed in manila:
milestone: bobcat-3 → bobcat-rc1
Changed in manila:
milestone: bobcat-rc1 → caracal-1
Vida Haririan (vhariria)
Changed in manila:
status: In Progress → Triaged
Revision history for this message
Vida Haririan (vhariria) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.