OpenStack Shared File Systems Service (Manila)

tempest test failure due to incorrect provisioned_capacity_gb

Bug #1943829 reported by Ashley Rodriguez on 2021-09-16

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Shared File Systems Service (Manila)	Triaged	Medium	haixin	OpenStack Shared File Systems Service (Manila) caracal-1

Bug Description

Description:

Encountered this bug within zuul logs (https://zuul.opendev.org/t/openstack/build/002f38456eae4c7c988cf0eb9f5bc60a/log/controller/logs/screen-m-sch.txt)
Involves line 97 in /opt/stack/manila/manila/scheduler/filters/capacity.py:
provisioned_ratio = ((host_state.provisioned_capacity_gb +
share_size) / total)
host_state.provisioned_capacity_gb is None type.

Logs:
Sep 15 16:18:41.295015 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-4ee65e6b-5874-474f-9c62-945e165b5d95 tempest-MigrationCancelNFSTest-1544969794 None] Starting with 3 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:80}}
Sep 15 16:18:41.295323 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-4ee65e6b-5874-474f-9c62-945e165b5d95 tempest-MigrationCancelNFSTest-1544969794 None] Filter AvailabilityZoneFilter returned 3 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:97}}
Sep 15 16:18:41.295595 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.capacity [None req-4ee65e6b-5874-474f-9c62-945e165b5d95 tempest-MigrationCancelNFSTest-1544969794 None] Space information for share creation on host ubuntu-focal-rax-dfw-0026509888@paris#gamma (requested / avail): 1/28 {{(pid=108015) host_passes /opt/stack/manila/manila/scheduler/filters/capacity.py:79}}
Sep 15 16:18:41.295859 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.capacity [None req-4ee65e6b-5874-474f-9c62-945e165b5d95 tempest-MigrationCancelNFSTest-1544969794 None] Space information for share creation on host ubuntu-focal-rax-dfw-0026509888@london#alpha (requested / avail): 1/29 {{(pid=108015) host_passes /opt/stack/manila/manila/scheduler/filters/capacity.py:79}}
Sep 15 16:18:41.296165 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR manila.scheduler.manager [None req-4ee65e6b-5874-474f-9c62-945e165b5d95 tempest-MigrationCancelNFSTest-1544969794 None] Failed to schedule create_share: unsupported operand type(s) for +: 'NoneType' and 'int'
Sep 15 16:18:41.298603 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Starting with 3 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:80}}
Sep 15 16:18:41.298909 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Filter AvailabilityZoneFilter returned 1 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:97}}
Sep 15 16:18:41.299192 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.capacity [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Space information for share creation on host ubuntu-focal-rax-dfw-0026509888@paris#gamma (requested / avail): 1/28 {{(pid=108015) host_passes /opt/stack/manila/manila/scheduler/filters/capacity.py:79}}
Sep 15 16:18:41.299476 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Filter CapacityFilter returned 1 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:97}}
Sep 15 16:18:41.299801 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Filter CapabilitiesFilter returned 1 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:97}}
Sep 15 16:18:41.300077 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.driver [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Driver Filter: Checking host 'ubuntu-focal-rax-dfw-0026509888@paris#gamma' {{(pid=108015) host_passes /opt/stack/manila/manila/scheduler/filters/driver.py:37}}
Sep 15 16:18:41.300336 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.driver [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Filter function not set :: passing host. {{(pid=108015) _check_filter_function /opt/stack/manila/manila/scheduler/filters/driver.py:53}}
Sep 15 16:18:41.300586 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.driver [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Result: True {{(pid=108015) host_passes /opt/stack/manila/manila/scheduler/filters/driver.py:40}}
Sep 15 16:18:41.300838 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.driver [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Done checking host 'ubuntu-focal-rax-dfw-0026509888@paris#gamma' {{(pid=108015) host_passes /opt/stack/manila/manila/scheduler/filters/driver.py:41}}
Sep 15 16:18:41.301105 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Filter DriverFilter returned 1 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:97}}
Sep 15 16:18:41.301373 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Filter ShareReplicationFilter returned 1 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:97}}
Sep 15 16:18:41.301632 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Filter CreateFromSnapshotFilter returned 1 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:97}}
Sep 15 16:18:41.302480 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Filter AffinityFilter returned 1 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:97}}
Sep 15 16:18:41.303199 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.filters.base [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Filter AntiAffinityFilter returned 1 host(s) {{(pid=108015) get_filtered_objects /opt/stack/manila/manila/scheduler/filters/base.py:97}}
Sep 15 16:18:41.303649 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.weighers.goodness [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Checking host 'ubuntu-focal-rax-dfw-0026509888@paris#gamma' {{(pid=108015) _weigh_object /opt/stack/manila/manila/scheduler/weighers/goodness.py:46}}
Sep 15 16:18:41.303921 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: WARNING manila.scheduler.weighers.goodness [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Goodness function not set :: defaulting to minimal goodness rating of 0.
Sep 15 16:18:41.304174 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: INFO manila.scheduler.weighers.goodness [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Goodness function result for host ubuntu-focal-rax-dfw-0026509888@paris#gamma: 0.
Sep 15 16:18:41.304442 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.weighers.goodness [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Goodness: 0 {{(pid=108015) _weigh_object /opt/stack/manila/manila/scheduler/weighers/goodness.py:48}}
Sep 15 16:18:41.304697 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: DEBUG manila.scheduler.weighers.goodness [None req-3d4c130c-6d50-40e1-86f0-890de6b78978 tempest-MigrationWithDifferentSnapshotSupportNFSTest-1108156932 None] Done checking host 'ubuntu-focal-rax-dfw-0026509888@paris#gamma' {{(pid=108015) _weigh_object /opt/stack/manila/manila/scheduler/weighers/goodness.py:49}}
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server [None req-4ee65e6b-5874-474f-9c62-945e165b5d95 tempest-MigrationCancelNFSTest-1544969794 None] Exception during message handling: TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/scheduler/manager.py", line 127, in create_share_instance
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server self._set_share_state_and_notify(
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server self.force_reraise()
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server raise self.value
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/scheduler/manager.py", line 118, in create_share_instance
Sep 15 16:18:41.328734 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server self.driver.schedule_create_share(context, request_spec,
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/scheduler/drivers/filter.py", line 91, in schedule_create_share
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server weighed_host = self._schedule_share(context,
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/scheduler/drivers/filter.py", line 247, in _schedule_share
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server hosts, last_filter = self.host_manager.get_filtered_hosts(
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/scheduler/host_manager.py", line 561, in get_filtered_hosts
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server return self.filter_handler.get_filtered_objects(filter_classes,
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/scheduler/filters/base.py", line 91, in get_filtered_objects
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server list_objs = list(objs)
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/scheduler/filters/base.py", line 44, in filter_all
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server if self._filter_one(obj, filter_properties):
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/scheduler/filters/base_host.py", line 27, in _filter_one
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server return self.host_passes(obj, filter_properties)
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server File "/opt/stack/manila/manila/scheduler/filters/capacity.py", line 97, in host_passes
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server provisioned_ratio = ((host_state.provisioned_capacity_gb +
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
Sep 15 16:18:41.329783 ubuntu-focal-rax-dfw-0026509888 manila-scheduler[108015]: ERROR oslo_messaging.rpc.server

Tags:

Vida Haririan (vhariria) on 2021-09-16

tags:

added: migration tempest

Revision history for this message

Vida Haririan (vhariria) wrote on 2021-09-23:

Additional comments: https://meetings.opendev.org/meetings/manila/2021/manila.2021-09-23-15.00.log.html

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-24: Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/810842

Changed in manila:
status:	New → In Progress

Goutham Pacha Ravi (gouthamr) on 2021-09-24

summary:	- tempest test failure when migrating + tempest test failure due to incorrect provisioned_capacity_gb
Changed in manila:
assignee:	nobody → haixin (haixin77)
importance:	Undecided → Medium
milestone:	none → yoga-1

Goutham Pacha Ravi (gouthamr) on 2021-11-29

Changed in manila:
milestone:	yoga-1 → yoga-2

Carlos Eduardo (silvacarlose) on 2022-01-27

Changed in manila:
milestone:	yoga-2 → yoga-3

Carlos Eduardo (silvacarlose) on 2022-03-04

Changed in manila:
milestone:	yoga-3 → zed-1

Carlos Eduardo (silvacarlose) on 2022-06-20

Changed in manila:
milestone:	zed-1 → zed-3

Carlos Eduardo (silvacarlose) on 2022-10-06

Changed in manila:
milestone:	zed-3 → antelope-3

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-13: Change abandoned on manila (master)

Change abandoned by "haixin <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/manila/+/810842

Carlos Eduardo (silvacarlose) on 2023-02-24

Changed in manila:
milestone:	antelope-3 → antelope-rc1

Goutham Pacha Ravi (gouthamr) on 2023-04-13

Changed in manila:
milestone:	antelope-rc1 → bobcat-1

Carlos Eduardo (silvacarlose) on 2023-06-21

Changed in manila:
milestone:	bobcat-1 → bobcat-2

Carlos Eduardo (silvacarlose) on 2023-08-17

Changed in manila:
milestone:	bobcat-2 → bobcat-3

Goutham Pacha Ravi (gouthamr) on 2023-09-19

Changed in manila:
milestone:	bobcat-3 → bobcat-rc1

Goutham Pacha Ravi (gouthamr) on 2023-09-21

Changed in manila:
milestone:	bobcat-rc1 → caracal-1

Vida Haririan (vhariria) on 2023-10-19

Changed in manila:
status:	In Progress → Triaged

Revision history for this message

Vida Haririan (vhariria) wrote on 2023-10-19:

See additional related discussions at https://meetings.opendev.org/meetings/manila/2023/manila.2023-10-19-15.00.log.html

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.