This behavior was observed in the CI on the ZFSOnLinux job; share creation can sporadically fail and the scheduler log shows an exception from the capacity filter:
manila-scheduler[7474]: DEBUG manila.scheduler.filters.base [None req-db0c6229-dda8-4206-a416-b24270dc3293 None None] Starting with 3 host(s) {{(pid=7474) get_filtered_objects /opt/stack/new/manila/manila/scheduler/filters/base.py:80}}
manila-scheduler[7474]: DEBUG manila.scheduler.filters.base [None req-db0c6229-dda8-4206-a416-b24270dc3293 None None] Filter AvailabilityZoneFilter returned 2 host(s) {{(pid=7474) get_filtered_objects /opt/stack/new/manila/manila/scheduler/filters/base.py:97}}
manila-scheduler[7474]: DEBUG manila.scheduler.filters.capacity [None req-db0c6229-dda8-4206-a416-b24270dc3293 None None] Space information for share creation on host ubuntu-bionic-inap-mtl01-0015815632@london#alpha (requested / avail): 1/29 {{(pid=7474) host_passes /opt/stack/new/manila/manila/scheduler/filters/capacity.py:77}}
manila-scheduler[7474]: ERROR manila.scheduler.manager [None req-db0c6229-dda8-4206-a416-b24270dc3293 None None] Failed to schedule create_share: unsupported operand type(s) for +: 'NoneType' and 'int'
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server [None req-db0c6229-dda8-4206-a416-b24270dc3293 None None] Exception during message handling: TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/scheduler/manager.py", line 121, in create_share_instance
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server context, ex, request_spec)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server self.force_reraise()
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server raise value
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/scheduler/manager.py", line 111, in create_share_instance
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server filter_properties)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/scheduler/drivers/filter.py", line 84, in schedule_create_share
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server filter_properties)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/scheduler/drivers/filter.py", line 225, in _schedule_share
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server hosts, filter_properties)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/scheduler/host_manager.py", line 544, in get_filtered_hosts
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server filter_properties)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/scheduler/filters/base.py", line 91, in get_filtered_objects
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server list_objs = list(objs)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/scheduler/filters/base.py", line 44, in filter_all
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server if self._filter_one(obj, filter_properties):
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/scheduler/filters/base_host.py", line 27, in _filter_one
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server return self.host_passes(obj, filter_properties)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/scheduler/filters/capacity.py", line 94, in host_passes
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server share_size) / total)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server
manila-scheduler[7474]: DEBUG manila.scheduler.filters.base [None req-d3b3d188-f6c2-44cd-96ef-6bbb275dae3c None None] Starting with 3 host(s) {{(pid=7474) get_filtered_objects /opt/stack/new/manila/manila/scheduler/filters/base.py:80}}
The line raising this exception is https://opendev.org/openstack/manila/src/commit/cd09242ba52b6fab304694c1780806dd614ea6de/manila/scheduler/filters/capacity.py#L93-L94
It appears that this is because the provisioned_capacity_gb is None. It's possible this is related to the recent change to the calculation of the provisioned_capacity_gb for drivers that are reporting thinly provisioned pools, but don't report a provisioned_capacity_gb:
https://review.opendev.org/#/c/718016/ (https://launchpad.net/bugs/1869712)
Logs from the ZFSOnLinux job failure are attached
Compressed log folder containing:
job-output.txt - Console log file m-sch-zfsonlinu x.txt. - Manila Scheduler log (where the exception is) 2.html - Test results visualized
screen-m-api.txt - Manila API log
screen-m-dat.txt - Manila Data log
screen-
screen-m-shr.txt - Manila share log
tempest_conf.txt - Tempest config file
tempest.txt - Tempest log file
testr_results-