zfsonlinux driver's provisioned_capacity_gb is None in the scheduler

Bug #1871768 reported by Goutham Pacha Ravi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Triaged
Medium
Unassigned

Bug Description

This behavior was observed in the CI on the ZFSOnLinux job; share creation can sporadically fail and the scheduler log shows an exception from the capacity filter:

manila-scheduler[7474]: DEBUG manila.scheduler.filters.base [None req-db0c6229-dda8-4206-a416-b24270dc3293 None None] Starting with 3 host(s) {{(pid=7474) get_filtered_objects /opt/stack/new/manila/manila/scheduler/filters/base.py:80}}
manila-scheduler[7474]: DEBUG manila.scheduler.filters.base [None req-db0c6229-dda8-4206-a416-b24270dc3293 None None] Filter AvailabilityZoneFilter returned 2 host(s) {{(pid=7474) get_filtered_objects /opt/stack/new/manila/manila/scheduler/filters/base.py:97}}
manila-scheduler[7474]: DEBUG manila.scheduler.filters.capacity [None req-db0c6229-dda8-4206-a416-b24270dc3293 None None] Space information for share creation on host ubuntu-bionic-inap-mtl01-0015815632@london#alpha (requested / avail): 1/29 {{(pid=7474) host_passes /opt/stack/new/manila/manila/scheduler/filters/capacity.py:77}}
manila-scheduler[7474]: ERROR manila.scheduler.manager [None req-db0c6229-dda8-4206-a416-b24270dc3293 None None] Failed to schedule create_share: unsupported operand type(s) for +: 'NoneType' and 'int'
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server [None req-db0c6229-dda8-4206-a416-b24270dc3293 None None] Exception during message handling: TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/scheduler/manager.py", line 121, in create_share_instance
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server context, ex, request_spec)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server self.force_reraise()
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server raise value
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/scheduler/manager.py", line 111, in create_share_instance
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server filter_properties)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/scheduler/drivers/filter.py", line 84, in schedule_create_share
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server filter_properties)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/scheduler/drivers/filter.py", line 225, in _schedule_share
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server hosts, filter_properties)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/scheduler/host_manager.py", line 544, in get_filtered_hosts
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server filter_properties)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/scheduler/filters/base.py", line 91, in get_filtered_objects
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server list_objs = list(objs)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/scheduler/filters/base.py", line 44, in filter_all
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server if self._filter_one(obj, filter_properties):
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/scheduler/filters/base_host.py", line 27, in _filter_one
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server return self.host_passes(obj, filter_properties)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/manila/manila/scheduler/filters/capacity.py", line 94, in host_passes
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server share_size) / total)
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
manila-scheduler[7474]: ERROR oslo_messaging.rpc.server
manila-scheduler[7474]: DEBUG manila.scheduler.filters.base [None req-d3b3d188-f6c2-44cd-96ef-6bbb275dae3c None None] Starting with 3 host(s) {{(pid=7474) get_filtered_objects /opt/stack/new/manila/manila/scheduler/filters/base.py:80}}

The line raising this exception is https://opendev.org/openstack/manila/src/commit/cd09242ba52b6fab304694c1780806dd614ea6de/manila/scheduler/filters/capacity.py#L93-L94

It appears that this is because the provisioned_capacity_gb is None. It's possible this is related to the recent change to the calculation of the provisioned_capacity_gb for drivers that are reporting thinly provisioned pools, but don't report a provisioned_capacity_gb:

https://review.opendev.org/#/c/718016/ (https://launchpad.net/bugs/1869712)

Logs from the ZFSOnLinux job failure are attached

Tags: zfsonlinux
Revision history for this message
Goutham Pacha Ravi (gouthamr) wrote :

Compressed log folder containing:

job-output.txt - Console log file
screen-m-api.txt - Manila API log
screen-m-dat.txt - Manila Data log
screen-m-sch-zfsonlinux.txt. - Manila Scheduler log (where the exception is)
screen-m-shr.txt - Manila share log
tempest_conf.txt - Tempest config file
tempest.txt - Tempest log file
testr_results-2.html - Test results visualized

Vida Haririan (vhariria)
Changed in manila:
milestone: none → victoria-1
importance: Undecided → Medium
Revision history for this message
Vida Haririan (vhariria) wrote :
Changed in manila:
milestone: victoria-1 → none
Revision history for this message
Douglas Viroel (dviroel) wrote :

Same issue was observed in NetApp CI. Isn't the first time that we caught this error in the logs.

https://logs.openstack.netapp.com/logs/38/775038/2/upstream-check/manila-cDOT-
ss/5a6f800/logs/screen-m-sch.txt.gz?level=DEBUG#_Feb_12_02_39_43_230719

Revision history for this message
Douglas Viroel (dviroel) wrote :

Compressed logs for NetApp CI: output, manila.conf, scheduler-log, api-log, tempest-log, tempest.conf

Vida Haririan (vhariria)
Changed in manila:
status: New → Triaged
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.