Cinder

cinder fails when *-get-iter api returns empty list

Bug #1826010 reported by Scott Stanton
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
New
Undecided
Unassigned

Bug Description

The vserver has 50 LIFs and the net-interface-get-iter API call is requesting 50 entries. Since 50 entries is exactly how many are returned the code asks for the next 50. The next run of the the API returns and empty set and causes cinder to crash.

Here is the data returned:
<attributes-list></attributes-list>
<num-records></num-records>

Error logs from /var/log/cinder/volume.log
2019-03-13 13:56:53.971 9208 TRACE cinder.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/netapp/dataontap/client/api.py", line 435, in __getitem__
2019-03-13 13:56:53.971 9208 TRACE cinder.openstack.common.threadgroup raise KeyError(_('No element by given name %s.') % (key))
2019-03-13 13:56:53.971 9208 TRACE cinder.openstack.common.threadgroup KeyError: u'No element by given name attributes-list.'

As a workaround the max-record in the API call was increased to 51 and the problem was not seen.

Cinder needs to be able to handle null values from any of the get-iter calls that it makes to ONTAP since the return value is an empty set and not an error message.

Tags: drivers netapp
Eric Harney (eharney)
tags: added: drivers netapp
Revision history for this message
Thiago Correa (thgcorrea) wrote :

Can we have more information on how to reproduce this?

Revision history for this message
Thiago Correa (thgcorrea) wrote :

Hi, we tried to reproduce this on both python3 and python2 environments using the latest and queens devstack releases respectively and it didn't happen.
We set the max_page_length to 3 having 3 LIFs:

Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: DEBUG cinder.volume.drivers.netapp.dataontap.client.client_cmode [None req-c771d812-b8fc-4396-ad7b-5be74cb69cc0 None None] MAX PAGE LENGTH: 3
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: DEBUG cinder.volume.drivers.netapp.dataontap.client.client_cmode [None req-c771d812-b8fc-4396-ad7b-5be74cb69cc0 None None] RESULT NET INTERFACE GET ITER = <results xmlns="http://www.netapp.com/filer/admin" status="passed">
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: <attributes-list>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: <net-interface-info>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: <address>10.193.154.138</address>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: <interface-name>nfs_cifs_data_lif_OSTK-select20-01_e0a</interface-name>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: <vserver>vserver_thg_dev1</vserver>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: </net-interface-info>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: <net-interface-info>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: <address>10.193.154.139</address>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: <interface-name>nfs_data_lif2</interface-name>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: <vserver>vserver_thg_dev1</vserver>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: </net-interface-info>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: <net-interface-info>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: <address>10.193.154.140</address>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: <interface-name>nfs_data_lif3</interface-name>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: <vserver>vserver_thg_dev1</vserver>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: </net-interface-info>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: </attributes-list>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: <next-tag></next-tag>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: <num-records>3</num-records>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: </results>
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]:
Oct 19 13:19:07 13-thiago-cinder-dev cinder-volume[12640]: DEBUG cinder.volume.drivers.netapp.dataontap.client.client_cmode [None req-c771d812-b8fc-4396-ad7b-5be74cb69cc0 None None] MAX PAGE LENGTH: 3

Can we have more info on the environment and how to reproduce this?

Revision history for this message
Scott Stanton (stantonntap) wrote :

I don't see a max_page_length in the API call, so I don't know what that will do. To reproduce it with 3 lifs you need to send the API call with "<max-records>3</max-records>" so it will only return 3 records. It should make another call to get 3 more and return a null value that Cinder will fail on.

Revision history for this message
Thiago Correa (thgcorrea) wrote :

From NetApp driver the API argument "max-records" is passed using max_page_length, like so:

api_args['max-records'] = max_page_length

So it is setting max-records to 3 on this call.

Revision history for this message
Scott Stanton (stantonntap) wrote :

All I can suggest is making 47 more LIFs and setting max_page_length to 50 like the customer had when they ran into the problem.

Revision history for this message
Thiago Correa (thgcorrea) wrote :

We performed the test with 50 LIFs and the result is the same: no error. We might need more information to reproduce this, such as the client's openstack version and the error's complete traceback.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.