Cinder

db online_data_migrations error when we set backend_host in cinder.conf

Bug #1824509 reported by Jie Li on 2019-04-12

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Cinder	New	Undecided	Unassigned

Bug Description

When we configured backend_host (not host) for cinder-volume HA, then we execute the cli "cinder-manage db online_data_migrations", and we can get the following error: "Error attempting to run shared_targets_online_data_migration: MessagingTimeout: Timed out waiting for a reply to message ID".

The problem can be reproduced by deploying cinder-volume with pacemaker.And I have set the backend_host = hostgroup,the "host" is not set.

The reason of this we found that there will be an rpc call for all service.host. When the service.host is hostgroup@ssd, beacuse the "hostgroup" host is virtual, so theis is error.

openstack version:Rocky

Ref:
https://github.com/openstack/cinder/blob/master/cinder/cmd/manage.py#L111

Revision history for this message

Jie Li (ramboman) wrote on 2019-04-12:

WechatIMG2125.png Edit (42.7 KiB, image/png)

Revision history for this message

Jie Li (ramboman) wrote on 2019-04-12:

the log:
520 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage Traceback (most recent call last):
521 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage File "/usr/lib/python2.7/site-packages/cinder/cmd/manage.py", line 350, in _run_migrati on
522 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage found, done = migration_meth(ctxt, count)
523 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage File "/usr/lib/python2.7/site-packages/cinder/cmd/manage.py", line 127, in shared_targe ts_online_data_migration
524 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage non_shared_hosts, total_vols_to_update = _get_non_shared_target_hosts(ctxt)
525 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage File "/usr/lib/python2.7/site-packages/cinder/cmd/manage.py", line 110, in _get_non_sha red_target_hosts
526 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage capabilities = rpcapi.get_capabilities(ctxt, service.host, True)
527 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage File "/usr/lib/python2.7/site-packages/cinder/volume/rpcapi.py", line 341, in get_capab ilities
528 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage return cctxt.call(ctxt, 'get_capabilities', discover=discover)
529 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
530 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage retry=self.retry)
531 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
532 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage retry=retry)
533 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584 , in send
534 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage call_monitor_timeout, retry=retry)
535 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573 , in _send
536 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage call_monitor_timeout)
537 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459 , in wait
538 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage message = self.waiters.get(msg_id, timeout=timeout)
539 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336 , in get
540 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage 'to message ID %s' % msg_id)
541 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage MessagingTimeout: Timed out waiting for a reply to message ID 79fc13882d1945d9b8a60e8fcbd eccb0
542 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage

the log:
520 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage Traceback (most recent call last):
521 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage   File "/usr/lib/python2.7/site-packages/cinder/cmd/manage.py", line 350, in _run_migrati    on
522 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage     found, done = migration_meth(ctxt, count)
523 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage   File "/usr/lib/python2.7/site-packages/cinder/cmd/manage.py", line 127, in shared_targe    ts_online_data_migration
524 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage     non_shared_hosts, total_vols_to_update = _get_non_shared_target_hosts(ctxt)
525 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage   File "/usr/lib/python2.7/site-packages/cinder/cmd/manage.py", line 110, in _get_non_sha    red_target_hosts
526 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage     capabilities = rpcapi.get_capabilities(ctxt, service.host, True)
527 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage   File "/usr/lib/python2.7/site-packages/cinder/volume/rpcapi.py", line 341, in get_capab    ilities
528 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage     return cctxt.call(ctxt, 'get_capabilities', discover=discover)
529 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call
530 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage     retry=self.retry)
531 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send
532 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage     retry=retry)
533 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584    , in send
534 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage     call_monitor_timeout, retry=retry)
535 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573    , in _send
536 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage     call_monitor_timeout)
537 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459    , in wait
538 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage     message = self.waiters.get(msg_id, timeout=timeout)
539 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336    , in get
540 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage     'to message ID %s' % msg_id)
541 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage MessagingTimeout: Timed out waiting for a reply to message ID 79fc13882d1945d9b8a60e8fcbd    eccb0
542 2019-04-12 16:38:59.542 139 ERROR cinder.cmd.manage

Revision history for this message

iain MacDonnell (imacdonn) wrote on 2019-04-15:

Similar to https://bugs.launchpad.net/cinder/+bug/1806156 - different circumstances, but a solution that doesn't rely on an RPC call could solve both...

Jie Li (ramboman) on 2019-04-16

summary:

- db online_data_migrations error
+ db online_data_migrations error when we set backend_host in cinder.conf

Revision history for this message

Jie Li (ramboman) wrote on 2019-04-16:

yes.This is similar to https://bugs.launchpad.net/cinder/+bug/1806156.But I think the db migration shouldn't rely on the service status.