lvm is reporting problems, not sending heartbeat. Service will appear "down"

Hey there,

It looks like you're running into an issue which could be a config problem, but also could be a bug. Do you have output of your cinder-volume logs? That would be helpful in determining the issue

Thanks!

Hi,

I got this issue resolved.

I saw lvm2-lvmetad service was disabled but it was active.

-----------------------
root@ubuntu-Compute1:~#
root@ubuntu-Compute1:~# systemctl status lvm2-lvmetad
● lvm2-lvmetad.service - LVM2 metadata daemon
   Loaded: loaded (/lib/systemd/system/lvm2-lvmetad.service; disabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-06-05 15:09:32 +08; 2h 10min ago
     Docs: man:lvmetad(8)
 Main PID: 1023 (lvmetad)
   CGroup: /system.slice/lvm2-lvmetad.service
           └─1023 /sbin/lvmetad -f

Jun 05 15:09:32 ubuntu-Compute1 systemd[1]: Started LVM2 metadata daemon.
-----------------------

Executed these series of commands and lvm2 was enabled and things started working.

 systemctl enable lvm2-lvmetad.service
 systemctl enable lvm2-lvmetad.socket
 systemctl start lvm2-lvmetad.service
 systemctl start lvm2-lvmetad.socket

 systemctl status lvm2-lvmetad

-----------------------
root@ubuntu-Compute1:~# systemctl status lvm2-lvmetad
● lvm2-lvmetad.service - LVM2 metadata daemon
   Loaded: loaded (/lib/systemd/system/lvm2-lvmetad.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-06-05 17:29:38 +08; 15h ago
     Docs: man:lvmetad(8)
 Main PID: 1024 (lvmetad)
    Tasks: 1
   Memory: 576.0K
      CPU: 13ms
   CGroup: /system.slice/lvm2-lvmetad.service
           └─1024 /sbin/lvmetad -f
-----------------------
Thanks

I am seeing similar after re-running the setup-openstack playbooks after a rolling replacement upgrade of the control plane from xenial->bionic, with OSA rocky deployment.

The playbooks failed for cinder at this step https://github.com/openstack/openstack-ansible-os_cinder/blob/f9aa4c5a30f54e23665cad9c3396c34d3b8a3287/tasks/cinder_db_setup.yml#L42-L56

The resulting logs for cinder-volume

May 22 10:19:11 infra1.os.mist.rd.bbc.co.uk cinder-volume[7979]: 2019-05-22 10:19:11.768 7979 WARNING cinder.volume.manager [req-43f92ce9-631a-4239-949a-4b84b69c4e01 - - - - -] Update driver status failed: (config name rbd_hdd) is uninitialized.
May 22 10:19:16 infra1.os.mist.rd.bbc.co.uk cinder-volume[7979]: 2019-05-22 10:19:16.750 7979 ERROR cinder.service [-] Manager for service cinder-volume infra1.os.mist.rd.bbc.co.uk@rbd_hdd is reporting problems, not sending heartbeat. Service will appear "down".
May 22 10:19:16 infra1.os.mist.rd.bbc.co.uk cinder-volume[7988]: 2019-05-22 10:19:16.912 7988 ERROR cinder.service [-] Manager for service cinder-volume infra1.os.mist.rd.bbc.co.uk@rbd_nvme is reporting problems, not sending heartbeat. Service will appear "down".
May 22 10:19:26 infra1.os.mist.rd.bbc.co.uk cinder-volume[7979]: 2019-05-22 10:19:26.751 7979 ERROR cinder.service [-] Manager for service cinder-volume infra1.os.mist.rd.bbc.co.uk@rbd_hdd is reporting problems, not sending heartbeat. Service will appear "down".
May 22 10:19:26 infra1.os.mist.rd.bbc.co.uk cinder-volume[7988]: 2019-05-22 10:19:26.912 7988 ERROR cinder.service [-] Manager for service cinder-volume infra1.os.mist.rd.bbc.co.uk@rbd_nvme is reporting problems, not sending heartbeat. Service will appear "down".
May 22 10:19:36 infra1.os.mist.rd.bbc.co.uk cinder-volume[7979]: 2019-05-22 10:19:36.751 7979 ERROR cinder.service [-] Manager for service cinder-volume infra1.os.mist.rd.bbc.co.uk@rbd_hdd is reporting problems, not sending heartbeat. Service will appear "down".
May 22 10:19:36 infra1.os.mist.rd.bbc.co.uk cinder-volume[7988]: 2019-05-22 10:19:36.913 7988 ERROR cinder.service [-] Manager for service cinder-volume infra1.os.mist.rd.bbc.co.uk@rbd_nvme is reporting problems, not sending heartbeat. Service will appear "down".
May 22 10:19:36 infra1.os.mist.rd.bbc.co.uk cinder-volume[7988]: 2019-05-22 10:19:36.917 7988 WARNING cinder.volume.manager [req-c5e58acc-465d-4f87-b7ec-5547b879261e - - - - -] Update driver status failed: (config name rbd_nvme) is uninitialized.
May 22 10:19:46 infra1.os.mist.rd.bbc.co.uk cinder-volume[7979]: 2019-05-22 10:19:46.751 7979 ERROR cinder.service [-] Manager for service cinder-volume infra1.os.mist.rd.bbc.co.uk@rbd_hdd is reporting problems, not sending heartbeat. Service will appear "down".
May 22 10:19:46 infra1.os.mist.rd.bbc.co.uk cinder-volume[7988]: 2019-05-22 10:19:46.913 7988 ERROR cinder.service [-] Manager for service cinder-volume infra1.os.mist.rd.bbc.co.uk@rbd_nvme is reporting problems, not sending heartbeat. Service will appear "down".

Restarting the cinder-volume service on the infra host brings the service back correctly, unfortunately there does not appear to be anything more useful in the log to determine the root cause.

I also got a similar issue.
It could potentially be caused by rerunning a playbook.
Restarting doesn't fix it for me, someone has a suggestion?

Jul 08 19:27:51 aio1 cinder-volume[25193]: 2019-07-08 19:27:51.616 25193 DEBUG oslo_concurrency.processutils [req-eabd3210-b1db-42ed-91b6-9f8f3d945a34 - - - - -] u'sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C vgs --version' failed. Not Retrying. execute /openstack/venvs/cinder-19.0.0.0rc4.dev3/lib/python2.7/site-packages/oslo_concurrency/processutils.py:457
Jul 08 19:27:51 aio1 cinder-volume[25193]: 2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager [req-eabd3210-b1db-42ed-91b6-9f8f3d945a34 - - - - -] Failed to initialize driver.: OSError: [Errno 2] No such file or directory
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager Traceback (most recent call last):
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager File "/openstack/venvs/cinder-19.0.0.0rc4.dev3/lib/python2.7/site-packages/cinder/volume/manager.py", line 469, in _init_host
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager self.driver.check_for_setup_error()
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager File "/openstack/venvs/cinder-19.0.0.0rc4.dev3/lib/python2.7/site-packages/cinder/volume/drivers/lvm.py", line 292, in check_for_setup_error
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager if volutils.supports_thin_provisioning():
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager File "/openstack/venvs/cinder-19.0.0.0rc4.dev3/lib/python2.7/site-packages/cinder/volume/utils.py", line 644, in supports_thin_provisioning
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager utils.get_root_helper())
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager File "/openstack/venvs/cinder-19.0.0.0rc4.dev3/lib/python2.7/site-packages/cinder/brick/local_dev/lvm.py", line 225, in supports_thin_provisioning
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager return LVM.get_lvm_version(root_helper) >= (2, 2, 95)
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager File "/openstack/venvs/cinder-19.0.0.0rc4.dev3/lib/python2.7/site-packages/cinder/brick/local_dev/lvm.py", line 202, in get_lvm_version
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager run_as_root=True)
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager File "/openstack/venvs/cinder-19.0.0.0rc4.dev3/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 391, in execute
                                           2019-07-08 19:27:51.618 25193 ERROR cinder.volume.manager env=env_variables)
                                           2019-07-0...