puppet-cinder

lvm: Volume attachment fails while setting up iscsi targets

Bug #1986518 reported by Takashi Kajinami on 2022-08-15

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	puppet-cinder	Fix Released	High	Takashi Kajinami

Bug Description

Currently scenario 005 job which has glance cinder backend is consistently failing because image creation fails.

https://zuul.opendev.org/t/openstack/build/e85ae399c2004d0dbea0a8c70e348d47

Looking at cinder log, it was confirmed cinder-volume could not set up iscsid targets correctly because of missing include line in /etc/tgt/targets.conf .

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e85/853070/2/check/puppet-openstack-integration-6-scenario005-tempest-ubuntu-focal/e85ae39/logs/cinder/cinder-volume.txt

2022-08-15 09:03:32.314 40690 WARNING cinder.volume.targets.tgt [req-301687b3-8535-4a00-a9a6-72d0b33fe782 bc52ace924554fb3ae19eadd73afa99e 827efe92f7a741bd9845dc996be72f7a - - -] Failed to create iscsi target for Volume ID: volume-a2aafaa1-fa10-469d-bb28-2a0d9cf6776b. It could be caused by problem with concurrency. Also please ensure your tgtd config file contains 'include /var/lib/cinder/volumes/*'
...
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager [req-301687b3-8535-4a00-a9a6-72d0b33fe782 bc52ace924554fb3ae19eadd73afa99e 827efe92f7a741bd9845dc996be72f7a - - -] Create export for volume failed (Resource could not be found.).: cinder.exception.NotFound: Resource could not be found.

2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager Traceback (most recent call last):
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 4785, in _connection_create
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager model_update = self.driver.create_export(ctxt.elevated(),
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/drivers/lvm.py", line 833, in create_export
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager export_info = self.target_driver.create_export(
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/targets/iscsi.py", line 194, in create_export
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager tid = self.create_iscsi_target(iscsi_name,
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/utils.py", line 664, in _wrapper
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager return r(f, *args, **kwargs)
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 404, in __call__
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager do = self.iter(retry_state=retry_state)
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 360, in iter
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager raise retry_exc.reraise()
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 193, in reraise
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager raise self.last_attempt.result()
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager return self.__get_result()
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager raise self._exception
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 407, in __call__
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager result = fn(*args, **kwargs)
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager File "/usr/lib/python3/dist-packages/cinder/volume/targets/tgt.py", line 207, in create_iscsi_target
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager raise exception.NotFound()
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager cinder.exception.NotFound: Resource could not be found.
2022-08-15 09:03:32.315 40690 ERROR cinder.volume.manager

Tags:

Takashi Kajinami (kajinamit) on 2022-08-15

Changed in puppet-cinder:
importance:	Undecided → High
assignee:	nobody → Takashi Kajinami (kajinamit)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-08-15: Fix proposed to puppet-cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/puppet-cinder/+/853155

Changed in puppet-cinder:
status:	New → In Progress

Revision history for this message

Takashi Kajinami (kajinamit) wrote on 2022-08-15:

In the past the cinder-common container provided the additional config file for tgt (/etc/tgt/conf.d/cinder_tgt.conf). However the file is no longer provided by the package since Wallaby.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-08-17: Fix merged to puppet-cinder (master)

Reviewed: https://review.opendev.org/c/openstack/puppet-cinder/+/853155
Committed: https://opendev.org/openstack/puppet-cinder/commit/94e4e8c5aeec35a08ebf99ef81e0260b09d35477
Submitter: "Zuul (22348)"
Branch: master

commit 94e4e8c5aeec35a08ebf99ef81e0260b09d35477
Author: Takashi Kajinami <email address hidden>
Date: Mon Aug 15 18:52:03 2022 +0900

lvm: Modify targets.conf in Ubuntu/Debian

    The cinder-common package in Ubuntu and Debian used to provide
    the additional config file for tgt daemon to allow the daemon to
    expose volumes managed by cinder. However that config file is no longer
    provided by the package since Wallaby release and now creating
    attachment fails with the following warning in cinder-volume.log.

    ```
    Failed to create iscsi target for Volume ID: volume-<UUID>. It could be
    caused by problem with concurrency. Also please ensure your tgtd config
    file contains 'include /var/lib/cinder/volumes/*'
    ```

This change ensures the include line is added in Ubuntu/Debian to fix
the error.

Closes-Bug: #1986518
Change-Id: I0c0615c919447164718c265af603fbc4f5e66b1e

Changed in puppet-cinder:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-08-19: Fix proposed to puppet-cinder (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/puppet-cinder/+/853643

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-08-19: Fix merged to puppet-cinder (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/puppet-cinder/+/853643
Committed: https://opendev.org/openstack/puppet-cinder/commit/34b9f69b682be62ba201d6199935cc59dd441df6
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 34b9f69b682be62ba201d6199935cc59dd441df6
Author: Takashi Kajinami <email address hidden>
Date: Mon Aug 15 18:52:03 2022 +0900

lvm: Modify targets.conf in Ubuntu/Debian

This change ensures the include line is added in Ubuntu/Debian to fix
the error.

    Closes-Bug: #1986518
    Change-Id: I0c0615c919447164718c265af603fbc4f5e66b1e
    (cherry picked from commit 94e4e8c5aeec35a08ebf99ef81e0260b09d35477)

tags:

added: in-stable-yoga

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-08-29: Fix proposed to puppet-cinder (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/puppet-cinder/+/854899

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-08-30: Fix merged to puppet-cinder (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/puppet-cinder/+/854899
Committed: https://opendev.org/openstack/puppet-cinder/commit/605430e6c437681f46c4dd309d84266b83fbb99c
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 605430e6c437681f46c4dd309d84266b83fbb99c
Author: Takashi Kajinami <email address hidden>
Date: Mon Aug 15 18:52:03 2022 +0900

lvm: Modify targets.conf in Ubuntu/Debian

This change ensures the include line is added in Ubuntu/Debian to fix
the error.

    Closes-Bug: #1986518
    Change-Id: I0c0615c919447164718c265af603fbc4f5e66b1e
    (cherry picked from commit 94e4e8c5aeec35a08ebf99ef81e0260b09d35477)
    (cherry picked from commit 34b9f69b682be62ba201d6199935cc59dd441df6)

tags:

added: in-stable-xena

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-08-30: Fix proposed to puppet-cinder (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/puppet-cinder/+/855137

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-09-01: Fix merged to puppet-cinder (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/puppet-cinder/+/855137
Committed: https://opendev.org/openstack/puppet-cinder/commit/f89314dc53ea874ffee9022924c66b67b85eabfb
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit f89314dc53ea874ffee9022924c66b67b85eabfb
Author: Takashi Kajinami <email address hidden>
Date: Mon Aug 15 18:52:03 2022 +0900

lvm: Modify targets.conf in Ubuntu/Debian

This change ensures the include line is added in Ubuntu/Debian to fix
the error.

tags:

added: in-stable-wallaby

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-10-20: Fix included in openstack/puppet-cinder 21.0.0

#10

This issue was fixed in the openstack/puppet-cinder 21.0.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-11-07: Fix included in openstack/puppet-cinder 18.5.1

#11

This issue was fixed in the openstack/puppet-cinder 18.5.1 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-05-17: Fix included in openstack/puppet-cinder 19.5.0

#12

This issue was fixed in the openstack/puppet-cinder 19.5.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-08-01: Fix included in openstack/puppet-cinder 20.4.0

#13

This issue was fixed in the openstack/puppet-cinder 20.4.0 release.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.