iSCSI volume detach does not correctly remove the multipath device descriptors
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Invalid
|
Low
|
Unassigned | ||
Ubuntu Cloud Archive |
Fix Released
|
Undecided
|
Unassigned | ||
Icehouse |
Fix Released
|
Undecided
|
Billy Olsen | ||
Juno |
Won't Fix
|
Undecided
|
Unassigned | ||
Kilo |
Fix Released
|
Undecided
|
Billy Olsen | ||
nova (Ubuntu) |
Fix Released
|
Low
|
Unassigned | ||
Trusty |
Fix Released
|
Low
|
Billy Olsen |
Bug Description
[Impact]
iSCSI volume detach does not correctly remove the multipath device descriptors.
The multipath devices are left on the compute node and multipath tools will occaisionally send IOs to known multipath devices.
[Test Case]
tested environment:
nova-compute on Ubuntu 14.04.1, iscsi_use_
I created 3 cinder volumes and attached them to a nova instance. Then I detach them one by one. First 2 volumes volumes detached successfully. 3rd volume also successfully detached but ended up with failed multipaths.
Here is the terminal log for last volume detach.
openstack@
+------
|
ID
| Status | Name | Size | Volume Type | Bootable |
Attached to
|
+------
| 56a63288-
None
| false | 5bd68785-
+------
openstack@
Fri Sep 19 21:38:13 JST 2014
360060160cf0036
size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=-1 status=active
| |- 4:0:0:42 sdb 8:16 active undef running
| |- 5:0:0:42 sdd 8:48 active undef running
| |- 6:0:0:42 sdf 8:80 active undef running
| `- 7:0:0:42 sdh 8:112 active undef running
`-+- policy='round-robin 0' prio=-1 status=enabled
|- 11:0:0:42 sdp 8:240 active undef running
|- 8:0:0:42 sdj 8:144 active undef running
|- 9:0:0:42 sdl 8:176 active undef running
`- 10:0:0:42 sdn 8:208 active undef running
openstack@
Fri Sep 19 21:38:19 JST 2014
tcp: [10] 172.23.
tcp: [3] 172.23.
tcp: [4] 172.23.
tcp: [5] 172.23.
tcp: [6] 172.23.
tcp: [7] 172.23.
tcp: [8] 172.23.
tcp: [9] 172.23.
openstack@
56a63288-
openstack@
openstack@
+------
|
ID
| Status | Name | Size | Volume Type | Bootable |
Attached to
|
+------
| 56a63288-
None
| false | 5bd68785-
+------
openstack@
openstack@
+------
|
ID
| Status | Name | Size | Volume Type | Bootable | Attached to |
+------
| 56a63288-
None
| false |
+------
|
openstack@
Fri Sep 19 21:39:23 JST 2014
360060160cf0036
size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=-1 status=active
| |- #:#:#:# - #:# active undef running
| |- #:#:#:# - #:# active undef running
| |- #:#:#:# - #:# active undef running
| `- #:#:#:# - #:# active undef running
`-+- policy='round-robin 0' prio=-1 status=enabled
|- #:#:#:# - #:# active undef running
|- #:#:#:# - #:# active undef running
|- #:#:#:# - #:# active undef running
`- #:#:#:# - #:# active undef running
openstack@
Fri Sep 19 21:39:27 JST 2014
iscsiadm: No active sessions.
Then I manually removed the multipaths,
openstack@
openstack@
openstack@
I think the problem is in,
virt/libvirt/
def _disconnect_
End of this method executes following code to call remove_
return
Therefore, first two volumes worked fine. However, when it comes to the last device (in this case 3rd one), this method return without calling _remove_
if not in_use:
# disconnect if no other multipath devices with same iqn
return
It just disconnect them but not remove them.
One of the reasons why we have to remove them is,
https:/
IMO, we should call _remove_
[Regression Potential]
- Low: the change is relatively minor and allows the code to also remove the device when the last path is removed. Should a regression occur, it should be limited to iscsi-multipath device detachment, which is a small portion of installations.
Related branches
- Ubuntu Server Developers: Pending requested
-
Diff: 157 lines (+137/-0)3 files modifieddebian/changelog (+8/-0)
debian/patches/fix-iscsi-detach.patch (+128/-0)
debian/patches/series (+1/-0)
CVE References
Changed in nova: | |
assignee: | nobody → Sampath Priyankara (sampath-priyankara) |
Changed in nova: | |
status: | New → Confirmed |
assignee: | Sampath Priyankara (sampath-priyankara) → Rafael David Tinoco (inaddy) |
tags: | added: cts |
tags: | added: juno-backport-potential |
Changed in nova: | |
assignee: | nobody → Felipe Reyes (freyes) |
Changed in nova: | |
importance: | Undecided → Low |
Changed in nova (Ubuntu): | |
importance: | Undecided → Medium |
importance: | Medium → Low |
Changed in nova (Ubuntu Trusty): | |
importance: | Undecided → Low |
Changed in nova (Ubuntu): | |
status: | Confirmed → Triaged |
Changed in nova (Ubuntu Trusty): | |
status: | Confirmed → Triaged |
Changed in nova: | |
assignee: | Felipe Reyes (freyes) → Jorge Niedbalski (niedbalski) |
Changed in nova: | |
assignee: | Jorge Niedbalski (niedbalski) → Felipe Reyes (freyes) |
Changed in nova: | |
assignee: | Felipe Reyes (freyes) → Jorge Niedbalski (niedbalski) |
tags: | added: sts |
tags: | removed: juno-backport-potential |
tags: |
added: sts-sru removed: cts |
Changed in cloud-archive: | |
status: | Confirmed → Fix Released |
tags: | removed: sts-sru |
Adding archive with the openstack cinder and nova logs for Control and Compute nodes.
And also their kernel logs.
W1DEV103 is the Controller node. (Cinder-vol, scheduler, api are also in this host.)
W1CN103 is the Compute node. (nova-compute and its needed other process are running on this host.)
Following detach API request caused the problem. 3751-4bdc- a9bf-a54318b2d9 85.
The request id: req-1f8a6f05-
The PW for the zip file is: "lkjse89'&3" (without the quotes).