Stale iSCSI sessions when detaching a volume from Windows VM while online

Bug #1431406 reported by Michael Steffens
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
New
Undecided
Unassigned
cinder (Ubuntu)
Triaged
Medium
Unassigned

Bug Description

We are regularly encountering this situation when deleting stacks managed by Heat. It can be reproduced without heat, however, just using Nova and Cinder:

1. Create a Windows guest VM, for example using CloudBase's image windows_server_2012_r2_standard_eval_kvm_20140607.

2. Create a Volume (50 GB) and attach volume to the instance.

3. Log into instance. Start Computer Mangement -> Disk Management.

4. Online the disk. Initialize and format the volume, assign drive letter D. Create some small garbage data on D:.

5. In Nova detach volume from instance. (Without shutting down the instance first. This is apparently what Heat does when deleting a stack.)

On the compute node you will now see dmesg and syslog being flooded with messages like

   [768938.979494] connection18:0: detected conn error (1020)

about once per second. On the compute node

  iscsiadm --mode session --print=1

displays the iSCSI initiatior session still logged in, while on the Cinder storage node

  tgtadm --lld iscsi --op show --mode target

shows that the iSCSI target is gone. The recurring connection errors on the compute node persist until manually logging off the iSCSI session. You may argue that performing the detachment while the volume is online and in use is unclean, therefore the issue being Heat's responsibility. However, even if that was the case, such an operation should not result in stale iSCSI sessions accumulating until manual intervention via root shell on the compute node.

Additional information:

 - We couldn't reproduce this problem with Linux quest instances. Even when detaching a volume while mounted and in use by the instance, iSCSI session are cleaned up gracefully.

 - We can reproduce this problem with both Icehouse and Juno.

 - We can reproduce the problem with both single and multi node OpenStack configurations, the latter using separate hosts for compute and storage.

Tags: iscsi windows
tags: added: iscsi windows
description: updated
Revision history for this message
Cory Stone (corystone) wrote :

I'm guessing this part is a typo, right?

"5. In Glance detach volume from instance." This is nova, right?

Which hypervisor is this? Also, are there nova logs showing any errors regarding the detach?

description: updated
description: updated
Revision history for this message
Michael Steffens (michael-steffens-b) wrote :

You are right of course, It's nova rather than glance, apologies. The hypervisor is kvm. I don't see anything suspicious in nova-compute.log, however, I'll attach an excerpt of nova-compute.log generated by such a heat stack destruction.

The result of this operation still is exactly as described above:

 - Instance is gone
 - Volume is retained (as requested in heat template for deletion_policy)
 - iSCSI initiatior is retained on compute node
 - stale connections reported on the compute node

Revision history for this message
Michael Steffens (michael-steffens-b) wrote :
Louis Bouchard (louis)
no longer affects: ubuntu
Revision history for this message
James Page (james-page) wrote :

This looks similar to bug 1436561

no longer affects: cinder (Ubuntu Trusty)
no longer affects: cinder (Ubuntu Vivid)
James Page (james-page)
Changed in cinder (Ubuntu):
importance: Undecided → Medium
Changed in cinder (Ubuntu):
status: New → Triaged
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.