ssh driver aborts if a node that was requested to be powered down is already powered down

Looking at drivers/modules/ssh.py, it seems as though the intended design is that we attempt to power-off a node, then check to see if it's powered off, and then return success/fail

I seem to be hitting a case where the initial check says that the VM is powered off; but a subsequent attempt to power off fails because it's already off. I'm not certain whether this is a race condition, or if the initial check was wrong; but either way, the result is the same: the system is in the desired state, but we throw an unhanded exception.

I think it would make more sense to catch this exception and check again to see if the system is now powered off. If it is, there's no need to let the error bubble up - the system is now in the desired state, so let's go. Of course, if the problem is that the initial check didn't catch the fact that it's powered off, presumably the second check wouldn't catch it either..

Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: 2014-11-25 15:43:17.898 5904 WARNING ironic.conductor.manager [-] Error in tear_down of node bd44c80e-f7b3-497f-8417-84fd5cd7e9a1: Failed to execute command via SSH: LC_ALL=C /usr/bin/virsh --connect qemu:///system destroy baremetal_1.
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: Traceback (most recent call last):
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 455, in fire_timers
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: timer()
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: cb(*args, **kw)
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/eventlet/greenthread.py", line 212, in main
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: result = function(*args, **kwargs)
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/conductor/manager.py", line 808, in _do_node_tear_down
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: node.target_provision_state = states.NOSTATE
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/oslo/utils/excutils.py", line 82, in __exit__
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: six.reraise(self.type_, self.value, self.tb)
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/conductor/manager.py", line 800, in _do_node_tear_down
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: new_state = task.driver.deploy.tear_down(task)
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/conductor/task_manager.py", line 129, in wrapper
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: return f(*args, **kwargs)
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/drivers/modules/pxe.py", line 364, in tear_down
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: manager_utils.node_power_action(task, states.POWER_OFF)
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/conductor/task_manager.py", line 129, in wrapper
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: return f(*args, **kwargs)
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/conductor/utils.py", line 118, in node_power_action
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: "Error: %(error)s") % {'target': target_state, 'error': e}
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/oslo/utils/excutils.py", line 82, in __exit__
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: six.reraise(self.type_, self.value, self.tb)
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/conductor/utils.py", line 111, in node_power_action
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: task.driver.power.set_power_state(task, new_state)
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/conductor/task_manager.py", line 129, in wrapper
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: return f(*args, **kwargs)
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/drivers/modules/ssh.py", line 548, in set_power_state
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: state = _power_off(ssh_obj, driver_info)
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/drivers/modules/ssh.py", line 464, in _power_off
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: _ssh_execute(ssh_obj, cmd_to_power_off)
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: File "/opt/stack/venvs/ironic/local/lib/python2.7/site-packages/ironic/drivers/modules/ssh.py", line 270, in _ssh_execute
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: raise exception.SSHCommandFailed(cmd=cmd_to_exec)
Nov 25 15:43:17 undercloud-undercloud-gxpandxta2cg ironic-conductor: SSHCommandFailed: Failed to execute command via SSH: LC_ALL=C /usr/bin/virsh --connect qemu:///system destroy baremetal_1.