Powering multiple SM15k nodes fails with HTTP error 401

Bug #1302828 reported by Blake Rouse
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Invalid
Critical
Unassigned
python-seamicroclient (Ubuntu)
Fix Released
Critical
Unassigned

Bug Description

How to Reproduce:

Check multiple SM15k nodes in MAAS, using the drop-down select "Start selected nodes". Some nodes will succeed, others will fail with PowerActionFail. In this case 10 nodes were selected, and more than one failed.

Error Output:

[2014-04-04 20:41:35,135: ERROR/Worker-8] power_on: sm15k failed: Command '# -*- mode: shell-script -*-
#
# Control a system via ipmipower, sending the seamicro specific hex codes
#

# Parameters
power_change=on
power_address=***.***.***.***
power_user=admin
power_pass=******
power_control=restapi2
system_id=48
ipmitool=/usr/bin/ipmitool

# IPMI power mode
power_mode=1

# Control power using IPMI
issue_ipmi_command() {
    ${ipmitool} -I lanplus \
        -H ${power_address} -U ${power_user}\
        -P ${power_pass} raw 0x2E 1 0x00 0x7d 0xab \
        ${power_mode} 0 ${system_id}
}

# Control power using REST v0.9
issue_rest_v09_command() {
python - << END
from provisioningserver.custom_hardware.seamicro import power_control_seamicro15k_v09
power_control_seamicro15k_v09("${power_address}", "${power_user}", "${power_pass}", "${system_id}", "${power_change}")
END
}

# Control power using REST v2
issue_rest_v2_command() {
python - << END
from provisioningserver.custom_hardware.seamicro import power_control_seamicro15k_v2
power_control_seamicro15k_v2("${power_address}", "${power_user}", "${power_pass}", "${system_id}", "${power_change}")
END
}

if [ "${power_control}" = "ipmi" ]
then
    issue_ipmi_command
elif [ "${power_control}" = "restapi" ]
then
    issue_rest_v09_command
elif [ "${power_control}" = "restapi2" ]
then
    issue_rest_v2_command
fi
' returned non-zero exit status 1:
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/lib/python2.7/dist-packages/provisioningserver/custom_hardware/seamicro.py", line 331, in power_control_seamicro15k_v2
    server.power_on()
  File "/usr/lib/python2.7/dist-packages/seamicroclient/v2/servers.py", line 28, in power_on
    self.manager.power_on(self, using_pxe)
  File "/usr/lib/python2.7/dist-packages/seamicroclient/v2/servers.py", line 114, in power_on
    self._action('power-on', server, action_params)
  File "/usr/lib/python2.7/dist-packages/seamicroclient/v2/servers.py", line 219, in _action
    return self.api.client.put(url, body=body)
  File "/usr/lib/python2.7/dist-packages/seamicroclient/client.py", line 194, in put
    return self._cs_request(url, 'PUT', **kwargs)
  File "/usr/lib/python2.7/dist-packages/seamicroclient/client.py", line 185, in _cs_request
    raise e
seamicroclient.exceptions.Unauthorized: Failure: SESSION_INVALID (HTTP 401)
[2014-04-04 20:41:35,138: ERROR/MainProcess] Task provisioningserver.tasks.power_on[de917f8f-2d54-4efd-b541-bdc5fc81acfc] raised unexpected: PowerActionFail()
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 218, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 398, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/provisioningserver/tasks.py", line 93, in wrapper
    func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/provisioningserver/tasks.py", line 179, in power_on
    issue_power_action(power_type, 'on', **kwargs)
  File "/usr/lib/python2.7/dist-packages/provisioningserver/tasks.py", line 164, in issue_power_action
    pa.execute(**kwargs)
  File "/usr/lib/python2.7/dist-packages/provisioningserver/power/poweraction.py", line 133, in execute
    self.run_shell(rendered)
  File "/usr/lib/python2.7/dist-packages/provisioningserver/power/poweraction.py", line 121, in run_shell
    raise PowerActionFail(self, e)
PowerActionFail

Revision history for this message
Andres Rodriguez (andreserl) wrote :

This is not really a MAAS issue, but rather a bug with the SM15K API. We are keeping it open for tracking purposes.

description: updated
Revision history for this message
Andres Rodriguez (andreserl) wrote :

Though I believe we can do a work in MAAS code to re-try powering on nodes whenever this appears.

Revision history for this message
Julian Edwards (julian-edwards) wrote :

Yes, we should be doing some work this cycle to make the power mechanism much more robust.

Changed in maas:
status: Confirmed → Triaged
importance: Undecided → Medium
tags: added: power
Revision history for this message
Blake Rouse (blake-rouse) wrote :

Also powering multiple nodes at once, can generate a connection refused. Assuming this is also due to having to many connections at once.

Revision history for this message
Blake Rouse (blake-rouse) wrote :

Sometimes I receive this error when powering on a node. It appears randomly.

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/lib/python2.7/dist-packages/provisioningserver/custom_hardware/seamicro.py", line 333, in power_control_seamicro15k_v2
    server.power_on(using_pxe=True)
  File "/usr/lib/python2.7/dist-packages/seamicroclient/v2/servers.py", line 28, in power_on
    self.manager.power_on(self, using_pxe)
  File "/usr/lib/python2.7/dist-packages/seamicroclient/v2/servers.py", line 114, in power_on
    self._action('power-on', server, action_params)
  File "/usr/lib/python2.7/dist-packages/seamicroclient/v2/servers.py", line 219, in _action
    return self.api.client.put(url, body=body)
  File "/usr/lib/python2.7/dist-packages/seamicroclient/client.py", line 194, in put
    return self._cs_request(url, 'PUT', **kwargs)
  File "/usr/lib/python2.7/dist-packages/seamicroclient/client.py", line 166, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/seamicroclient/client.py", line 147, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/seamicroclient/client.py", line 121, in request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.220.102', port=80): Max retries exceeded with url: /v2.0/servers/9/0 (Caused by <class 'socket.error'>: [Errno 111] Connection refused)

Revision history for this message
Andres Rodriguez (andreserl) wrote :

Invalidating this for MAAS< as this bug has been found to be realted to python-seamicroclient.

Changed in maas:
status: Triaged → Confirmed
importance: Medium → Critical
Changed in python-seamicroclient (Ubuntu):
status: New → Confirmed
importance: Undecided → Critical
Changed in maas:
status: Confirmed → Invalid
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-seamicroclient - 0.2.0-0ubuntu1

---------------
python-seamicroclient (0.2.0-0ubuntu1) trusty; urgency=medium

  * New upstream release (LP: #1305220)
    - Remove use of auth_token for REST API. Fixes concurrent access by
      using user/password each time instead of auth_token (LP: #1302828)
  * debian/watch: Updated to include correct URL link.
 -- Andres Rodriguez <email address hidden> Wed, 09 Apr 2014 13:37:27 -0400

Changed in python-seamicroclient (Ubuntu):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.