Simultaneous adding/removing nodes failed on pre-deployment hook 'yum clean all'

Seems that it is related to orchestration.

2015-04-20T04:22:06 warning: [734] Problem while performing cmd. Try to repeat: 0 attempt
2015-04-20T04:22:07 debug: [734] 5b56cd90-0969-45aa-8544-1a5a67bc5934: MC agent 'execute_shell_command', method 'execute', results: {:sender=>"7", :statuscode=>0, :statusmsg=>"OK", :data=>{:stdout=>"", :ex
it_code=>127, :stderr=>"sh: yum: command not found\n"}}

For now I didn't reproduce it, I'll try again in some time.

>> sh: yum: command not found

This probably means that nodes were not provisioned and rebooted after that. In Nailgun we can see that node was not send for provisioning.

Andrey, was it reproduced with system tests or manually?

+1. This is mean that we run this tasks on 'bootstrap' nodes. Also looks like this bug with controllers connect with this https://bugs.launchpad.net/fuel/+bug/1394188

After observation i think that this bug is most similar: https://bugs.launchpad.net/fuel/+bug/1450051

Reproduced in https://bugs.launchpad.net/fuel/+bug/1452826

Fix proposed to branch: master
Review: https://review.openstack.org/181424

Guys, looks like we have problem in other place. I've checked logic and looks like original reason is this:

2015-04-20 04:17:14.036 INFO [7f2b1f8ae700] (receiver) RPC method remove_nodes_resp received: {"status": "ready", "progress": 100, "task_uuid": "5c7cfc4a-91ae-449b-8fb4-7745b2998839", "nodes": [{"uid": "7"}, {"uid": "3"}]}
2015-04-20 04:17:15.119 ERROR [7f2b1f8ae700] (receiverd) Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nailgun/rpc/receiverd.py", line 52, in consume_msg
    callback(**body["args"])
  File "/usr/lib/python2.6/site-packages/nailgun/rpc/receiver.py", line 90, in remove_nodes_resp
    objects.NodeCollection.lock_for_update(locked_nodes).all()
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2300, in all
    return list(self)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2412, in __iter__
    return self._execute_and_instances(context)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2427, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 729, in execute
    return meth(self, multiparams, params)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/sql/elements.py", line 321, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
    context)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception
    exc_info
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
    context)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
    cursor.execute(statement, parameters)
DBAPIError: (TransactionRollbackError) deadlock detected

Change abandoned by Vladimir Sharshov (<email address hidden>) on branch: master
Review: https://review.openstack.org/181424
Reason: I think that i try to find problem in wrong place.
https://bugs.launchpad.net/fuel/+bug/1446202/comments/9

The reason of error is not in Deadlock, but in wrong nodes list to deploy in the task

This fix https://review.openstack.org/#/c/183081/ can partially solve one of consequent then removed node present in the list. But if node already return to online state, we still have problem with wrong nodes list

Updated: this fix should solve consequent in both cases if problem node not critical. In our case problem node is Cinder with fail_if_error -> false.

But we still do not find original source of problem. Also this related fix will not help in case of critical role: mongo, primary-controller and so on.

Related fix proposed to branch: master
Review: https://review.openstack.org/184115

Reviewed: https://review.openstack.org/184245
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=ea7ea8ffff4b3f0f54ea21d6ef6ca9a01fe2f7aa
Submitter: Jenkins
Branch: master

commit ea7ea8ffff4b3f0f54ea21d6ef6ca9a01fe2f7aa
Author: Dmitry Shulyak <email address hidden>
Date: Tue May 19 16:23:40 2015 +0300

    Do not include pending deletion node in upgradable nodes

    Previously pending for deletion nodes were included in deployment
    if they were selected for update, by deploying other roles

    You can observe one example of this behaviour in test

    Closes-Bug: 1446202
    Change-Id: Ic80e1421eea6a3227fc8d890bfdbab933da72fbc

http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/6.1.custom_system_test/119/ (OK)
It is red, because of problem in middle between tests and Jenkins, but tests finished with success.