Failed to restart rabbitmq-server during OpenStack environment update

Bug #1361691 reported by Kirill Omelchenko
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
High
Fuel Library (Deprecated)
5.0.x
Fix Released
High
Fuel Library (Deprecated)

Bug Description

On the next env:
-CentOS, HA, 3xControllers, 1xComputes, 1xStorage
-Networking: nova flat
-Storage: Cinder LVM

Steps to reproduce:
1. Setup 5.0-26 iso master node
2. Add corresponding nodes and configure networks
3. Deploy
4. Run OSTF and Network Verification tests (both pass successfully)
5. Start an upgrade using fuel-master-upgrade-466 image.
6. Upgrade finishes successfully
7. Go to Fuel UI and start an update of an env

Expected:
Update finishes without any errors

Actual:
The update freezes on the firs controller at about 90%.
Overall update progress stopped at 14%.
After a while the whole process erred:
"Update has failed. Timeout of deployment is exceeded."

I found next errors in logs of the effected node:
http://paste.openstack.org/show/100386/

Revision history for this message
Kirill Omelchenko (komelchenko) wrote :
Changed in fuel:
assignee: Igor Kalnitsky (ikalnitsky) → Fuel Library Team (fuel-library)
status: New → Confirmed
Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

which version of environment did you choose to update from and to ?

Mike Scherbakov (mihgen)
summary: - Failed to restart reabbitmq-server during MOS environment update
+ Failed to restart rabbitmq-server during MOS environment update
summary: - Failed to restart rabbitmq-server during MOS environment update
+ Failed to restart rabbitmq-server during OpenStack environment update
Revision history for this message
Kirill Omelchenko (komelchenko) wrote :

The initial version of Fuel was 5.0 and upgraded to 5.1.

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

this test case is invalid as we do not upgrade 5.0 envs to 5.1

Revision history for this message
Kirill Omelchenko (komelchenko) wrote :

I have confused the versions. So in fact I was updating 5.0 env to 5.0.2 (2014.1.1-5.0.2 to be exact).

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

The problem is that 5.0.2 contains a workaround for bug https://bugs.launchpad.net/mos/+bug/1341656 which requires rabbitmq service restart which may fail for some reason. As soon as this workaround is replaced by proper implementation we will not need to restart rabbitmq server and this bug will be fixed. Let's keep it open until https://bugs.launchpad.net/mos/+bug/1341656 is fixed and then close.

Dmitry Pyzhov (dpyzhov)
no longer affects: fuel/5.1.x
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

The review currently outstanding for bug #1341656 is about terminating rabbitmq connections:
https://review.openstack.org/117500

Is there also a rabbitmq restart code somewhere that also needs to be backed out to resolve this bug?

Revision history for this message
Mike Scherbakov (mihgen) wrote :

Restart of Rabbit should be out of scope for patching story. We should restart no more than just openstack services, as we update only openstack & its python deps packages. If, for some reason, we update something which requires rabbit restart - then we should revert those changes to puppet modules.

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :
Revision history for this message
Kirill Omelchenko (komelchenko) wrote :

Verified on:

{

   "build_id": "2014-09-04_00-01-17",

   "ostf_sha": "4dcd99cc4bfa19f52d4b87ed321eb84ff03844da",

   "build_number": "496",

   "auth_required": true,

   "api": "1.0",

   "nailgun_sha": "3d965a7acd9361a361da92970d5bc5a1be77c969",

   "production": "docker",

   "fuelmain_sha": "435c43fe558898566ef8254ec09cc12bcb809c4b",

   "astute_sha": "8cc2c5b29d75ec2f60787e225564426a0508b3e4",

   "feature_groups": [

      "mirantis"

   ],

   "release": "5.1",

   "fuellib_sha": "bc23f1359b35ec90d5c7bc9f33778b4f2922d409"

}

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.