Fuel for OpenStack

Rollback is impossible due to package conflicts

Bug #1364068 reported by Kirill Omelchenko
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
Critical
Vladimir Kuklin
Fuel for OpenStack 5.1
6.0.x
Won't Fix
Critical
Dmitry Ilyin
Fuel for OpenStack 6.0

Bug Description

Scenario:
1. Setup fuel maste node from 5.0-26 iso
2. Create a cluster of next conf:
- HA, CentOS, NovaVlan
- 1 Controller
- 1 Compute
- 1 Cinder
3. Setup (is successful)
4. upgrade master node with the help of the tar ball
(in my case it was fuel-master-upgrade-491)
5. Update the env from 2014.1 to 2014.1.1-5.0.2

Expected:
Update finishes successfully.

Actual:
Update has failed. Check these nodes:
'Untitled (xx:xx)'
During update the next error appeared in puppet log on the controller:
http://paste.openstack.org/show/104276/

More logs are in the attached snapshot.

See original description
Revision history for this message
Kirill Omelchenko (komelchenko) wrote :
Revision history for this message
Ihor Kalnytskyi (ikalnytskyi) wrote :

This bug should be resolved as a part of this one:

    https://bugs.launchpad.net/mos/+bug/1359705

Please make sure that python-route is know to be updated in the bug above.

Changed in fuel:
milestone: none → 5.1
status: New → Confirmed
Revision history for this message
Dmitry Mescheryakov (dmitrymex) wrote :

Igor, the bug you are referencing to is already fixed and, to my understanding, the fix is present in ISO #491

Changed in fuel:
importance: High → Critical
Revision history for this message
Dmitry Mescheryakov (dmitrymex) wrote :

Also, since patching is completely broken, setting importance to 'critical'

Ihor Kalnytskyi (ikalnytskyi)
Changed in fuel:
assignee: Igor Kalnitsky (ikalnitsky) → Fuel Library Team (fuel-library)
Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

Kirill, please comment, is this related to rollback? As I can see, puppet is trying to downgrade python-routes, but what is really weird is that it tries to install python routes 1.10.3-2.el6 which I cannot find in our repo.

Revision history for this message
Tomasz 'Zen' Napierala (tzn) wrote :

Is anyone working on this bug? If so, please assign it accordingly and set appriopriate status.

Vladimir Kuklin (vkuklin)
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Stanislaw Bogatkin (sbogatkin)
Revision history for this message
Kirill Omelchenko (komelchenko) wrote :

Vladimir it's just an update of the env, no downgrade took place.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-astute (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/118324

Vladimir Kuklin (vkuklin)
summary: - puppet: Couldn't update python-routes package during cluster update
+ puppet: Couldn't update python-routes package during cluster rollback
Revision history for this message
Vladimir Kuklin (vkuklin) wrote : Re: puppet: Couldn't update python-routes package during cluster rollback

In order to mitigate this issue we need to implement pre and post hooks and trigger them depending on the current version and future version of the environment.

Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

Specifically for this bug we should run

`yum downgrade python-nova openstack-nova-common openstack-nova-conductor openstack-nova-api openstack-nova-console openstack-nova-cert openstack-nova-objectstore openstack-nova-scheduler openstack-nova-novncproxy python-keystone openstack-keystone python-routes`

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-astute (master)

Fix proposed to branch: master
Review: https://review.openstack.org/118432

Changed in fuel:
assignee: Stanislaw Bogatkin (sbogatkin) → Vladimir Kuklin (vkuklin)
status: Confirmed → In Progress
Vladimir Kuklin (vkuklin)
summary: - puppet: Couldn't update python-routes package during cluster rollback
+ Rollback is impossible due to package conflicts
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Vladimir Kuklin (vkuklin) → Vladimir Sharshov (vsharshov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-astute (master)

Reviewed: https://review.openstack.org/118324
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=3cb89755ab7d246b103aa3dc613c47df13cf4e80
Submitter: Jenkins
Branch: master

commit 3cb89755ab7d246b103aa3dc613c47df13cf4e80
Author: Vladimir Sharshov <email address hidden>
Date: Tue Sep 2 16:45:23 2014 +0400

    Pre deploy actions

    Change-Id: I2e6b5f33c68124422f49248c07893cabad4c4e4f
    Related-Bug: #1364068

OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Vladimir Sharshov (vsharshov) → Vladimir Kuklin (vkuklin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-astute (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/119105

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-astute (master)

Fix proposed to branch: master
Review: https://review.openstack.org/119209

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-astute (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/119221

Kirill Omelchenko (komelchenko)
description: updated
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Vladimir Kuklin (vkuklin) → Vladimir Sharshov (vsharshov)
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Vladimir Sharshov (vsharshov) → Vladimir Kuklin (vkuklin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-astute (master)

Change abandoned by Vladimir Kuklin (<email address hidden>) on branch: master
Review: https://review.openstack.org/119221

Evgeniy L (rustyrobot)
tags: added: patching
Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

https://review.openstack.org/#/c/120469/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Vladimir Kuklin (<email address hidden>) on branch: master
Review: https://review.openstack.org/119105

Changed in fuel:
assignee: Vladimir Kuklin (vkuklin) → Dmitry Ilyin (idv1985)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Vladimir Kuklin (<email address hidden>) on branch: master
Review: https://review.openstack.org/119209

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Vladimir Kuklin (<email address hidden>) on branch: master
Review: https://review.openstack.org/118432

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-astute (stable/5.1)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/120479

OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Dmitry Ilyin (idv1985) → Vladimir Kuklin (vkuklin)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-astute (master)

Reviewed: https://review.openstack.org/120469
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=a3e5da62af91b99f958ab958161d3dcec09c657b
Submitter: Jenkins
Branch: master

commit a3e5da62af91b99f958ab958161d3dcec09c657b
Author: Dmitry Ilyin <email address hidden>
Date: Wed Sep 10 20:21:51 2014 +0400

    Add hooks to workaround patching bugs

    Add several pre and post patching hooks and tests
    for them.

    * Stop or migrate corosync services from controller node
      and return them after the deployment

    * Stop openstack services and kill pids that was
      not stopped together with their children

    * Remove openstack packages before deployment
      to prevent conflicts during the update and rollback

    Change-Id: Icfc17308d955e55b5e5b31a9c17e0b6d13d8ba10
    Closes-bug: #1364081
    Closes-bug: #1365635
    Closes-bug: #1364068
    Co-Authored-By: Vladimir Sharshov <email address hidden>
    Co-Authored-By: Bogdan Dobrelya <email address hidden>
    Co-Authored-By: Vladimir Kuklin <email address hidden>

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-astute (stable/5.1)

Reviewed: https://review.openstack.org/120479
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13
Submitter: Jenkins
Branch: stable/5.1

commit f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13
Author: Dmitry Ilyin <email address hidden>
Date: Wed Sep 10 20:21:51 2014 +0400

    Add hooks to workaround patching bugs

    Add several pre and post patching hooks and tests
    for them.

    * Stop or migrate corosync services from controller node
      and return them after the deployment

    * Stop openstack services and kill pids that was
      not stopped together with their children

    * Remove openstack packages before deployment
      to prevent conflicts during the update and rollback

    Change-Id: Icfc17308d955e55b5e5b31a9c17e0b6d13d8ba10
    Closes-bug: #1364081
    Closes-bug: #1365635
    Closes-bug: #1364068
    Co-Authored-By: Vladimir Sharshov <email address hidden>
    Co-Authored-By: Bogdan Dobrelya <email address hidden>
    Co-Authored-By: Vladimir Kuklin <email address hidden>

Irina Povolotskaya (ipovolotskaya)
tags: added: release-notes
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.