Fuel for OpenStack

Packages versions issue after env 5.0 is upgraded to 5.1

Series 6.0.x
Bug #1364081

Bug #1364081 reported by Dennis Dmitriev on 2014-09-01

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Fix Committed	High	Vladimir Kuklin	Fuel for OpenStack 5.1.1-updates
	6.0.x	Fix Committed	High	Dmitry Ilyin	Fuel for OpenStack 6.0-updates

Bug Description

Steps to reproduce:
      1. Deploy cluster 1 with release 2014.1 (5.0) and Ubuntu/HA
      2. Upload and unpack tarball with fuel-5.1 release (build #491) to /var
      3. run upgrade.sh with openstack
      4. Navigate to UI and update cluster 1 to release 2014.1.1-5.0.2
      5. Wait until update has finished and update cluster 1 back to release 2014.1 (5.0)

Expected result: all packages versions are returned to the same versions as it were before updating to 5.0.2

Actual result: after updating back to 5.0 several packages keep installed versions from 5.0.2 :

Package: python-oslo.messaging
Originally on the 5.0: 1.3.0~a9-0ubuntu1
After updating to 5.0.2: 1.3.0-fuel5.0.2~mira3
After updating back to 5.0: 1.3.0-fuel5.0.2~mira3

Package: python-sqlalchemy
Originally on the 5.0: 0.8.2-1~cloud1
After updating to 5.0.2: 0.8.2-ubuntu1
After updating back to 5.0: 0.8.2-ubuntu1

Package: python-sqlalchemy-ext
Originally on the 5.0: 0.8.2-1~cloud1
After updating to 5.0.2: 0.8.2-ubuntu1
After updating back to 5.0: 0.8.2-ubuntu1

Package: python-testtools
Originally on the 5.0: 0.9.32-0ubuntu1~cloud0
After updating to 5.0.2: 0.9.35-0ubuntu1
After updating back to 5.0: 0.9.35-0ubuntu1

May be connected with https://bugs.launchpad.net/mos/+bug/1359705

{"build_id": "2014-09-01_00-01-17", "ostf_sha": "4dcd99cc4bfa19f52d4b87ed321eb84ff03844da", "build_number": "491", "auth_required": true, "api": "1.0", "nailgun_sha": "d25ed02948a8be773e2bd87cfe583ef7be866bb2", "production": "docker", "fuelmain_sha": "109812be3425408dd7be192b5debf109cb1edd4c", "astute_sha": "bc60b7d027ab244039f48c505ac52ab8eb0a990c", "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha": "2cfa83119ae90b13a5bac6a844bdadfaf5aeb13f"}

Tags:

Dennis Dmitriev (ddmitriev) on 2014-09-01

Changed in mos:
milestone:	none → 5.1
affects:	mos → fuel
Changed in fuel:
milestone:	5.1 → none
milestone:	none → 5.1

Nastya Urlapova (aurlapova) on 2014-09-01

Changed in fuel:
assignee:	nobody → Fuel Library Team (fuel-library)

Revision history for this message

Vladimir Kuklin (vkuklin) wrote on 2014-09-02:

Dennis, which steps did you follow to do this? It looks like, older manifests and repos were not configured for this deployment and puppet just ran a regular deployment with newer manifests and newer packages. Actually, we do rollback only if deployment fails and then Nailgun switches repositories and manifests to older ones. I am not aware of the test case where we allow user to rollback his environment if update was successful.

Changed in fuel:
status:	New → Confirmed
importance:	Undecided → High

Revision history for this message

Vladimir Kuklin (vkuklin) wrote on 2014-09-02:

Also, Dennis, please, always add diagnostic snapshot to bug report

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Fuel Python Team (fuel-python)

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2014-09-02:

Vladimir, I did all of it through Fuel UI. When the cluster upgraded to 5.0.2, I ran OSTF and then selected "Actions" -> "Update" to 2014.1
Most of packages were returned to their previous versions except these ones.

Revision history for this message

Ihor Kalnytskyi (ikalnytskyi) wrote on 2014-09-02:

I've just re-test and can confirm that old repos and puppets were successfully synced to nodes.
So there's an issue of library side.

Changed in fuel:
assignee:	Fuel Python Team (fuel-python) → Fuel Library Team (fuel-library)

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2014-09-02:

Revision history for this message

Vladimir Kuklin (vkuklin) wrote on 2014-09-02:

Dennis, are you sure that rollback process ended succesfully? Because according to this bug patching rollback is broken now: https://bugs.launchpad.net/fuel/+bug/1364068

Revision history for this message

Vladimir Kuklin (vkuklin) wrote on 2014-09-02:

Also, please attach diagnostic snapshot

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2014-09-02:

As I see the bug https://bugs.launchpad.net/fuel/+bug/1364068 is about CentOS (I haven't finished with testing CentOS yet).
I will attach diagnostic snapshot after reproducing the bug on another environment.

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2014-09-09:

I have made update and rollback for CentOS and Ubuntu clusters.

Steps to reproduce:
        1. Create env with Fuel 5.0.26 (2014.1)
        2. Create CentOS cluster: (simple, neutron/vlan, ceilometer, ceph vol/ceph images, 1 controller+mongo, 4 compute+cinder+ceph)
        3. Store packages versions from cluster nodes to a file (5.0.0.26.orig.sorted.txt for centos and 5.0.0.26-before-update.sorted.txt for ubuntu)
        4. Upgrade env to 5.1 using http://jenkins-product.srt.mirantis.net:8080/job/fuel_master_iso/493/
        5. Update CentOS cluster to 5.0.2
        6. Store packages versions from cluster nodes to a file (5.0.2.versions.sorted.txt for centos and 5.0.2-updated.sorted.txt for ubuntu)
        7. Run OSTF
        8. Make diagnostic snapshot (centos-5.0.2-fuel-snapshot and ubuntu-5.0.2-fuel-snapshot was made)
        9. Rollback cluster to 5.0.0.26
        10. Store packages versions from cluster nodes to a file (5.0.0.26-reverted.versions.sorted.txt for centos and 5.0.0.26-reverted.sorted.txt for ubuntu)
        11. Make diagnostic snapshot (centos-reverted-to-5.0.0.26 and ubuntu-reverted-to-5.0.0.26)

The following are diagnostic snapshots:
centos-5.0.2-fuel-snapshot-2014-09-09_09-13-41.tgz
ubuntu-5.0.2-fuel-snapshot-2014-09-03_14-06-18.tgz
centos-reverted-to-5.0.0.26~ot-2014-09-09_11-43-48.tgz
ubuntu-reverted-to-5.0.0.26~ot-2014-09-03_14-06-18.tgz

The following are packages lists before updating cluster, after updating and after rollback (there is lists for controller and just one of compute nodes since they are the same):
centos-reverted-5.0.0.26.tar.gz
ubuntu-reverted-5.0.0.26.tar.gz

P.S. Updating of CentOS cluster to 5.0.2 failed, it is described in the https://bugs.launchpad.net/fuel/+bug/1367158 . Rollback for CentOS failed too. So on compute nodes some packages were not downgraded.

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2014-09-09:

#10

centos-5.0.2-fuel-snapshot-2014-09-09_09-13-41.tgz Edit (23.6 MiB, application/x-tar)

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2014-09-09:

#11

ubuntu-5.0.2-fuel-snapshot-2014-09-03_14-06-18.tgz Edit (9.7 MiB, application/x-tar)

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2014-09-09:

#12

centos-reverted-to-5.0.0.26-fuel-snapshot-2014-09-09_11-43-48.tgz Edit (28.3 MiB, application/x-tar)

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2014-09-09:

#13

ubuntu-reverted-to-5.0.0.26-fuel-snapshot-2014-09-03_14-06-18.tgz Edit (12.2 MiB, application/x-tar)

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2014-09-09:

#14

centos-reverted-5.0.0.26.tar.gz Edit (9.0 KiB, application/x-tar)

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2014-09-09:

#15

ubuntu-reverted-5.0.0.26.tar.gz Edit (114.1 KiB, application/x-tar)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-10: Fix proposed to fuel-astute (master)

#16

Fix proposed to branch: master
Review: https://review.openstack.org/120469

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Dmitry Ilyin (idv1985)
status:	Confirmed → In Progress

OpenStack Infra (hudson-openstack) on 2014-09-10

Changed in fuel:
assignee:	Dmitry Ilyin (idv1985) → Vladimir Kuklin (vkuklin)

OpenStack Infra (hudson-openstack) on 2014-09-10

Changed in fuel:
assignee:	Vladimir Kuklin (vkuklin) → Dmitry Ilyin (idv1985)

Revision history for this message

Vladimir Kuklin (vkuklin) wrote on 2014-09-10:

#17

https://review.openstack.org/#/c/120469/

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-10: Fix proposed to fuel-astute (stable/5.1)

#18

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/120479

OpenStack Infra (hudson-openstack) on 2014-09-11

Changed in fuel:
assignee:	Dmitry Ilyin (idv1985) → Vladimir Kuklin (vkuklin)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-11: Fix merged to fuel-astute (master)

#19

Reviewed: https://review.openstack.org/120469
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=a3e5da62af91b99f958ab958161d3dcec09c657b
Submitter: Jenkins
Branch: master

commit a3e5da62af91b99f958ab958161d3dcec09c657b
Author: Dmitry Ilyin <email address hidden>
Date: Wed Sep 10 20:21:51 2014 +0400

Add hooks to workaround patching bugs

Add several pre and post patching hooks and tests
for them.

* Stop or migrate corosync services from controller node
and return them after the deployment

* Stop openstack services and kill pids that was
not stopped together with their children

* Remove openstack packages before deployment
to prevent conflicts during the update and rollback

    Change-Id: Icfc17308d955e55b5e5b31a9c17e0b6d13d8ba10
    Closes-bug: #1364081
    Closes-bug: #1365635
    Closes-bug: #1364068
    Co-Authored-By: Vladimir Sharshov <email address hidden>
    Co-Authored-By: Bogdan Dobrelya <email address hidden>
    Co-Authored-By: Vladimir Kuklin <email address hidden>

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-11: Fix merged to fuel-astute (stable/5.1)

#20

Reviewed: https://review.openstack.org/120479
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13
Submitter: Jenkins
Branch: stable/5.1

commit f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13
Author: Dmitry Ilyin <email address hidden>
Date: Wed Sep 10 20:21:51 2014 +0400

Add hooks to workaround patching bugs

Add several pre and post patching hooks and tests
for them.

* Stop or migrate corosync services from controller node
and return them after the deployment

* Stop openstack services and kill pids that was
not stopped together with their children

* Remove openstack packages before deployment
to prevent conflicts during the update and rollback

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-12: Related fix proposed to fuel-astute (master)

#21

Related fix proposed to branch: master
Review: https://review.openstack.org/121194

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-22: Related fix proposed to fuel-library (stable/5.0)

#22

Related fix proposed to branch: stable/5.0
Review: https://review.openstack.org/123140

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-22: Change abandoned on fuel-astute (master)

#23

Change abandoned by Dmitry Ilyin (<email address hidden>) on branch: master
Review: https://review.openstack.org/121194

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-01-26: Change abandoned on fuel-library (stable/5.0)

#24

Change abandoned by Dmitry Ilyin (<email address hidden>) on branch: stable/5.0
Review: https://review.openstack.org/123140

Stanislaw Bogatkin (sbogatkin) on 2015-02-05

Changed in fuel:
milestone:	6.1 → 5.1.2

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.