Fuel for OpenStack

db-migration: if instance.attributes.generated['deployed_before']['value'] != value: KeyError: 'deployed_before'

Bug #1551339 reported by Ekaterina Khomyakova
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Artem Roma
Fuel for OpenStack 9.0
8.0.x
Fix Released
High
Rodion Tikunov
Fuel for OpenStack 8.0-mu-2

Bug Description

Steps to reproduce:
1. Deploy environment with any settings using 7.0 Fuel
2. Upgrade fuel to 8.0 (ISO #570) using fuel-octane
3. Add Controller to the existing environment
4. Deploy the environment
5. Wait before all nodes will ready

Observed behavior:
All nodes are ready but task stays in progress.

Root cause:
After deployment astute reports to "deploy_resp" RPC method but:

 File "/usr/lib/python2.7/site-packages/nailgun/objects/cluster.py", line 366, in set_deployed_before_flag
 if instance.attributes.generated['deployed_before']['value'] != value: KeyError: 'deployed_before'
(full trace http://paste.openstack.org/show/489100/ )

Looks like db migration is broken and it does not updated required field

See original description
Revision history for this message
Ekaterina Khomyakova (ekhomyakova) wrote :
Revision history for this message
Ekaterina Khomyakova (ekhomyakova) wrote :
Ilya Bumarskov (ibumarskov)
tags: added: team-upgrades
Changed in fuel:
importance: Undecided → High
assignee: nobody → Fuel Octane Dev Team (fuel-octane)
Revision history for this message
Vladimir Khlyunev (vkhlyunev) wrote :

RCA: /etc/cobbler/authorized_keys in cobbler's docker was not updated with restored SSH public key.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-octane (master)

Fix proposed to branch: master
Review: https://review.openstack.org/286438

Changed in fuel:
assignee: Fuel Octane Dev Team (fuel-octane) → Oleg S. Gelbukh (gelbuhos)
status: New → In Progress
Oleg S. Gelbukh (gelbuhos)
Changed in fuel:
milestone: none → 9.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-octane (master)

Reviewed: https://review.openstack.org/286438
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=f78d006175cf9c158d3c196702b1db1a4d234ddf
Submitter: Jenkins
Branch: master

commit f78d006175cf9c158d3c196702b1db1a4d234ddf
Author: Oleg Gelbukh <email address hidden>
Date: Tue Mar 1 08:24:39 2016 +0000

    Restore the authorized_keys file in 'cobbler' container

    SSH restore must go before Cobbler restore to ensure
    that /etc/cobbler/authorized_keys file is updated
    with appropriate public key.

    Change-Id: Ie9493e883885cfc167920bfe35d5545d05c378e9
    Closes-bug: 1551339

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-octane (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/287303

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-octane (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/287304

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-octane (stable/8.0)

Reviewed: https://review.openstack.org/287303
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=3105a92a3bda61d7addbccf943e4f0daddb43165
Submitter: Jenkins
Branch: stable/8.0

commit 3105a92a3bda61d7addbccf943e4f0daddb43165
Author: Oleg Gelbukh <email address hidden>
Date: Tue Mar 1 08:24:39 2016 +0000

    Restore the authorized_keys file in 'cobbler' container

    SSH restore must go before Cobbler restore to ensure
    that /etc/cobbler/authorized_keys file is updated
    with appropriate public key.

    Change-Id: Ie9493e883885cfc167920bfe35d5545d05c378e9
    Closes-bug: 1551339
    (cherry picked from commit f78d006175cf9c158d3c196702b1db1a4d234ddf)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-octane (stable/7.0)

Reviewed: https://review.openstack.org/287304
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=2cf3c5dfe72c0db70a9be82b459df6185fe8b93d
Submitter: Jenkins
Branch: stable/7.0

commit 2cf3c5dfe72c0db70a9be82b459df6185fe8b93d
Author: Oleg Gelbukh <email address hidden>
Date: Tue Mar 1 08:24:39 2016 +0000

    Restore the authorized_keys file in 'cobbler' container

    SSH restore must go before Cobbler restore to ensure
    that /etc/cobbler/authorized_keys file is updated
    with appropriate public key.

    Change-Id: Ie9493e883885cfc167920bfe35d5545d05c378e9
    Closes-bug: 1551339
    (cherry picked from commit f78d006175cf9c158d3c196702b1db1a4d234ddf)

Revision history for this message
Vladimir Khlyunev (vkhlyunev) wrote : Re: Deploying task freezes at 100% in old environment after upgrade 7.0 > 8.0

Issue still exists

Changed in fuel:
status: Fix Committed → Confirmed
Vladimir Khlyunev (vkhlyunev)
description: updated
description: updated
Revision history for this message
Oleg S. Gelbukh (gelbuhos) wrote :

Is it correct to assume that, since it is the DB upgrade issue, it does not affect version 7.0?

Revision history for this message
Vladimir Khlyunev (vkhlyunev) wrote :

Yes, 7.0 does not affected

no longer affects: fuel/7.0.x
Oleg S. Gelbukh (gelbuhos)
Changed in fuel:
assignee: Oleg S. Gelbukh (gelbuhos) → nobody
assignee: nobody → Fuel Python Team (fuel-python)
Vladimir Khlyunev (vkhlyunev)
summary: - Deploying task freezes at 100% in old environment after upgrade 7.0 >
- 8.0
+ db-migration: if
+ instance.attributes.generated['deployed_before']['value'] != value:
+ KeyError: 'deployed_before'
Revision history for this message
Oleg S. Gelbukh (gelbuhos) wrote :

Note that this issue effectively renders the env unmanageable after the upgrade+scale-out operation.

Oleg S. Gelbukh (gelbuhos)
tags: added: feature-upgrade
removed: area-pce-vcenter team-upgrades vcenter
Dmitry Pyzhov (dpyzhov)
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Fuel Octane Dev Team (fuel-octane)
tags: added: area-python team-upgrades
Oleg S. Gelbukh (gelbuhos)
Changed in fuel:
status: Confirmed → In Progress
Oleg S. Gelbukh (gelbuhos)
Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Oleg S. Gelbukh (gelbuhos) wrote :

We implemented a workaround for this bug in upgrade automation scripts for 8.0 and 9.0. However, this issue still needs to be fixed in Nailgun code base and should be back ported to version 8.0-mu1.

Changed in fuel:
status: Fix Committed → Triaged
assignee: Fuel Octane Dev Team (fuel-octane) → Artem Roma (aroma-x)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (master)

Fix proposed to branch: master
Review: https://review.openstack.org/292804

Changed in fuel:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (master)

Reviewed: https://review.openstack.org/292804
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=cfa146e9dd7d1579fc4baa7580aedb8afb57414c
Submitter: Jenkins
Branch: master

commit cfa146e9dd7d1579fc4baa7580aedb8afb57414c
Author: Artem Roma <email address hidden>
Date: Tue Mar 15 12:37:07 2016 +0200

    Set 'deployed_before' attribute in cluster attributes if not present

    Before this change if master node had been upgraded to Fuel of
    versions >= 8.0, attempt to re/deploy existing clusters would have failed
    as attributes of those clusters would have lack 'deployed_before' field.

    Since processing of the flag is temporary solution it should not be
    coded in data base migration logic. Thus the change introduces creating
    of the attribute with given value if it is not already present.

    Change-Id: I302ddb340d2ecf14b1fa414fda71aa7f32363bdf
    Closes-Bug: #1551339

Changed in fuel:
status: In Progress → Fix Committed
Oleg S. Gelbukh (gelbuhos)
no longer affects: fuel/mitaka
Revision history for this message
Rodion Tikunov (rtikunov) wrote :

Please, provide here the link on documentation with instruction for how to upgrade MOS from 7.0 to 8.0. It needs for reproducing this bug.

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/fuel-web (stable/8.0)

Fix proposed to branch: stable/8.0
Change author: Artem Roma <email address hidden>
Review: https://review.fuel-infra.org/21257

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/321414

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Change abandoned on openstack/fuel-web (stable/8.0)

Change abandoned by Rodion Tikunov <email address hidden> on branch: stable/8.0
Review: https://review.fuel-infra.org/21257
Reason: Wrong repo

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-web (stable/8.0)

Reviewed: https://review.openstack.org/321414
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=48ee65f15e346cff46e0bf0cfba9dbe9afe43528
Submitter: Jenkins
Branch: stable/8.0

commit 48ee65f15e346cff46e0bf0cfba9dbe9afe43528
Author: Artem Roma <email address hidden>
Date: Tue Mar 15 12:37:07 2016 +0200

    Set 'deployed_before' attribute in cluster attributes if not present

    Before this change if master node had been upgraded to Fuel of
    versions >= 8.0, attempt to re/deploy existing clusters would have failed
    as attributes of those clusters would have lack 'deployed_before' field.

    Since processing of the flag is temporary solution it should not be
    coded in data base migration logic. Thus the change introduces creating
    of the attribute with given value if it is not already present.

    Change-Id: I302ddb340d2ecf14b1fa414fda71aa7f32363bdf
    Closes-Bug: #1551339
    (cherry picked from commit cfa146e9dd7d1579fc4baa7580aedb8afb57414c)

Dmitry Belyaninov (dbelyaninov)
tags: added: on-verification
Revision history for this message
Ekaterina Shutova (eshutova) wrote :

Verified on MOS 8.0 + MU3 updates.
Problem was not reproduced since some workaround was already merged to fuel-octane.
Steps from description were used for checking fix.

tags: removed: on-verification
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-web (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/340827

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.