tripleo

Mitaka to Newton undercloud upgrade needs "nova db online_data_migrations" after the undercloud upgrade

Bug #1656791 reported by Marios Andreou
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
puppet-nova
Fix Released
High
Marios Andreou
tripleo
Fix Released
High
Marios Andreou
tripleo pike-1 "pike-1"

Bug Description

As first discussed at https://bugzilla.redhat.com/show_bug.cgi?id=1411856 when upgrading Mitaka->Newton->Ocata, you need to have run the nova db online_data_migrations after the Newton undercloud upgrade, otherwise the subsequent Ocata undercloud upgrade will fail:

Notice: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: ValidationError: Migration cannot continue until all these have been migrated to the api database. Please run `nova-manage db online_migrations' on Newton code before continuing.There are still 8 unmigrated flavors. ESC[0m
Error: /usr/bin/nova-manage db sync returned 1 instead of one of [0]ESC[0m
Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: change from notrun to 0 failed: /usr/bin/nova-manage db sync returned 1 instead of one of [0]

The db syncs usually happen in https://github.com/openstack/instack-undercloud/blob/554977801aad85f3ac08e16762fcdc8550c95754/elements/puppet-stack-config/puppet-stack-config.pp#L26 but since this is an upgrade only action, and a newton only one at that I think we should just fix this in tripleoclient (incoming review) and only for stable/newton.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to python-tripleoclient (stable/newton)

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/420637

Revision history for this message
Marios Andreou (marios-b) wrote :

in fact going to move to puppet-stack-config as per https://bugzilla.redhat.com/show_bug.cgi?id=1411856#c4

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to instack-undercloud (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/421262

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on python-tripleoclient (stable/newton)

Change abandoned by Marios Andreou (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/420637
Reason: abandoning in favor of adding to puppet-nova and then puppet-stack-config as discussed on the bug... see

https://review.openstack.org/#/c/421262/

Revision history for this message
Marios Andreou (marios-b) wrote :

adding puppet-nova because of the need to add the online-data-migrations there as well

affects: tripleo → puppet-nova
Changed in puppet-nova:
milestone: ocata-3 → none
affects: puppet-nova → tripleo
Revision history for this message
Marios Andreou (marios-b) wrote :

sorry for noise, i assumed I could add more than one project. assigning back to tripleo then

Revision history for this message
Marios Andreou (marios-b) wrote :

So, relevant reviews are

puppet-nova@ https://review.openstack.org/#/c/421249/ "Add a class to run the db online_data_migrations" Ibb119c280b5b6d07db06a63dd9efcc2526355f5f

instack-undercloud@ https://review.openstack.org/#/c/421262/ "Adds the nova db online data migrations to the list of db sync" Ib7a495491e8fa16ab62a1c415cef30ee18664b35

As discussed in the BZ mentioned above, I think we need these backported to stable/newton too

Changed in puppet-nova:
assignee: nobody → Marios Andreou (marios-b)
status: New → In Progress
Marios Andreou (marios-b)
Changed in tripleo:
milestone: none → ocata-3
Emilien Macchi (emilienm)
Changed in puppet-nova:
importance: Undecided → High
tags: added: puppet upgrade
Marios Andreou (marios-b)
Changed in tripleo:
milestone: ocata-3 → ocata-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to puppet-nova (master)

Reviewed: https://review.openstack.org/421249
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=50c1733e799025bc0c9c07745fc0fde7cf15b378
Submitter: Jenkins
Branch: master

commit 50c1733e799025bc0c9c07745fc0fde7cf15b378
Author: marios <email address hidden>
Date: Tue Jan 17 14:38:36 2017 +0200

    Add a class to run the db online_data_migrations

    This adds support for running the online_data_migrations
    which needs to happen after a dbsync when you upgrade to
    Newton.

    Change-Id: Ibb119c280b5b6d07db06a63dd9efcc2526355f5f
    Related-Bug: 1656791

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to instack-undercloud (master)

Reviewed: https://review.openstack.org/421262
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=bbbb9081018b8d5de6772ad55175ef444c2e855b
Submitter: Jenkins
Branch: master

commit bbbb9081018b8d5de6772ad55175ef444c2e855b
Author: marios <email address hidden>
Date: Tue Jan 17 15:01:06 2017 +0200

    Adds the nova db online data migrations to the list of db sync

    We aren't currently running the online_data_migrations nova db
    sync in puppet-stack-config

    Change-Id: Ib7a495491e8fa16ab62a1c415cef30ee18664b35
    Related-Bug: 1656791
    Depends-On: Ibb119c280b5b6d07db06a63dd9efcc2526355f5f

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to puppet-nova (stable/newton)

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/427623

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/newton)

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/427752

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to instack-undercloud (master)

Reviewed: https://review.openstack.org/425323
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=f4db8c8012a0336b1a365f5c84509c6077c22a88
Submitter: Jenkins
Branch: master

commit f4db8c8012a0336b1a365f5c84509c6077c22a88
Author: Sven Anderson <email address hidden>
Date: Wed Jan 25 18:57:02 2017 +0100

    Add a pre-upgrade online_data_migration step for undercloud

    A complete upgrade from M->N would require a db sync followed by a
    db online_data_migration step after the package upgrades. Because of
    the referred bug this however cannot be guaranteed to have happened,
    which would result in an upgrade failure when later upgrading from
    N->O. Therefore we need to do an additional online_data_migration
    before the package upgrades happen, to make sure the subsequent db
    sync will not fail.

    Related-Bug: #1656791
    Change-Id: I87dad6166e335577711ddb02156d7c5146e7ea96

Revision history for this message
Sven Anderson (ansiwen) wrote :

I created an etherpad to track all the different tasks that are required to solve this issue completely: https://etherpad.openstack.org/p/tripleo-nova-online-data-migration

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to instack-undercloud (stable/newton)

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/431590

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to puppet-nova (stable/newton)

Reviewed: https://review.openstack.org/427623
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=0e8534cb57f032a1bccf2bf1c9293d5748aaaaf8
Submitter: Jenkins
Branch: stable/newton

commit 0e8534cb57f032a1bccf2bf1c9293d5748aaaaf8
Author: marios <email address hidden>
Date: Tue Jan 17 14:38:36 2017 +0200

    Add a class to run the db online_data_migrations

    This adds support for running the online_data_migrations
    which needs to happen after a dbsync when you upgrade to
    Newton.

    Change-Id: Ibb119c280b5b6d07db06a63dd9efcc2526355f5f
    Related-Bug: 1656791
    (cherry picked from commit 50c1733e799025bc0c9c07745fc0fde7cf15b378)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to instack-undercloud (stable/newton)

Reviewed: https://review.openstack.org/431590
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=0e530b489d841af0bb83f25af5a8e8f0ccbd8f51
Submitter: Jenkins
Branch: stable/newton

commit 0e530b489d841af0bb83f25af5a8e8f0ccbd8f51
Author: marios <email address hidden>
Date: Tue Jan 17 15:01:06 2017 +0200

    Adds the nova db online data migrations to the list of db sync

    We aren't currently running the online_data_migrations nova db
    sync in puppet-stack-config

    Change-Id: Ib7a495491e8fa16ab62a1c415cef30ee18664b35
    Related-Bug: 1656791
    Depends-On: Ibb119c280b5b6d07db06a63dd9efcc2526355f5f
    (cherry picked from commit bbbb9081018b8d5de6772ad55175ef444c2e855b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/newton)

Reviewed: https://review.openstack.org/427752
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=47857c48098b5d7c9a778102702def3b467e85e2
Submitter: Jenkins
Branch: stable/newton

commit 47857c48098b5d7c9a778102702def3b467e85e2
Author: Sven Anderson <email address hidden>
Date: Wed Feb 1 15:53:46 2017 +0100

    Run online_data_migrations during minor updates.

    We didn't complete the M->N upgrades correctly with a `nova-manage db
    online_data_migrations` command before, which might result in a
    performance impairment. So, as a stop-gap-solution we run it here,
    which is a noop if there is nothing to do.

    Change-Id: I7c8b04c1a0640ce3222d22920f37492c6387759e
    Related-Bug: #1656791

Emilien Macchi (emilienm)
Changed in tripleo:
milestone: ocata-rc1 → ocata-rc2
Emilien Macchi (emilienm)
Changed in tripleo:
milestone: ocata-rc2 → pike-1
Marios Andreou (marios-b)
Changed in puppet-nova:
status: In Progress → Fix Released
Changed in tripleo:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.