Bug #1794364 “'nova-manage db online_data_migrations' count fail...” : Series pike : Bugs : OpenStack Compute (nova)

melanie witt (melwitt) on 2018-09-25

tags:

added: nova-manage

Revision history for this message

Matt Riedemann (mriedem) wrote on 2018-09-25:

#1

(3:05:42 PM) mriedem: that populate_uuids migration was added in queens, and looking at a grenade run from queens, we should see bdms getting migrated to have a uuid, and i'm not seeing any results in the table for a grenade run on queens
(3:05:42 PM) mriedem: http://logs.openstack.org/48/604448/1/check/neutron-grenade/19d8166/logs/grenade.sh.txt.gz
(3:05:46 PM) mriedem: so yeah, likely busted

Changed in nova:
status:	New → Confirmed
importance:	Undecided → High
tags:	added: upgrade

iain MacDonnell (imacdonn) on 2018-09-26

Changed in nova:
assignee:	nobody → iain MacDonnell (imacdonn)
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-26: Fix proposed to nova (master)

#2

Fix proposed to branch: master
Review: https://review.openstack.org/605329

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-27: Fix proposed to nova (stable/rocky)

#3

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/605828

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-27: Related fix proposed to nova (master)

#4

Related fix proposed to branch: master
Review: https://review.openstack.org/605836

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-27: Fix proposed to nova (stable/queens)

#5

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/605839

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-27: Fix proposed to nova (stable/pike)

#6

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/605840

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-27: Fix proposed to nova (stable/ocata)

#7

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/605842

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-28: Fix merged to nova (master)

#8

Reviewed: https://review.openstack.org/605329
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c4c6dc736e084f2d919b9bcf93d26df87e0341c9
Submitter: Zuul
Branch: master

commit c4c6dc736e084f2d919b9bcf93d26df87e0341c9
Author: imacdonn <email address hidden>
Date: Wed Sep 26 04:30:49 2018 +0000

nova-manage - fix online_data_migrations counts

    When running online_data_migrations in batches, totals were not
    being accumulated - rather the counts each batch would clobber
    those from the previous one, and the last batch would run no
    migrations, so the totals were reported as zero.

Change-Id: Ib616f2efb69baa16e18601d27b747220bbefeb16
Closes-Bug: #1794364

Changed in nova:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-02: Fix merged to nova (stable/rocky)

#9

Reviewed: https://review.openstack.org/605828
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fc9c8772012b528176c3f58d99a58491bb1aedb4
Submitter: Zuul
Branch: stable/rocky

commit fc9c8772012b528176c3f58d99a58491bb1aedb4
Author: imacdonn <email address hidden>
Date: Wed Sep 26 04:30:49 2018 +0000

nova-manage - fix online_data_migrations counts

    When running online_data_migrations in batches, totals were not
    being accumulated - rather the counts each batch would clobber
    those from the previous one, and the last batch would run no
    migrations, so the totals were reported as zero.

    Change-Id: Ib616f2efb69baa16e18601d27b747220bbefeb16
    Closes-Bug: #1794364
    (cherry picked from commit c4c6dc736e084f2d919b9bcf93d26df87e0341c9)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-02: Fix merged to nova (stable/queens)

#10

Reviewed: https://review.openstack.org/605839
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d8b99f05abb2fa8964a326958804f47daf95da64
Submitter: Zuul
Branch: stable/queens

commit d8b99f05abb2fa8964a326958804f47daf95da64
Author: imacdonn <email address hidden>
Date: Wed Sep 26 04:30:49 2018 +0000

nova-manage - fix online_data_migrations counts

    When running online_data_migrations in batches, totals were not
    being accumulated - rather the counts each batch would clobber
    those from the previous one, and the last batch would run no
    migrations, so the totals were reported as zero.

    Change-Id: Ib616f2efb69baa16e18601d27b747220bbefeb16
    Closes-Bug: #1794364
    (cherry picked from commit c4c6dc736e084f2d919b9bcf93d26df87e0341c9)
    (cherry picked from commit fc9c8772012b528176c3f58d99a58491bb1aedb4)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-03: Fix merged to nova (stable/pike)

#11

Reviewed: https://review.openstack.org/605840
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=25e73d5cbf66984f69deb0e808a25f97772bd9dd
Submitter: Zuul
Branch: stable/pike

commit 25e73d5cbf66984f69deb0e808a25f97772bd9dd
Author: imacdonn <email address hidden>
Date: Wed Sep 26 04:30:49 2018 +0000

nova-manage - fix online_data_migrations counts

    When running online_data_migrations in batches, totals were not
    being accumulated - rather the counts each batch would clobber
    those from the previous one, and the last batch would run no
    migrations, so the totals were reported as zero.

    Change-Id: Ib616f2efb69baa16e18601d27b747220bbefeb16
    Closes-Bug: #1794364
    (cherry picked from commit c4c6dc736e084f2d919b9bcf93d26df87e0341c9)
    (cherry picked from commit fc9c8772012b528176c3f58d99a58491bb1aedb4)
    (cherry picked from commit d8b99f05abb2fa8964a326958804f47daf95da64)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-04: Fix merged to nova (stable/ocata)

#12

Reviewed: https://review.openstack.org/605842
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5d33e554e4c96cec85274e8f46f3a7fa2b040b17
Submitter: Zuul
Branch: stable/ocata

commit 5d33e554e4c96cec85274e8f46f3a7fa2b040b17
Author: imacdonn <email address hidden>
Date: Wed Sep 26 04:30:49 2018 +0000

nova-manage - fix online_data_migrations counts

    When running online_data_migrations in batches, totals were not
    being accumulated - rather the counts each batch would clobber
    those from the previous one, and the last batch would run no
    migrations, so the totals were reported as zero.

    Change-Id: Ib616f2efb69baa16e18601d27b747220bbefeb16
    Closes-Bug: #1794364
    (cherry picked from commit c4c6dc736e084f2d919b9bcf93d26df87e0341c9)
    (cherry picked from commit fc9c8772012b528176c3f58d99a58491bb1aedb4)
    (cherry picked from commit d8b99f05abb2fa8964a326958804f47daf95da64)
    (cherry picked from commit 25e73d5cbf66984f69deb0e808a25f97772bd9dd)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-08: Fix included in openstack/nova 18.0.2

#13

This issue was fixed in the openstack/nova 18.0.2 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-08: Fix included in openstack/nova 16.1.6

#14

This issue was fixed in the openstack/nova 16.1.6 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-08: Fix included in openstack/nova 15.1.5

#15

This issue was fixed in the openstack/nova 15.1.5 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-08: Fix included in openstack/nova 17.0.7

#16

This issue was fixed in the openstack/nova 17.0.7 release.

iain MacDonnell (imacdonn) on 2018-10-16

Changed in cinder:
assignee:	nobody → iain MacDonnell (imacdonn)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-17: Fix proposed to cinder (master)

#17

Fix proposed to branch: master
Review: https://review.openstack.org/611463

Changed in cinder:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-26: Related fix merged to nova (master)

#18

Reviewed: https://review.openstack.org/605836
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c86f309c564f4a394285068bbdb4f138cfff1651
Submitter: Zuul
Branch: master

commit c86f309c564f4a394285068bbdb4f138cfff1651
Author: Matt Riedemann <email address hidden>
Date: Thu Sep 27 14:53:10 2018 -0400

Add more documentation for online_data_migrations CLI

    This is a follow up to commit c4c6dc736 to clarify some
    confusing comments in the code, add more comments in
    the actual runtime code, and also provide an example
    in the CLI man page docs along with an explanation of
    the output, specifically for the case that $found>0
    but done=0 and what that means.

Change-Id: I0691caab2c44d3189504c54e51bb263ecdc5d1d2
Related-Bug: #1794364

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-31: Fix merged to cinder (master)

#19

Reviewed: https://review.openstack.org/611463
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=d47486d317e8cdb1cdab50d73f6484289ab082d4
Submitter: Zuul
Branch: master

commit d47486d317e8cdb1cdab50d73f6484289ab082d4
Author: imacdonn <email address hidden>
Date: Wed Oct 17 22:48:18 2018 +0000

cinder-manage online_data_migrations fixes

Addresses some issues with this command:

    1) When used without the --max-count option, the summary table will
       always show zero migrations run, because it only accounts for the
       last batch, and the loop only exits when the last batch does no work.

    2) "remaining" counts cannot be accurate, given the way migrations are
       implemented, because the "found" count refers to the number of rows
       that exist in the database, not the number that still need the
       migration applied.

    3) In the case where no migrations are successful, but some raise
       exceptions, the command was exiting with status zero, which usually
       indicates "success". This can cause issues that cause migration
       failures to go unnoticed, especially when automated.

    4) When exceptions do occur, a minimally useful message is output, and
       no detail about the exception is available to the user. The exception
       detail should be logged.

    5) Inaccuracies in the documentation - "--max_number" should be
       "--max-count", and stale references to the "--ignore_state" option,
       which was removed in [1]

The solution for (3) introduces a new exit status, 2. See release note
for details.

    These changes are aligned with equivalents [2][3] for the nova-manage
    command, except for the calculation of "Total Needed" - nova seems to
    interpret the "found" count differently/inconsistently.

    [1] https://review.openstack.org/510201
    [2] https://review.openstack.org/605828
    [3] https://review.openstack.org/608091

    Change-Id: I878480eb2359625cde839b073230844acc645cba
    Closes-Bug: #1794364
    Closes-Bug: #1796192

Reviewed:  https://review.openstack.org/611463
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=d47486d317e8cdb1cdab50d73f6484289ab082d4
Submitter: Zuul
Branch:    master

commit d47486d317e8cdb1cdab50d73f6484289ab082d4
Author: imacdonn <iain.macdonnell@oracle.com>
Date:   Wed Oct 17 22:48:18 2018 +0000

cinder-manage online_data_migrations fixes
    
    Addresses some issues with this command:
    
    1) When used without the --max-count option, the summary table will
       always show zero migrations run, because it only accounts for the
       last batch, and the loop only exits when the last batch does no work.
    
    2) "remaining" counts cannot be accurate, given the way migrations are
       implemented, because the "found" count refers to the number of rows
       that exist in the database, not the number that still need the
       migration applied.
    
    3) In the case where no migrations are successful, but some raise
       exceptions, the command was exiting with status zero, which usually
       indicates "success". This can cause issues that cause migration
       failures to go unnoticed, especially when automated.
    
    4) When exceptions do occur, a minimally useful message is output, and
       no detail about the exception is available to the user. The exception
       detail should be logged.
    
    5) Inaccuracies in the documentation - "--max_number" should be
       "--max-count", and stale references to the "--ignore_state" option,
       which was removed in [1]
    
    The solution for (3) introduces a new exit status, 2. See release note
    for details.
    
    These changes are aligned with equivalents [2][3] for the nova-manage
    command, except for the calculation of "Total Needed" - nova seems to
    interpret the "found" count differently/inconsistently.
    
    [1] https://review.openstack.org/510201
    [2] https://review.openstack.org/605828
    [3] https://review.openstack.org/608091
    
    Change-Id: I878480eb2359625cde839b073230844acc645cba
    Closes-Bug: #1794364
    Closes-Bug: #1796192

Changed in cinder:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-31: Fix proposed to cinder (stable/rocky)

#20

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/614617

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-11-26: Fix merged to cinder (stable/rocky)

#21

Reviewed: https://review.openstack.org/614617
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=74fd810ad1a8561ba150925d19feb8cbe598fe84
Submitter: Zuul
Branch: stable/rocky

commit 74fd810ad1a8561ba150925d19feb8cbe598fe84
Author: imacdonn <email address hidden>
Date: Wed Oct 17 22:48:18 2018 +0000

cinder-manage online_data_migrations fixes

Addresses some issues with this command:

    1) When used without the --max-count option, the summary table will
       always show zero migrations run, because it only accounts for the
       last batch, and the loop only exits when the last batch does no work.

    2) "remaining" counts cannot be accurate, given the way migrations are
       implemented, because the "found" count refers to the number of rows
       that exist in the database, not the number that still need the
       migration applied.

    3) In the case where no migrations are successful, but some raise
       exceptions, the command was exiting with status zero, which usually
       indicates "success". This can cause issues that cause migration
       failures to go unnoticed, especially when automated.

    4) When exceptions do occur, a minimally useful message is output, and
       no detail about the exception is available to the user. The exception
       detail should be logged.

    5) Inaccuracies in the documentation - "--max_number" should be
       "--max-count", and stale references to the "--ignore_state" option,
       which was removed in [1]

The solution for (3) introduces a new exit status, 2. See release note
for details.

    These changes are aligned with equivalents [2][3] for the nova-manage
    command, except for the calculation of "Total Needed" - nova seems to
    interpret the "found" count differently/inconsistently.

    [1] https://review.openstack.org/510201
    [2] https://review.openstack.org/605828
    [3] https://review.openstack.org/608091

    Change-Id: I878480eb2359625cde839b073230844acc645cba
    Closes-Bug: #1794364
    Closes-Bug: #1796192
    (cherry picked from commit d47486d317e8cdb1cdab50d73f6484289ab082d4)

Reviewed:  https://review.openstack.org/614617
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=74fd810ad1a8561ba150925d19feb8cbe598fe84
Submitter: Zuul
Branch:    stable/rocky

commit 74fd810ad1a8561ba150925d19feb8cbe598fe84
Author: imacdonn <iain.macdonnell@oracle.com>
Date:   Wed Oct 17 22:48:18 2018 +0000

cinder-manage online_data_migrations fixes
    
    Addresses some issues with this command:
    
    1) When used without the --max-count option, the summary table will
       always show zero migrations run, because it only accounts for the
       last batch, and the loop only exits when the last batch does no work.
    
    2) "remaining" counts cannot be accurate, given the way migrations are
       implemented, because the "found" count refers to the number of rows
       that exist in the database, not the number that still need the
       migration applied.
    
    3) In the case where no migrations are successful, but some raise
       exceptions, the command was exiting with status zero, which usually
       indicates "success". This can cause issues that cause migration
       failures to go unnoticed, especially when automated.
    
    4) When exceptions do occur, a minimally useful message is output, and
       no detail about the exception is available to the user. The exception
       detail should be logged.
    
    5) Inaccuracies in the documentation - "--max_number" should be
       "--max-count", and stale references to the "--ignore_state" option,
       which was removed in [1]
    
    The solution for (3) introduces a new exit status, 2. See release note
    for details.
    
    These changes are aligned with equivalents [2][3] for the nova-manage
    command, except for the calculation of "Total Needed" - nova seems to
    interpret the "found" count differently/inconsistently.
    
    [1] https://review.openstack.org/510201
    [2] https://review.openstack.org/605828
    [3] https://review.openstack.org/608091
    
    Change-Id: I878480eb2359625cde839b073230844acc645cba
    Closes-Bug: #1794364
    Closes-Bug: #1796192
    (cherry picked from commit d47486d317e8cdb1cdab50d73f6484289ab082d4)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-12-03: Fix included in openstack/cinder 13.0.2

#22

This issue was fixed in the openstack/cinder 13.0.2 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-03-21: Fix included in openstack/cinder 14.0.0.0rc1

#23

This issue was fixed in the openstack/cinder 14.0.0.0rc1 release candidate.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-03-22: Fix included in openstack/nova 19.0.0.0rc1

#24

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

OpenStack Compute (nova)

'nova-manage db online_data_migrations' count fail

Bug Description

Other bug subscribers

Remote bug watches

	Status	Importance	Assigned to
Cinder	Fix Released	Undecided	iain MacDonnell
OpenStack Compute (nova)	Fix Released	High	iain MacDonnell
Ocata	Fix Committed	High	Matt Riedemann
Pike	Fix Committed	High	Matt Riedemann
Queens	Fix Committed	High	Matt Riedemann
Rocky	Fix Committed	High	Matt Riedemann