kolla-ansible

Upgrade to Train fails while running Cinder online schema migration

  1. Series ussuri
  2. Bug #1880753
Bug #1880753 reported by Pierre Riteau
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Fix Released
High
Pierre Riteau
kolla-ansible 11.0.0 "victoria"
Train
Fix Committed
High
Mark Goddard
Ussuri
Fix Committed
High
Mark Goddard
Victoria
Fix Released
High
Pierre Riteau
kolla-ansible 11.0.0 "victoria"

Bug Description

During an upgrade from Stein to Train, Kolla Ansible fails while running TASK [cinder : Running Cinder online schema migration]

This is because the `--max_count 10` option is used, which returns 1 while migrations are processed. According to the upgrade documentation, the command should be rerun while the exit status is 1: https://docs.openstack.org/cinder/train/upgrade.html

  msg: Container exited with non-zero return code 1
  rc: 1
  stderr: |-
    + sudo -E kolla_set_configs
    INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
    INFO:__main__:Validating config file
    INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
    INFO:__main__:Copying service configuration files
    INFO:__main__:Deleting /etc/cinder/cinder.conf
    INFO:__main__:Copying /var/lib/kolla/config_files/cinder.conf to /etc/cinder/cinder.conf
    INFO:__main__:Setting permission for /etc/cinder/cinder.conf
    INFO:__main__:Copying /var/lib/kolla/config_files/cinder-wsgi.conf to /etc/httpd/conf.d/cinder-wsgi.conf
    INFO:__main__:Setting permission for /etc/httpd/conf.d/cinder-wsgi.conf
    INFO:__main__:Writing out command to execute
    INFO:__main__:Setting permission for /var/lib/cinder
    INFO:__main__:Setting permission for /var/lib/cinder/tmp
    INFO:__main__:Setting permission for /var/log/kolla/cinder
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-api.log
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-wsgi.log
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-api-access.log.2.gz
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-wsgi.log.5.gz
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-volume.log
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-wsgi.log.2.gz
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-manage.log
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-wsgi.log.6.gz
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-wsgi.log.3.gz
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-api-access.log.1
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-wsgi.log.1
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-api-access.log.6.gz
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-api-access.log.3.gz
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-wsgi.log.4.gz
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-api-access.log
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-scheduler.log
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-api-access.log.4.gz
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-api-access.log.5.gz
    ++ cat /run_command
    + CMD='httpd -DFOREGROUND'
    + ARGS=
    + sudo kolla_copy_cacerts
    + [[ ! -n '' ]]
    + . kolla_extend_start
    ++ [[ ! -d /var/log/kolla/cinder ]]
    +++ stat -c %a /var/log/kolla/cinder
    ++ [[ 755 != \7\5\5 ]]
    ++ . /usr/local/bin/kolla_cinder_extend_start
    +++ set -o errexit
    +++ [[ -n '' ]]
    +++ [[ -n 0 ]]
    +++ [[ -n 0 ]]
    +++ cinder-manage db online_data_migrations --max_count 10
    Deprecated: Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
  stderr_lines: <omitted>
  stdout: |-
    10 rows matched query untyped_volumes_online_data_migration, 10 migrated
    +-----------------------------------------+--------------+-----------+
    | Migration | Total Needed | Completed |
    +-----------------------------------------+--------------+-----------+
    | untyped_volumes_online_data_migration | 10 | 10 |
    +-----------------------------------------+--------------+-----------+
  stdout_lines: <omitted>

**Environment**:

* CentOS 7.8
* Kolla-Ansible version: stable/train
* Docker image Install type (source/binary): binary
* Docker image distribution: centos
* Official images from Docker Hub

Revision history for this message
Pierre Riteau (priteau) wrote :

Workaround:

- set cinder_max_number_osm to a high number
- run upgrade: it will fail again, but will have processed many more migrations
- run upgrade again, which should work since cinder-manage will return 0

Mark Goddard (mgoddard)
Changed in kolla-ansible:
importance: Undecided → High
Revision history for this message
Mark Goddard (mgoddard) wrote :

I suspect this was inadvertently caused by https://review.opendev.org/#/c/712055/. While the logic was incorrect previously, it caused us to ignore the max count. Probably the right solution is to stop passing a max count through from kolla-ansible. Alternatively we could add a loop in kolla, but what would we do between iterations? sleep?

Revision history for this message
Pierre Riteau (priteau) wrote :

It's not clear in what circumstances it is important to use a custom max_count. Note that if max_count is not passed to cinder-manage, it will run batches of 50 migrations until complete (i.e. only one invocation should be needed).

If someone wants to implement the loop logic, I would suggest defaulting to no sleep but allowing to customize it via a variable. But if no one really needs to use that feature, let's stop passing max_count.

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

I opt for not using this variable there.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/735617

Changed in kolla-ansible:
assignee: nobody → Mark Goddard (mgoddard)
status: Triaged → In Progress
OpenStack Infra (hudson-openstack)
Changed in kolla-ansible:
assignee: Mark Goddard (mgoddard) → Pierre Riteau (priteau)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/735617
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=55c0787d00d678b2fbfc9e99a6e4b4ce5c9a82ca
Submitter: Zuul
Branch: master

commit 55c0787d00d678b2fbfc9e99a6e4b4ce5c9a82ca
Author: Mark Goddard <email address hidden>
Date: Mon Jun 15 16:19:49 2020 +0100

    Remove max count from Cinder online schema migration

    During an upgrade from Stein to Train, Kolla Ansible fails while running
    TASK [cinder : Running Cinder online schema migration]

    This is because the `--max_count 10` option is used, which returns 1
    while migrations are processed. According to the upgrade documentation,
    the command should be rerun while the exit status is 1:
    https://docs.openstack.org/cinder/train/upgrade.html

    This issue was introduced by a change to the image [1] which fixed a bug
    in the way that the max count was interpreted, but exposed an issue in
    using the max count.

    This change fixes the issue by ceasing to pass MAX_NUMBER, which will
    cause all migrations to occur in a single pass.

    [1] https://review.opendev.org/#/c/712055

    Change-Id: Ia786d037f5484f18294188639c956d4ed5ffbc2a
    Closes-Bug: #1880753

Changed in kolla-ansible:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/736316

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/736317

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/736318

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/ussuri)

Reviewed: https://review.opendev.org/736316
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=851180d26661e2cc2a5dc5ec06f22bb1f11efb87
Submitter: Zuul
Branch: stable/ussuri

commit 851180d26661e2cc2a5dc5ec06f22bb1f11efb87
Author: Mark Goddard <email address hidden>
Date: Mon Jun 15 16:19:49 2020 +0100

    Remove max count from Cinder online schema migration

    During an upgrade from Stein to Train, Kolla Ansible fails while running
    TASK [cinder : Running Cinder online schema migration]

    This is because the `--max_count 10` option is used, which returns 1
    while migrations are processed. According to the upgrade documentation,
    the command should be rerun while the exit status is 1:
    https://docs.openstack.org/cinder/train/upgrade.html

    This issue was introduced by a change to the image [1] which fixed a bug
    in the way that the max count was interpreted, but exposed an issue in
    using the max count.

    This change fixes the issue by ceasing to pass MAX_NUMBER, which will
    cause all migrations to occur in a single pass.

    [1] https://review.opendev.org/#/c/712055

    Change-Id: Ia786d037f5484f18294188639c956d4ed5ffbc2a
    Closes-Bug: #1880753
    (cherry picked from commit 55c0787d00d678b2fbfc9e99a6e4b4ce5c9a82ca)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/train)

Reviewed: https://review.opendev.org/736317
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=6f227c2cdf4590790616210fa94340594822c4d4
Submitter: Zuul
Branch: stable/train

commit 6f227c2cdf4590790616210fa94340594822c4d4
Author: Mark Goddard <email address hidden>
Date: Mon Jun 15 16:19:49 2020 +0100

    Remove max count from Cinder online schema migration

    During an upgrade from Stein to Train, Kolla Ansible fails while running
    TASK [cinder : Running Cinder online schema migration]

    This is because the `--max_count 10` option is used, which returns 1
    while migrations are processed. According to the upgrade documentation,
    the command should be rerun while the exit status is 1:
    https://docs.openstack.org/cinder/train/upgrade.html

    This issue was introduced by a change to the image [1] which fixed a bug
    in the way that the max count was interpreted, but exposed an issue in
    using the max count.

    This change fixes the issue by ceasing to pass MAX_NUMBER, which will
    cause all migrations to occur in a single pass.

    [1] https://review.opendev.org/#/c/712055

    Change-Id: Ia786d037f5484f18294188639c956d4ed5ffbc2a
    Closes-Bug: #1880753
    (cherry picked from commit 55c0787d00d678b2fbfc9e99a6e4b4ce5c9a82ca)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/stein)

Reviewed: https://review.opendev.org/736318
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=041943fa1c4b25ace589728abb909fe6624acd8e
Submitter: Zuul
Branch: stable/stein

commit 041943fa1c4b25ace589728abb909fe6624acd8e
Author: Mark Goddard <email address hidden>
Date: Mon Jun 15 16:19:49 2020 +0100

    Remove max count from Cinder online schema migration

    During an upgrade from Stein to Train, Kolla Ansible fails while running
    TASK [cinder : Running Cinder online schema migration]

    This is because the `--max_count 10` option is used, which returns 1
    while migrations are processed. According to the upgrade documentation,
    the command should be rerun while the exit status is 1:
    https://docs.openstack.org/cinder/train/upgrade.html

    This issue was introduced by a change to the image [1] which fixed a bug
    in the way that the max count was interpreted, but exposed an issue in
    using the max count.

    This change fixes the issue by ceasing to pass MAX_NUMBER, which will
    cause all migrations to occur in a single pass.

    [1] https://review.opendev.org/#/c/712055

    Change-Id: Ia786d037f5484f18294188639c956d4ed5ffbc2a
    Closes-Bug: #1880753
    (cherry picked from commit 55c0787d00d678b2fbfc9e99a6e4b4ce5c9a82ca)

tags: added: in-stable-stein
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.