kolla-ansible

Upgrade fails to bootstrap grafana when monasca enabled

Bug #1931037 reported by Scott Shambarger
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Invalid
High
Unassigned
Wallaby
Fix Committed
High
Unassigned
kolla-ansible 12.0.0 "wallaby"

Bug Description

Upgrading from Victoria to Wallaby with Monasca enabled fails waiting for Grafana to start.

Grafana defaults to enabled if Monasca is enabled in Wallaby (was default disabled in Victoria), so the database user and tables don't necessarily exist prior to the upgrade.

Running `kolla-ansible upgrade` then attempts to start the Grafana container, but fails to include the bootstrap task (which creates the db) so the container is unable to start with a database connection error, from the grafana container logs:

  service init failed: failed to check table existence: Error 1045: Access denied for user 'grafana'@'xxx' (using password: YES)

I'm guessing normally bootstrap isn't required on upgrade, but since the enabled state for Grafana is changed during upgrade, it is in this case.

I added `import_tasks: bootstrap.yml` to upgrade.yml, and the upgrade completed without error.

Revision history for this message
Mark Goddard (mgoddard) wrote :

Good catch Scott.

Changed in kolla-ansible:
importance: Undecided → High
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/795072

Radosław Piliszek (yoctozepto)
Changed in kolla-ansible:
status: Triaged → Invalid
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/795072
Committed: https://opendev.org/openstack/kolla-ansible/commit/12dc474830b3a6b1e8687829cca3fd3963c67b41
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 12dc474830b3a6b1e8687829cca3fd3963c67b41
Author: Mark Goddard <email address hidden>
Date: Mon Jun 7 10:34:36 2021 +0100

    grafana: add bootstrap during upgrade

    Upgrading from Victoria to Wallaby with Monasca enabled fails waiting
    for Grafana to start.

    Grafana defaults to enabled if Monasca is enabled in Wallaby (was
    default disabled in Victoria), so the database user and tables don't
    necessarily exist prior to the upgrade.

    Running `kolla-ansible upgrade` then attempts to start the Grafana
    container, but fails to include the bootstrap task (which creates the
    db) so the container is unable to start with a database connection
    error, from the grafana container logs:

      service init failed: failed to check table existence: Error 1045:
      Access denied for user 'grafana'@'xxx' (using password: YES)

    This change performs a bootstrap of Grafana during upgrade. It is
    required only for the Wallaby release where the default change is
    introduced.

    Closes-Bug: #1931037
    Co-Authored-By: Scott Shambarger

    Change-Id: Ib32b78cfd6f3fe535d30d8aa48f3187b9aec746e

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 12.0.0.0rc2

This issue was fixed in the openstack/kolla-ansible 12.0.0.0rc2 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.