[Yoga/Jammy] Octavia fails on hook "ovsdb-subordinate-relation-changed" due to missing CA

Bas, please include in the bug description all of the information indicated in https://docs.openstack.org/charm-guide/yoga/community/software-bug.html

e.g. the essential information:

an overview of your environment

a detailed description of the problem and how it differs from your expected outcome

specific software versions in use

Ubuntu

OpenStack

Juju (see Juju versions)

MAAS

Even better is to include in the title the OpenStack version and the Ubuntu series. For any charms being mentioned the juju channel (and series if different from the main model).

Thanks.

An overview of the versions used can be found in the first link, but for completeness:

======
maas - 3.2.8-
juju - 2.9.44
cpe-foundation - 2.21
infra-ubuntu - focal
ceph - quincy/stable
openstack-charms - yoga/stable
charmed-kubernetes - 1.27
cloud-init - 23.2.1-0ubuntu0~22.04.1
fce-container-image - ubuntu:jammy
openstack - yoga
sku - fcb-master-yoga-jammy
legacy-lma - latest/candidate
solutions-qa-ci - d0ba8ed0
landscape-server - 23.03+16-0landscape0
cos-lite - latest/stable:11
fcbtest - latest/beta
======

Our environment is using a persistent Maas 3.2.8 deployment with a juju 2.9.44 controller bootstrapped and an OpenStack bundle (https://oil-jenkins.canonical.com/artifacts/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/generated/generated/openstack/bundle.yaml) deployed.

The problem is that the Octavia units go into an error state during deployment after Vault is initialized but before Octavia is initialized. The expected behaviour is that the Octavia units do not go into an error state. A more detailed overview of the symptoms can be found in the first comment.

> An overview of the versions used can be found in the first link ...

Just as a minor comment, but if this information isn't included in the description, then every person who reads the bug has to go and find that information (and know that it's in the first link). By including it, and a commentary on what is expected, what is wrong, etc. this really helps in comprehension of the bug, understanding which versions maybe affected and provides really important context. Thank you for the additional information.

We have 5 more occurrences today, I wonder if this is a regression in the Octavia charm revision 158 (released 2 days ago). I am looking at commit [1] in particular as a possible cause, though I would have expected that to be present in other charms as well.

[1] https://github.com/juju/charm-helpers/commit/58097330bd15cfd7b539481c55718c6cee8ca124

@bas; yup, good sleuthing! It's a very subtle error, and I don't think the logic is quite correct. Obviously, there is an error in assuming that the 'ca' key is appearing in the data sent from the certificate provider if it's a legacy request. Unpicking this might be a bit complex due to two different methods of requested certificates on the relation.

I think this may be related to the vault caching issue as the show-unit for octavia/0 is

     vault/0:
        in-scope: true
        data:
          ca: |-
            -----BEGIN CERTIFICATE-----
            MIID7TCCAtWgAwIBAgIUfk0uomNXrMmdlGzOur44HWh/v5gwDQYJKoZIhvcNAQEL
            ...
            lots of data ...
            ...
            /3VHMeHB6G8kcfYZtdq5LeLtrC0eRstDY6uXlLz1afXWAwqxwbtt9FlRNttK2ZNh
            dnueH3hzZlhhITaN2Tsg+GDYb0McdBTPEDUeeaVP2R5LZZl+OnG7OA==
            -----END RSA PRIVATE KEY-----
          private-address: 10.246.168.166
      vault/1:
        in-scope: true
        data:
          egress-subnets: 10.246.168.167/32
          ingress-address: 10.246.168.167
          private-address: 10.246.168.167
      vault/2:
        in-scope: true
        data:
          egress-subnets: 10.246.168.168/32
          ingress-address: 10.246.168.168
          private-address: 10.246.168.168

The code in question (from charm-helpers/charmhelpers/contrib/openstack
/cert_utils.py) is:

def get_requests_for_local_unit(relation_name=None):
    """Extract any certificates data targeted at this unit down relation_name.

    :param relation_name: str Name of relation to check for data.
    :returns: List of bundles of certificates.
    :rtype: List of dicts
    """
    local_name = local_unit().replace('/', '_')
    raw_certs_key = '{}.processed_requests'.format(local_name)
    relation_name = relation_name or 'certificates'
    bundles = []
    for rid in relation_ids(relation_name):
        sent = relation_get(rid=rid, unit=local_unit())
        legacy_keys = ['certificate_name', 'common_name']
        is_legacy_request = set(sent).intersection(legacy_keys)
        for unit in related_units(rid):
            data = relation_get(rid=rid, unit=unit)
            if data.get(raw_certs_key):
                bundles.append({
                    'ca': data['ca'],
                    'chain': data.get('chain'),
                    'certs': json.loads(data[raw_certs_key])})
            elif is_legacy_request:
                bundles.append({
                    'ca': data['ca'],
                    'chain': data.get('chain'),
                    'certs': {sent['common_name']:
                              {'cert': data.get(local_name + '.server.cert'),
                               'key': data.get(local_name + '.server.key')}}})

But the main issue is, I believe, a timing. Octavia requests a certificate using the old form, but it has not yet been processed. i.e. raw_certs_key == {localname}processed_requests is not present in the data bag from vault, but the two keys 'certificate_name' and 'common_name' are present in the outbound relation from octavia. This results in the "elif is_legacy_request" being true, and thus the erroneous access to data['ca'] which isn't available in the data bag.

And essentially, it's a timing issue. Vault hasn't been initialised, so it's not sending any certs, but due to how hooks are handled in reactive charms, it's still running the handler render which goes off to look at all the relations for the context.

Fixing this may be as simple as just checking that 'ca' is available, and if not, bailing o...

charm-helpers fix: https://github.com/juju/charm-helpers/pull/824

Will need backporting once it is merged.

Related fix that probably caused this issue: https://bugs.launchpad.net/charm-ironic-api/+bug/2022980

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/890556

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-octavia/+/890565

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/890556
Committed: https://opendev.org/openstack/charm-ironic-api/commit/113ccde9d445bedec482efdec12841aa32ecc91a
Submitter: "Zuul (22348)"
Branch: master

commit 113ccde9d445bedec482efdec12841aa32ecc91a
Author: Alex Kavanagh <email address hidden>
Date: Fri Aug 4 17:16:42 2023 +0100

    Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/824
    Closes-Bug: #2028683

    Change-Id: I3d454dbddda8006178797d381cd07856faca897d

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/890565
Committed: https://opendev.org/openstack/charm-octavia/commit/fc85965b3bb9601a4b273b07c6d55bcbd5917620
Submitter: "Zuul (22348)"
Branch: master

commit fc85965b3bb9601a4b273b07c6d55bcbd5917620
Author: Alex Kavanagh <email address hidden>
Date: Fri Aug 4 17:16:42 2023 +0100

    Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/824
    Closes-Bug: #2028683

    Change-Id: Id3636d55f26d05ffed994e3ccc0575ec66a5e389

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891487

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891496

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891537

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891546

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891707

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891716

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891851

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891860

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891890

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891899

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891958

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891967

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891996

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/charm-octavia/+/892005

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/892022

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-octavia/+/892031

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/891546
Committed: https://opendev.org/openstack/charm-octavia/commit/ee1259750892536e1206ce182836c5983b861836
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit ee1259750892536e1206ce182836c5983b861836
Author: Alex Kavanagh <email address hidden>
Date: Tue Aug 15 22:22:12 2023 +0100

    [2023.1] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/825
    Closes-Bug: #2028683

    Change-Id: I0e7e2e67109f39a507b69de9d4f1e6abccf32f58

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/891537
Committed: https://opendev.org/openstack/charm-ironic-api/commit/2a4af9398493be9d11d8dbe0751e51aa0fb39556
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 2a4af9398493be9d11d8dbe0751e51aa0fb39556
Author: Alex Kavanagh <email address hidden>
Date: Tue Aug 15 22:22:12 2023 +0100

    [2023.1] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/825
    Closes-Bug: #2028683

    Change-Id: Ie91b970c7d8233bcd27e3ebaa5804e23ab40e443

I'm also encountering this exact same issue today, this fix has not yet been released for Yoga/Stable unfortunately.

On Thu, 2023-08-31 at 18:25 +0000, Alan Baghumian wrote:
> I'm also encountering this exact same issue today, this fix has not yet
> been released for Yoga/Stable unfortunately.

that's correct and in sync with the tasks this bug has, you can monitor this patch too -
https://review.opendev.org/c/openstack/charm-octavia/+/891496

Change abandoned by "Alex Kavanagh <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891967
Reason: Abandoning in favour of https://review.opendev.org/c/openstack/charm-octavia/+/892336 which is also a rebuild request, and thus essentially duplicates the rebuild/push.

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/892022
Committed: https://opendev.org/openstack/charm-ironic-api/commit/f376e92af7a3ca9b537a1635a1cc740d34cf8c05
Submitter: "Zuul (22348)"
Branch: stable/zed

commit f376e92af7a3ca9b537a1635a1cc740d34cf8c05
Author: Alex Kavanagh <email address hidden>
Date: Fri Aug 18 17:39:31 2023 +0100

    [zed] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/826
    Closes-Bug: #2028683

    Change-Id: Ic9c479d1076b22419706dfa8fb7940033d13d46e

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/891487
Committed: https://opendev.org/openstack/charm-ironic-api/commit/dcdf7a7e856a2d8bc0aaeaccc22d3ad60e326655
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit dcdf7a7e856a2d8bc0aaeaccc22d3ad60e326655
Author: Alex Kavanagh <email address hidden>
Date: Tue Aug 15 15:13:22 2023 +0100

    [yoga] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/827
    Closes-Bug: #2028683

    Change-Id: I214d62984423c50e4c32b051e64ab5b2e851b9a8

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/891707
Committed: https://opendev.org/openstack/charm-ironic-api/commit/8389fc923e01931b72dbddbe661f4c76e78e76b1
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 8389fc923e01931b72dbddbe661f4c76e78e76b1
Author: Alex Kavanagh <email address hidden>
Date: Thu Aug 17 13:55:52 2023 +0100

    [xena] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/828
    Closes-Bug: #2028683

    Change-Id: If06ad8d782a47edaa320da8d710211f3ec04e28c

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/891851
Committed: https://opendev.org/openstack/charm-ironic-api/commit/7330d5ae7f78b3bdb94c5b72480c06958c534798
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 7330d5ae7f78b3bdb94c5b72480c06958c534798
Author: Alex Kavanagh <email address hidden>
Date: Thu Aug 17 16:28:50 2023 +0100

    [wallaby] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/829
    Closes-Bug: #2028683

    Change-Id: I3c7ca65fca93a43d659b3c4cd48ae2ef7dc35c79

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/892031
Committed: https://opendev.org/openstack/charm-octavia/commit/dfd3f35a5641814401dea7df68e17e66c0b3bf5f
Submitter: "Zuul (22348)"
Branch: stable/zed

commit dfd3f35a5641814401dea7df68e17e66c0b3bf5f
Author: Alex Kavanagh <email address hidden>
Date: Fri Aug 18 17:39:31 2023 +0100

    [zed] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/826
    Closes-Bug: #2028683

    Change-Id: I51b9784be407f724515aed655f4325be40057871

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/891496
Committed: https://opendev.org/openstack/charm-octavia/commit/57561cff59b6155709a791237ad885b01bc2fb20
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 57561cff59b6155709a791237ad885b01bc2fb20
Author: Alex Kavanagh <email address hidden>
Date: Tue Aug 15 15:13:22 2023 +0100

    [yoga] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/827
    Closes-Bug: #2028683

    Change-Id: I41ea5a58e0391d398fab2ecd44999e6f32a4054b

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/891716
Committed: https://opendev.org/openstack/charm-octavia/commit/aab20025c9219a76ceebcccbd04a481aeaeff54e
Submitter: "Zuul (22348)"
Branch: stable/xena

commit aab20025c9219a76ceebcccbd04a481aeaeff54e
Author: Alex Kavanagh <email address hidden>
Date: Thu Aug 17 13:55:52 2023 +0100

    [xena] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/828
    Closes-Bug: #2028683

    Change-Id: I1f7368e2851402d67f71f69590d1be6c8ddd25e3

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/891860
Committed: https://opendev.org/openstack/charm-octavia/commit/5f8111f0eb8aac8985da3e98fc1f91dade3d1db8
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 5f8111f0eb8aac8985da3e98fc1f91dade3d1db8
Author: Alex Kavanagh <email address hidden>
Date: Thu Aug 17 16:28:50 2023 +0100

    [wallaby] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/829
    Closes-Bug: #2028683

    Change-Id: Id883860c21bff001e693760a0535fd427e9c6d18

Finally got tired of waiting and manually copied over the charmhelpers/contrib/openstack/cert_utils.py file to my Octavia units and performed a juju resolve. They went past the error and all green now!

Hopefully the updated charm will be available since this is not a proper way of fixing things :-D

$ sudo cp -v cert_utils.py /var/lib/juju/agents/unit-octavia-<unit#>/.venv/lib/python3.8/site-packages/charmhelpers/contrib/openstack/cert_utils.py

Model Controller Cloud/Region Version SLA Timestamp
vstack home-lab-default Home-Lab/default 2.9.37 unsupported 17:56:44-07:00

App Version Status Scale Charm Channel Rev Exposed Message
lds-client-focal active 3 landscape-client stable 49 no System successfully registered
octavia 10.0.0 active 3 octavia yoga/stable 193 no Unit is ready
octavia-hacluster active 3 hacluster 2.0.3/stable 113 no Unit is ready and clustered
octavia-mysql-router 8.0.34 active 3 mysql-router 8.0/stable 90 no Unit is ready
octavia-ovn-chassis 22.03.2 active 3 ovn-chassis 22.03/stable 143 no Unit is ready

Unit Workload Agent Machine Public address Ports Message
octavia/17* active idle 9/lxd/12 10.1.15.145 9876/tcp Unit is ready
  lds-client-focal/691 active idle 10.1.15.145 System successfully registered
  octavia-hacluster/99 active idle 10.1.15.145 Unit is ready and clustered
  octavia-mysql-router/122* active idle 10.1.15.145 Unit is ready
  octavia-ovn-chassis/119* active idle 10.1.15.145 Unit is ready
octavia/18 active idle 102/lxd/5 10.1.15.146 9876/tcp Unit is ready
  lds-client-focal/690 active idle 10.1.15.146 System successfully registered
  octavia-hacluster/98 active idle 10.1.15.146 Unit is ready and clustered
  octavia-mysql-router/123 active idle 10.1.15.146 Unit is ready
  octavia-ovn-chassis/118 active idle 10.1.15.146 Unit is ready
octavia/19 active idle 104/lxd/4 10.1.15.147 9876/tcp Unit is ready
  lds-client-focal/689 active idle 10.1.15.147 System successfully registered
  octavia-hacluster/97* active idle 10.1.15.147 Unit is ready and clustered
  octavia-mysql-router/121 active idle 10.1.15.147 Unit is ready
  octavia-ovn-chassis/117 active idle 10.1.15.147 Unit is ready

Machine State Address Inst id Series AZ Message
9 started 10.1.8.12 os-vm-2 focal default Deployed
9/lxd/12 started 10.1.15.145 juju-b096f0-9-lxd-12 focal default Container started
102 started 10.1.8.25 ...

Hi @alanback

I'm not sure which channel of the octavia-charm you are using, but the fix has landed to all of the following channels: ussuri, wallaby, xena, yoga, zed, 2023.1 (antelope) and master. The victoria and (bionic - queens, rocky, stein, train) are currently not building.

There's a tooling issue, as the patches have landed but the bug was not updated. I've manually updated the ones that have landed now.

Sorry for the incorrect information.

Thanks!

Hi Alex,

I am on Yoga/Stable (Focal) and upgraded the charm to revision 193 when it cam out a couple of weeks ago:

channels: |
  yoga/stable: 20bbe01 2023-09-08 (193) 41MB

$ juju status octavia
Model Controller Cloud/Region Version SLA Timestamp
vstack home-lab-default Home-Lab/default 2.9.37 unsupported 09:22:04-07:00

App Version Status Scale Charm Channel Rev Exposed Message
lds-client-focal active 3 landscape-client stable 49 no System successfully registered
octavia 10.0.0 active 3 octavia yoga/stable 193 no Unit is ready
octavia-hacluster active 3 hacluster 2.0.3/stable 113 no Unit is ready and clustered
octavia-mysql-router 8.0.34 active 3 mysql-router 8.0/stable 90 no Unit is ready
octavia-ovn-chassis 22.03.2 active 3 ovn-chassis 22.03/stable 143 no Unit is ready

However that did not fix this issue until I manually copied the file yesterday.

So, if 193 is indeed the latest for Yoga/Stable, it did not fix the problem.

Hope this helps!

Best,
Alan

On Sun, 2023-09-24 at 16:24 +0000, Alan Baghumian wrote:
> Hi Alex,
>
> I am on Yoga/Stable (Focal) and upgraded the charm to revision 193 when
> it cam out a couple of weeks ago:[...]
when you upgraded, were the hooks in error state?, if they were, you need to use the "--force-units"
flag to make juju upgrade the charm, otherwise it will wait for the hook failures get resolved.

You can check the output of "juju show-status-log <entity name>" to know when/if the upgrade-charm
hook was executed.

Hi; you made me check! :)

I downloaded the charm manually for octavia (juju download octavia --channel=yoga/stable) and checked the charmhelpers/contrib/cert_utils.py at line 417 and it does contain the fix.

Felipe raises a point on whether the units were actually upgraded rather than still waiting on failed hooks. Another method would be to add a unit (if possible) and see if it works okay.

Hi @Felipe! You nailed it:

$ juju show-status-log octavia/19 --days 30 | grep -c upgrade-charm
0

The hook never actually ran even after running juju resolve on units.

Hi @Alex! Sorry I made you double check this and thanks much for checking.

I am now running the upgrade hook.

Best,
Alan

We are good! Just refreshed to xena/stable and then back to yoga/stable. All good.

$ juju refresh octavia --channel xena/stable

$ juju refresh octavia --channel yoga/stable

Best,
Alan

On Tue, 2023-09-26 at 16:51 +0000, Alan Baghumian wrote:
> We are good! Just refreshed to xena/stable and then back to yoga/stable.
> All good.

that's great.