[Yoga/Jammy] Octavia fails on hook "ovsdb-subordinate-relation-changed" due to missing CA

Bug #2028683 reported by Bas de Bruijne
34
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Charm Helpers
Fix Released
Critical
Alex Kavanagh
OpenStack Ironic API Charm
Status tracked in Trunk
2023.1
Fix Committed
Undecided
Unassigned
Trunk
Fix Committed
Undecided
Unassigned
Ussuri
In Progress
Undecided
Unassigned
Victoria
In Progress
Undecided
Unassigned
Wallaby
Fix Committed
Undecided
Unassigned
Xena
Fix Committed
Undecided
Unassigned
Yoga
Fix Committed
Undecided
Unassigned
Zed
Fix Committed
Undecided
Unassigned
OpenStack Octavia Charm
Status tracked in Trunk
2023.1
Fix Released
Undecided
Unassigned
Trunk
Fix Committed
Critical
Alex Kavanagh
Ussuri
Fix Released
Undecided
Unassigned
Victoria
In Progress
Undecided
Unassigned
Wallaby
Fix Released
Undecided
Unassigned
Xena
Fix Released
Undecided
Unassigned
Yoga
Fix Released
Undecided
Unassigned
Zed
Fix Released
Undecided
Unassigned

Bug Description

[edit]: from comment #2:

======
maas - 3.2.8-
juju - 2.9.44
cpe-foundation - 2.21
infra-ubuntu - focal
ceph - quincy/stable
openstack-charms - yoga/stable
charmed-kubernetes - 1.27
cloud-init - 23.2.1-0ubuntu0~22.04.1
fce-container-image - ubuntu:jammy
openstack - yoga
sku - fcb-master-yoga-jammy
legacy-lma - latest/candidate
solutions-qa-ci - d0ba8ed0
landscape-server - 23.03+16-0landscape0
cos-lite - latest/stable:11
fcbtest - latest/beta
======

---

The problem is that the Octavia units go into an error state during deployment after Vault is initialized but before Octavia is initialized. The expected behaviour is that the Octavia units do not go into an error state. A more detailed overview of the symptoms can be found in the first comment.

In test run https://solutions.qa.canonical.com/testruns/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/, the octavia units get into an error state:

================
octavia/0 error idle 3/lxd/9 10.246.164.228 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"
  filebeat/77 active idle 10.246.164.228 Filebeat ready.
  hacluster-octavia/1 active idle 10.246.164.228 Unit is ready and clustered
  landscape-client/77 maintenance idle 10.246.164.228 Need computer-title and juju-info to proceed
  logrotated/77 active idle 10.246.164.228 Unit is ready.
  nrpe/85 active idle 10.246.164.228 icmp,5666/tcp Ready
  octavia-mysql-router/2 active idle 10.246.164.228 Unit is ready
  octavia-ovn-chassis/1 active idle 10.246.164.228 Unit is ready
  prometheus-grok-exporter/77 active idle 10.246.164.228 9144/tcp Unit is ready
  public-policy-routing/48 active idle 10.246.164.228 Unit is ready
  telegraf/77 active idle 10.246.164.228 9103/tcp Monitoring octavia/0 (source version/commit 23.01-8-...)
  ubuntu-advantage/77 active idle 10.246.164.228 Attached (esm-apps,esm-infra)
octavia/1 error idle 4/lxd/9 10.246.167.116 9876/tcp hook failed: "ovsdb-cms-relation-changed"
  filebeat/78 active idle 10.246.167.116 Filebeat ready.
  hacluster-octavia/2 active idle 10.246.167.116 Unit is ready and clustered
  landscape-client/78 maintenance idle 10.246.167.116 Need computer-title and juju-info to proceed
  logrotated/78 active idle 10.246.167.116 Unit is ready.
  nrpe/86 active idle 10.246.167.116 icmp,5666/tcp Ready
  octavia-mysql-router/1 active idle 10.246.167.116 Unit is ready
  octavia-ovn-chassis/2 active idle 10.246.167.116 Unit is ready
  prometheus-grok-exporter/78 active idle 10.246.167.116 9144/tcp Unit is ready
  public-policy-routing/49 active idle 10.246.167.116 Unit is ready
  telegraf/78 active idle 10.246.167.116 9103/tcp Monitoring octavia/1 (source version/commit 23.01-8-...)
  ubuntu-advantage/78 active idle 10.246.167.116 Attached (esm-apps,esm-infra)
octavia/2* error idle 5/lxd/9 10.246.167.69 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"
  filebeat/75 active idle 10.246.167.69 Filebeat ready.
  hacluster-octavia/0* active idle 10.246.167.69 Unit is ready and clustered
  landscape-client/75 maintenance idle 10.246.167.69 Need computer-title and juju-info to proceed
  logrotated/75 active idle 10.246.167.69 Unit is ready.
  nrpe/83 active idle 10.246.167.69 icmp,5666/tcp Ready
  octavia-mysql-router/0* active idle 10.246.167.69 Unit is ready
  octavia-ovn-chassis/0* active idle 10.246.167.69 Unit is ready
  prometheus-grok-exporter/75 active idle 10.246.167.69 9144/tcp Unit is ready
  public-policy-routing/46 active idle 10.246.167.69 Unit is ready
  telegraf/75 active idle 10.246.167.69 9103/tcp Monitoring octavia/2 (source version/commit 23.01-8-...)
  ubuntu-advantage/75 active idle 10.246.167.69 Attached (esm-apps,esm-infra)
================

In the debug log, we see the following message:
================
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 808, in render_configs
    _render(os.path.basename(conf))
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render
    charmhelpers.core.templating.render(
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 80, in render
    content = template.render(context)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1291, in render
    self.environment.handle_exception()
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 926, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "templates/openstack_https_frontend.conf", line 1, in top-level template code
    {% if options.endpoints -%}
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 475, in getattr
    return getattr(obj, attribute)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/adapters.py", line 1025, in endpoints
    int_port = ch_cluster.determine_api_port(
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 265, in determine_api_port
    if https():
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 228, in https
    cert_utils.get_requests_for_local_unit("certificates")
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/openstack/cert_utils.py", line 424, in get_requests_for_local_unit
    'ca': data['ca'],
KeyError: 'ca'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-octavia-0/charm/reactive/octavia_handlers.py", line 272, in render
    octavia_charm.render_with_interfaces(
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 827, in render_with_interfaces
    self.render_configs(
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 814, in render_configs
    _render('_'.join(conf.split(os.path.sep))[1:])
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render
    charmhelpers.core.templating.render(
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 79, in render
    raise e
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 74, in render
    template = template_env.get_template(source)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1000, in get_template
    return self._load_template(name, globals)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 959, in _load_template
    template = self.loader.load(self, name, self.make_globals(globals))
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/loaders.py", line 575, in load
    raise TemplateNotFound(name)
jinja2.exceptions.TemplateNotFound: etc_apache2_sites-available_openstack_https_frontend.conf
================

It looks like the CA isn't found, but then a template is also missing from the charm. I'm not sure what the template would be for.

Crashdumps and configs can be found here:
https://oil-jenkins.canonical.com/artifacts/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/index.html

tags: added: cdo-qa foundations-engine
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Bas, please include in the bug description all of the information indicated in https://docs.openstack.org/charm-guide/yoga/community/software-bug.html

e.g. the essential information:

an overview of your environment

a detailed description of the problem and how it differs from your expected outcome

specific software versions in use

Ubuntu

OpenStack

Juju (see Juju versions)

MAAS

Even better is to include in the title the OpenStack version and the Ubuntu series. For any charms being mentioned the juju channel (and series if different from the main model).

Thanks.

Changed in charm-octavia:
status: New → Incomplete
Revision history for this message
Bas de Bruijne (basdbruijne) wrote :

An overview of the versions used can be found in the first link, but for completeness:

======
maas - 3.2.8-
juju - 2.9.44
cpe-foundation - 2.21
infra-ubuntu - focal
ceph - quincy/stable
openstack-charms - yoga/stable
charmed-kubernetes - 1.27
cloud-init - 23.2.1-0ubuntu0~22.04.1
fce-container-image - ubuntu:jammy
openstack - yoga
sku - fcb-master-yoga-jammy
legacy-lma - latest/candidate
solutions-qa-ci - d0ba8ed0
landscape-server - 23.03+16-0landscape0
cos-lite - latest/stable:11
fcbtest - latest/beta
======

Our environment is using a persistent Maas 3.2.8 deployment with a juju 2.9.44 controller bootstrapped and an OpenStack bundle (https://oil-jenkins.canonical.com/artifacts/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/generated/generated/openstack/bundle.yaml) deployed.

The problem is that the Octavia units go into an error state during deployment after Vault is initialized but before Octavia is initialized. The expected behaviour is that the Octavia units do not go into an error state. A more detailed overview of the symptoms can be found in the first comment.

summary: - Octavia fails on hook "ovsdb-subordinate-relation-changed" due to
- missing CA
+ [Yoga/Jammy] Octavia fails on hook "ovsdb-subordinate-relation-changed"
+ due to missing CA
Changed in charm-octavia:
status: Incomplete → New
description: updated
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

> An overview of the versions used can be found in the first link ...

Just as a minor comment, but if this information isn't included in the description, then every person who reads the bug has to go and find that information (and know that it's in the first link). By including it, and a commentary on what is expected, what is wrong, etc. this really helps in comprehension of the bug, understanding which versions maybe affected and provides really important context. Thank you for the additional information.

description: updated
Revision history for this message
Bas de Bruijne (basdbruijne) wrote :

We have 5 more occurrences today, I wonder if this is a regression in the Octavia charm revision 158 (released 2 days ago). I am looking at commit [1] in particular as a possible cause, though I would have expected that to be present in other charms as well.

[1] https://github.com/juju/charm-helpers/commit/58097330bd15cfd7b539481c55718c6cee8ca124

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

@bas; yup, good sleuthing! It's a very subtle error, and I don't think the logic is quite correct. Obviously, there is an error in assuming that the 'ca' key is appearing in the data sent from the certificate provider if it's a legacy request. Unpicking this might be a bit complex due to two different methods of requested certificates on the relation.

Changed in charm-octavia:
status: New → Triaged
importance: Undecided → Critical
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :
Download full text (3.2 KiB)

I think this may be related to the vault caching issue as the show-unit for octavia/0 is

     vault/0:
        in-scope: true
        data:
          ca: |-
            -----BEGIN CERTIFICATE-----
            MIID7TCCAtWgAwIBAgIUfk0uomNXrMmdlGzOur44HWh/v5gwDQYJKoZIhvcNAQEL
            ...
            lots of data ...
            ...
            /3VHMeHB6G8kcfYZtdq5LeLtrC0eRstDY6uXlLz1afXWAwqxwbtt9FlRNttK2ZNh
            dnueH3hzZlhhITaN2Tsg+GDYb0McdBTPEDUeeaVP2R5LZZl+OnG7OA==
            -----END RSA PRIVATE KEY-----
          private-address: 10.246.168.166
      vault/1:
        in-scope: true
        data:
          egress-subnets: 10.246.168.167/32
          ingress-address: 10.246.168.167
          private-address: 10.246.168.167
      vault/2:
        in-scope: true
        data:
          egress-subnets: 10.246.168.168/32
          ingress-address: 10.246.168.168
          private-address: 10.246.168.168

The code in question (from charm-helpers/charmhelpers/contrib/openstack
/cert_utils.py) is:

def get_requests_for_local_unit(relation_name=None):
    """Extract any certificates data targeted at this unit down relation_name.

    :param relation_name: str Name of relation to check for data.
    :returns: List of bundles of certificates.
    :rtype: List of dicts
    """
    local_name = local_unit().replace('/', '_')
    raw_certs_key = '{}.processed_requests'.format(local_name)
    relation_name = relation_name or 'certificates'
    bundles = []
    for rid in relation_ids(relation_name):
        sent = relation_get(rid=rid, unit=local_unit())
        legacy_keys = ['certificate_name', 'common_name']
        is_legacy_request = set(sent).intersection(legacy_keys)
        for unit in related_units(rid):
            data = relation_get(rid=rid, unit=unit)
            if data.get(raw_certs_key):
                bundles.append({
                    'ca': data['ca'],
                    'chain': data.get('chain'),
                    'certs': json.loads(data[raw_certs_key])})
            elif is_legacy_request:
                bundles.append({
                    'ca': data['ca'],
                    'chain': data.get('chain'),
                    'certs': {sent['common_name']:
                              {'cert': data.get(local_name + '.server.cert'),
                               'key': data.get(local_name + '.server.key')}}})

But the main issue is, I believe, a timing. Octavia requests a certificate using the old form, but it has not yet been processed. i.e. raw_certs_key == {localname}processed_requests is not present in the data bag from vault, but the two keys 'certificate_name' and 'common_name' are present in the outbound relation from octavia. This results in the "elif is_legacy_request" being true, and thus the erroneous access to data['ca'] which isn't available in the data bag.

And essentially, it's a timing issue. Vault hasn't been initialised, so it's not sending any certs, but due to how hooks are handled in reactive charms, it's still running the handler render which goes off to look at all the relations for the context.

Fixing this may be as simple as just checking that 'ca' is available, and if not, bailing o...

Read more...

Changed in charm-octavia:
assignee: nobody → Alex Kavanagh (ajkavanagh)
Changed in charm-octavia:
status: Triaged → In Progress
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

charm-helpers fix: https://github.com/juju/charm-helpers/pull/824

Will need backporting once it is merged.

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Related fix that probably caused this issue: https://bugs.launchpad.net/charm-ironic-api/+bug/2022980

Changed in charm-helpers:
status: New → In Progress
importance: Undecided → Critical
assignee: nobody → Alex Kavanagh (ajkavanagh)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ironic-api (master)
Changed in charm-ironic-api:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-octavia (master)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ironic-api (master)

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/890556
Committed: https://opendev.org/openstack/charm-ironic-api/commit/113ccde9d445bedec482efdec12841aa32ecc91a
Submitter: "Zuul (22348)"
Branch: master

commit 113ccde9d445bedec482efdec12841aa32ecc91a
Author: Alex Kavanagh <email address hidden>
Date: Fri Aug 4 17:16:42 2023 +0100

    Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/824
    Closes-Bug: #2028683

    Change-Id: I3d454dbddda8006178797d381cd07856faca897d

Changed in charm-ironic-api:
status: In Progress → Fix Committed
Changed in charm-octavia:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-octavia (master)

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/890565
Committed: https://opendev.org/openstack/charm-octavia/commit/fc85965b3bb9601a4b273b07c6d55bcbd5917620
Submitter: "Zuul (22348)"
Branch: master

commit fc85965b3bb9601a4b273b07c6d55bcbd5917620
Author: Alex Kavanagh <email address hidden>
Date: Fri Aug 4 17:16:42 2023 +0100

    Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/824
    Closes-Bug: #2028683

    Change-Id: Id3636d55f26d05ffed994e3ccc0575ec66a5e389

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ironic-api (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891487

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-octavia (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891496

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ironic-api (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891537

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-octavia (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891546

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ironic-api (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891707

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-octavia (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891716

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ironic-api (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891851

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-octavia (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891860

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ironic-api (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891890

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-octavia (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891899

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ironic-api (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891958

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-octavia (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891967

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ironic-api (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/891996

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-octavia (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/charm-octavia/+/892005

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ironic-api (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/892022

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-octavia (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-octavia/+/892031

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-octavia (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/891546
Committed: https://opendev.org/openstack/charm-octavia/commit/ee1259750892536e1206ce182836c5983b861836
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit ee1259750892536e1206ce182836c5983b861836
Author: Alex Kavanagh <email address hidden>
Date: Tue Aug 15 22:22:12 2023 +0100

    [2023.1] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/825
    Closes-Bug: #2028683

    Change-Id: I0e7e2e67109f39a507b69de9d4f1e6abccf32f58

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ironic-api (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/891537
Committed: https://opendev.org/openstack/charm-ironic-api/commit/2a4af9398493be9d11d8dbe0751e51aa0fb39556
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 2a4af9398493be9d11d8dbe0751e51aa0fb39556
Author: Alex Kavanagh <email address hidden>
Date: Tue Aug 15 22:22:12 2023 +0100

    [2023.1] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/825
    Closes-Bug: #2028683

    Change-Id: Ie91b970c7d8233bcd27e3ebaa5804e23ab40e443

Revision history for this message
Alan Baghumian (alanbach) wrote :

I'm also encountering this exact same issue today, this fix has not yet been released for Yoga/Stable unfortunately.

Revision history for this message
Felipe Reyes (freyes) wrote : Re: [Bug 2028683] Re: [Yoga/Jammy] Octavia fails on hook "ovsdb-subordinate-relation-changed" due to missing CA

On Thu, 2023-08-31 at 18:25 +0000, Alan Baghumian wrote:
> I'm also encountering this exact same issue today, this fix has not yet
> been released for Yoga/Stable unfortunately.

that's correct and in sync with the tasks this bug has, you can monitor this patch too -
https://review.opendev.org/c/openstack/charm-octavia/+/891496

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on charm-octavia (stable/ussuri)

Change abandoned by "Alex Kavanagh <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-octavia/+/891967
Reason: Abandoning in favour of https://review.opendev.org/c/openstack/charm-octavia/+/892336 which is also a rebuild request, and thus essentially duplicates the rebuild/push.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ironic-api (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/892022
Committed: https://opendev.org/openstack/charm-ironic-api/commit/f376e92af7a3ca9b537a1635a1cc740d34cf8c05
Submitter: "Zuul (22348)"
Branch: stable/zed

commit f376e92af7a3ca9b537a1635a1cc740d34cf8c05
Author: Alex Kavanagh <email address hidden>
Date: Fri Aug 18 17:39:31 2023 +0100

    [zed] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/826
    Closes-Bug: #2028683

    Change-Id: Ic9c479d1076b22419706dfa8fb7940033d13d46e

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ironic-api (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/891487
Committed: https://opendev.org/openstack/charm-ironic-api/commit/dcdf7a7e856a2d8bc0aaeaccc22d3ad60e326655
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit dcdf7a7e856a2d8bc0aaeaccc22d3ad60e326655
Author: Alex Kavanagh <email address hidden>
Date: Tue Aug 15 15:13:22 2023 +0100

    [yoga] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/827
    Closes-Bug: #2028683

    Change-Id: I214d62984423c50e4c32b051e64ab5b2e851b9a8

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ironic-api (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/891707
Committed: https://opendev.org/openstack/charm-ironic-api/commit/8389fc923e01931b72dbddbe661f4c76e78e76b1
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 8389fc923e01931b72dbddbe661f4c76e78e76b1
Author: Alex Kavanagh <email address hidden>
Date: Thu Aug 17 13:55:52 2023 +0100

    [xena] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/828
    Closes-Bug: #2028683

    Change-Id: If06ad8d782a47edaa320da8d710211f3ec04e28c

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ironic-api (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/891851
Committed: https://opendev.org/openstack/charm-ironic-api/commit/7330d5ae7f78b3bdb94c5b72480c06958c534798
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 7330d5ae7f78b3bdb94c5b72480c06958c534798
Author: Alex Kavanagh <email address hidden>
Date: Thu Aug 17 16:28:50 2023 +0100

    [wallaby] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/829
    Closes-Bug: #2028683

    Change-Id: I3c7ca65fca93a43d659b3c4cd48ae2ef7dc35c79

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-octavia (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/892031
Committed: https://opendev.org/openstack/charm-octavia/commit/dfd3f35a5641814401dea7df68e17e66c0b3bf5f
Submitter: "Zuul (22348)"
Branch: stable/zed

commit dfd3f35a5641814401dea7df68e17e66c0b3bf5f
Author: Alex Kavanagh <email address hidden>
Date: Fri Aug 18 17:39:31 2023 +0100

    [zed] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/826
    Closes-Bug: #2028683

    Change-Id: I51b9784be407f724515aed655f4325be40057871

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-octavia (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/891496
Committed: https://opendev.org/openstack/charm-octavia/commit/57561cff59b6155709a791237ad885b01bc2fb20
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 57561cff59b6155709a791237ad885b01bc2fb20
Author: Alex Kavanagh <email address hidden>
Date: Tue Aug 15 15:13:22 2023 +0100

    [yoga] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/827
    Closes-Bug: #2028683

    Change-Id: I41ea5a58e0391d398fab2ecd44999e6f32a4054b

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-octavia (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/891716
Committed: https://opendev.org/openstack/charm-octavia/commit/aab20025c9219a76ceebcccbd04a481aeaeff54e
Submitter: "Zuul (22348)"
Branch: stable/xena

commit aab20025c9219a76ceebcccbd04a481aeaeff54e
Author: Alex Kavanagh <email address hidden>
Date: Thu Aug 17 13:55:52 2023 +0100

    [xena] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/828
    Closes-Bug: #2028683

    Change-Id: I1f7368e2851402d67f71f69590d1be6c8ddd25e3

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-octavia (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/891860
Committed: https://opendev.org/openstack/charm-octavia/commit/5f8111f0eb8aac8985da3e98fc1f91dade3d1db8
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 5f8111f0eb8aac8985da3e98fc1f91dade3d1db8
Author: Alex Kavanagh <email address hidden>
Date: Thu Aug 17 16:28:50 2023 +0100

    [wallaby] Ensure get_requests_for_local_unit doesn't fail on incomplete relation

    This is a rebuild/make sync for charms to pickup the fix in charmhelpers to fix
    any inadvertant accesses of ['ca'] in the relation data before it is available
    from vault in the certificates relation. Fix in charmhelpers is in [1].

    [1] https://github.com/juju/charm-helpers/pull/829
    Closes-Bug: #2028683

    Change-Id: Id883860c21bff001e693760a0535fd427e9c6d18

Revision history for this message
Alan Baghumian (alanbach) wrote :
Download full text (3.4 KiB)

Finally got tired of waiting and manually copied over the charmhelpers/contrib/openstack/cert_utils.py file to my Octavia units and performed a juju resolve. They went past the error and all green now!

Hopefully the updated charm will be available since this is not a proper way of fixing things :-D

$ sudo cp -v cert_utils.py /var/lib/juju/agents/unit-octavia-<unit#>/.venv/lib/python3.8/site-packages/charmhelpers/contrib/openstack/cert_utils.py

Model Controller Cloud/Region Version SLA Timestamp
vstack home-lab-default Home-Lab/default 2.9.37 unsupported 17:56:44-07:00

App Version Status Scale Charm Channel Rev Exposed Message
lds-client-focal active 3 landscape-client stable 49 no System successfully registered
octavia 10.0.0 active 3 octavia yoga/stable 193 no Unit is ready
octavia-hacluster active 3 hacluster 2.0.3/stable 113 no Unit is ready and clustered
octavia-mysql-router 8.0.34 active 3 mysql-router 8.0/stable 90 no Unit is ready
octavia-ovn-chassis 22.03.2 active 3 ovn-chassis 22.03/stable 143 no Unit is ready

Unit Workload Agent Machine Public address Ports Message
octavia/17* active idle 9/lxd/12 10.1.15.145 9876/tcp Unit is ready
  lds-client-focal/691 active idle 10.1.15.145 System successfully registered
  octavia-hacluster/99 active idle 10.1.15.145 Unit is ready and clustered
  octavia-mysql-router/122* active idle 10.1.15.145 Unit is ready
  octavia-ovn-chassis/119* active idle 10.1.15.145 Unit is ready
octavia/18 active idle 102/lxd/5 10.1.15.146 9876/tcp Unit is ready
  lds-client-focal/690 active idle 10.1.15.146 System successfully registered
  octavia-hacluster/98 active idle 10.1.15.146 Unit is ready and clustered
  octavia-mysql-router/123 active idle 10.1.15.146 Unit is ready
  octavia-ovn-chassis/118 active idle 10.1.15.146 Unit is ready
octavia/19 active idle 104/lxd/4 10.1.15.147 9876/tcp Unit is ready
  lds-client-focal/689 active idle 10.1.15.147 System successfully registered
  octavia-hacluster/97* active idle 10.1.15.147 Unit is ready and clustered
  octavia-mysql-router/121 active idle 10.1.15.147 Unit is ready
  octavia-ovn-chassis/117 active idle 10.1.15.147 Unit is ready

Machine State Address Inst id Series AZ Message
9 started 10.1.8.12 os-vm-2 focal default Deployed
9/lxd/12 started 10.1.15.145 juju-b096f0-9-lxd-12 focal default Container started
102 started 10.1.8.25 ...

Read more...

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Hi @alanback

I'm not sure which channel of the octavia-charm you are using, but the fix has landed to all of the following channels: ussuri, wallaby, xena, yoga, zed, 2023.1 (antelope) and master. The victoria and (bionic - queens, rocky, stein, train) are currently not building.

There's a tooling issue, as the patches have landed but the bug was not updated. I've manually updated the ones that have landed now.

Sorry for the incorrect information.

Thanks!

Changed in charm-helpers:
status: In Progress → Fix Released
Revision history for this message
Alan Baghumian (alanbach) wrote :

Hi Alex,

I am on Yoga/Stable (Focal) and upgraded the charm to revision 193 when it cam out a couple of weeks ago:

channels: |
  yoga/stable: 20bbe01 2023-09-08 (193) 41MB

$ juju status octavia
Model Controller Cloud/Region Version SLA Timestamp
vstack home-lab-default Home-Lab/default 2.9.37 unsupported 09:22:04-07:00

App Version Status Scale Charm Channel Rev Exposed Message
lds-client-focal active 3 landscape-client stable 49 no System successfully registered
octavia 10.0.0 active 3 octavia yoga/stable 193 no Unit is ready
octavia-hacluster active 3 hacluster 2.0.3/stable 113 no Unit is ready and clustered
octavia-mysql-router 8.0.34 active 3 mysql-router 8.0/stable 90 no Unit is ready
octavia-ovn-chassis 22.03.2 active 3 ovn-chassis 22.03/stable 143 no Unit is ready

However that did not fix this issue until I manually copied the file yesterday.

So, if 193 is indeed the latest for Yoga/Stable, it did not fix the problem.

Hope this helps!

Best,
Alan

Revision history for this message
Felipe Reyes (freyes) wrote : Re: [Bug 2028683] Re: [Yoga/Jammy] Octavia fails on hook "ovsdb-subordinate-relation-changed" due to missing CA

On Sun, 2023-09-24 at 16:24 +0000, Alan Baghumian wrote:
> Hi Alex,
>
> I am on Yoga/Stable (Focal) and upgraded the charm to revision 193 when
> it cam out a couple of weeks ago:[...]
when you upgraded, were the hooks in error state?, if they were, you need to use the "--force-units"
flag to make juju upgrade the charm, otherwise it will wait for the hook failures get resolved.

You can check the output of "juju show-status-log <entity name>" to know when/if the upgrade-charm
hook was executed.

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Hi; you made me check! :)

I downloaded the charm manually for octavia (juju download octavia --channel=yoga/stable) and checked the charmhelpers/contrib/cert_utils.py at line 417 and it does contain the fix.

Felipe raises a point on whether the units were actually upgraded rather than still waiting on failed hooks. Another method would be to add a unit (if possible) and see if it works okay.

Revision history for this message
Alan Baghumian (alanbach) wrote :

Hi @Felipe! You nailed it:

$ juju show-status-log octavia/19 --days 30 | grep -c upgrade-charm
0

The hook never actually ran even after running juju resolve on units.

Hi @Alex! Sorry I made you double check this and thanks much for checking.

I am now running the upgrade hook.

Best,
Alan

Revision history for this message
Alan Baghumian (alanbach) wrote :

We are good! Just refreshed to xena/stable and then back to yoga/stable. All good.

$ juju refresh octavia --channel xena/stable

$ juju refresh octavia --channel yoga/stable

Best,
Alan

Revision history for this message
Felipe Reyes (freyes) wrote :

On Tue, 2023-09-26 at 16:51 +0000, Alan Baghumian wrote:
> We are good! Just refreshed to xena/stable and then back to yoga/stable.
> All good.

that's great.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.