Activity log for bug #2028683

Date Who What changed Old value New value Message
2023-07-25 15:27:14 Bas de Bruijne bug added bug
2023-07-25 15:27:38 Bas de Bruijne tags cdo-qa foundations-engine
2023-07-25 17:54:00 Alex Kavanagh charm-octavia: status New Incomplete
2023-07-25 18:11:10 Bas de Bruijne summary Octavia fails on hook "ovsdb-subordinate-relation-changed" due to missing CA [Yoga/Jammy] Octavia fails on hook "ovsdb-subordinate-relation-changed" due to missing CA
2023-07-25 18:11:14 Bas de Bruijne charm-octavia: status Incomplete New
2023-07-25 20:01:45 Alex Kavanagh description In test run https://solutions.qa.canonical.com/testruns/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/, the octavia units get into an error state: ================ octavia/0 error idle 3/lxd/9 10.246.164.228 9876/tcp hook failed: "ovsdb-subordinate-relation-changed" filebeat/77 active idle 10.246.164.228 Filebeat ready. hacluster-octavia/1 active idle 10.246.164.228 Unit is ready and clustered landscape-client/77 maintenance idle 10.246.164.228 Need computer-title and juju-info to proceed logrotated/77 active idle 10.246.164.228 Unit is ready. nrpe/85 active idle 10.246.164.228 icmp,5666/tcp Ready octavia-mysql-router/2 active idle 10.246.164.228 Unit is ready octavia-ovn-chassis/1 active idle 10.246.164.228 Unit is ready prometheus-grok-exporter/77 active idle 10.246.164.228 9144/tcp Unit is ready public-policy-routing/48 active idle 10.246.164.228 Unit is ready telegraf/77 active idle 10.246.164.228 9103/tcp Monitoring octavia/0 (source version/commit 23.01-8-...) ubuntu-advantage/77 active idle 10.246.164.228 Attached (esm-apps,esm-infra) octavia/1 error idle 4/lxd/9 10.246.167.116 9876/tcp hook failed: "ovsdb-cms-relation-changed" filebeat/78 active idle 10.246.167.116 Filebeat ready. hacluster-octavia/2 active idle 10.246.167.116 Unit is ready and clustered landscape-client/78 maintenance idle 10.246.167.116 Need computer-title and juju-info to proceed logrotated/78 active idle 10.246.167.116 Unit is ready. nrpe/86 active idle 10.246.167.116 icmp,5666/tcp Ready octavia-mysql-router/1 active idle 10.246.167.116 Unit is ready octavia-ovn-chassis/2 active idle 10.246.167.116 Unit is ready prometheus-grok-exporter/78 active idle 10.246.167.116 9144/tcp Unit is ready public-policy-routing/49 active idle 10.246.167.116 Unit is ready telegraf/78 active idle 10.246.167.116 9103/tcp Monitoring octavia/1 (source version/commit 23.01-8-...) ubuntu-advantage/78 active idle 10.246.167.116 Attached (esm-apps,esm-infra) octavia/2* error idle 5/lxd/9 10.246.167.69 9876/tcp hook failed: "ovsdb-subordinate-relation-changed" filebeat/75 active idle 10.246.167.69 Filebeat ready. hacluster-octavia/0* active idle 10.246.167.69 Unit is ready and clustered landscape-client/75 maintenance idle 10.246.167.69 Need computer-title and juju-info to proceed logrotated/75 active idle 10.246.167.69 Unit is ready. nrpe/83 active idle 10.246.167.69 icmp,5666/tcp Ready octavia-mysql-router/0* active idle 10.246.167.69 Unit is ready octavia-ovn-chassis/0* active idle 10.246.167.69 Unit is ready prometheus-grok-exporter/75 active idle 10.246.167.69 9144/tcp Unit is ready public-policy-routing/46 active idle 10.246.167.69 Unit is ready telegraf/75 active idle 10.246.167.69 9103/tcp Monitoring octavia/2 (source version/commit 23.01-8-...) ubuntu-advantage/75 active idle 10.246.167.69 Attached (esm-apps,esm-infra) ================ In the debug log, we see the following message: ================ Traceback (most recent call last): File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 808, in render_configs _render(os.path.basename(conf)) File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render charmhelpers.core.templating.render( File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 80, in render content = template.render(context) File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1291, in render self.environment.handle_exception() File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 926, in handle_exception raise rewrite_traceback_stack(source=source) File "templates/openstack_https_frontend.conf", line 1, in top-level template code {% if options.endpoints -%} File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 475, in getattr return getattr(obj, attribute) File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/adapters.py", line 1025, in endpoints int_port = ch_cluster.determine_api_port( File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 265, in determine_api_port if https(): File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 228, in https cert_utils.get_requests_for_local_unit("certificates") File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/openstack/cert_utils.py", line 424, in get_requests_for_local_unit 'ca': data['ca'], KeyError: 'ca' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main bus.dispatch(restricted=restricted_mode) File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch _invoke(other_handlers) File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke handler.invoke() File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke self._action(*args) File "/var/lib/juju/agents/unit-octavia-0/charm/reactive/octavia_handlers.py", line 272, in render octavia_charm.render_with_interfaces( File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 827, in render_with_interfaces self.render_configs( File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 814, in render_configs _render('_'.join(conf.split(os.path.sep))[1:]) File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render charmhelpers.core.templating.render( File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 79, in render raise e File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 74, in render template = template_env.get_template(source) File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1000, in get_template return self._load_template(name, globals) File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 959, in _load_template template = self.loader.load(self, name, self.make_globals(globals)) File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/loaders.py", line 575, in load raise TemplateNotFound(name) jinja2.exceptions.TemplateNotFound: etc_apache2_sites-available_openstack_https_frontend.conf ================ It looks like the CA isn't found, but then a template is also missing from the charm. I'm not sure what the template would be for. Crashdumps and configs can be found here: https://oil-jenkins.canonical.com/artifacts/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/index.html [edit]: from comment #2: ====== maas - 3.2.8- juju - 2.9.44 cpe-foundation - 2.21 infra-ubuntu - focal ceph - quincy/stable openstack-charms - yoga/stable charmed-kubernetes - 1.27 cloud-init - 23.2.1-0ubuntu0~22.04.1 fce-container-image - ubuntu:jammy openstack - yoga sku - fcb-master-yoga-jammy legacy-lma - latest/candidate solutions-qa-ci - d0ba8ed0 landscape-server - 23.03+16-0landscape0 cos-lite - latest/stable:11 fcbtest - latest/beta ====== --- In test run https://solutions.qa.canonical.com/testruns/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/, the octavia units get into an error state: ================ octavia/0 error idle 3/lxd/9 10.246.164.228 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"   filebeat/77 active idle 10.246.164.228 Filebeat ready.   hacluster-octavia/1 active idle 10.246.164.228 Unit is ready and clustered   landscape-client/77 maintenance idle 10.246.164.228 Need computer-title and juju-info to proceed   logrotated/77 active idle 10.246.164.228 Unit is ready.   nrpe/85 active idle 10.246.164.228 icmp,5666/tcp Ready   octavia-mysql-router/2 active idle 10.246.164.228 Unit is ready   octavia-ovn-chassis/1 active idle 10.246.164.228 Unit is ready   prometheus-grok-exporter/77 active idle 10.246.164.228 9144/tcp Unit is ready   public-policy-routing/48 active idle 10.246.164.228 Unit is ready   telegraf/77 active idle 10.246.164.228 9103/tcp Monitoring octavia/0 (source version/commit 23.01-8-...)   ubuntu-advantage/77 active idle 10.246.164.228 Attached (esm-apps,esm-infra) octavia/1 error idle 4/lxd/9 10.246.167.116 9876/tcp hook failed: "ovsdb-cms-relation-changed"   filebeat/78 active idle 10.246.167.116 Filebeat ready.   hacluster-octavia/2 active idle 10.246.167.116 Unit is ready and clustered   landscape-client/78 maintenance idle 10.246.167.116 Need computer-title and juju-info to proceed   logrotated/78 active idle 10.246.167.116 Unit is ready.   nrpe/86 active idle 10.246.167.116 icmp,5666/tcp Ready   octavia-mysql-router/1 active idle 10.246.167.116 Unit is ready   octavia-ovn-chassis/2 active idle 10.246.167.116 Unit is ready   prometheus-grok-exporter/78 active idle 10.246.167.116 9144/tcp Unit is ready   public-policy-routing/49 active idle 10.246.167.116 Unit is ready   telegraf/78 active idle 10.246.167.116 9103/tcp Monitoring octavia/1 (source version/commit 23.01-8-...)   ubuntu-advantage/78 active idle 10.246.167.116 Attached (esm-apps,esm-infra) octavia/2* error idle 5/lxd/9 10.246.167.69 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"   filebeat/75 active idle 10.246.167.69 Filebeat ready.   hacluster-octavia/0* active idle 10.246.167.69 Unit is ready and clustered   landscape-client/75 maintenance idle 10.246.167.69 Need computer-title and juju-info to proceed   logrotated/75 active idle 10.246.167.69 Unit is ready.   nrpe/83 active idle 10.246.167.69 icmp,5666/tcp Ready   octavia-mysql-router/0* active idle 10.246.167.69 Unit is ready   octavia-ovn-chassis/0* active idle 10.246.167.69 Unit is ready   prometheus-grok-exporter/75 active idle 10.246.167.69 9144/tcp Unit is ready   public-policy-routing/46 active idle 10.246.167.69 Unit is ready   telegraf/75 active idle 10.246.167.69 9103/tcp Monitoring octavia/2 (source version/commit 23.01-8-...)   ubuntu-advantage/75 active idle 10.246.167.69 Attached (esm-apps,esm-infra) ================ In the debug log, we see the following message: ================ Traceback (most recent call last):   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 808, in render_configs     _render(os.path.basename(conf))   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render     charmhelpers.core.templating.render(   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 80, in render     content = template.render(context)   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1291, in render     self.environment.handle_exception()   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 926, in handle_exception     raise rewrite_traceback_stack(source=source)   File "templates/openstack_https_frontend.conf", line 1, in top-level template code     {% if options.endpoints -%}   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 475, in getattr     return getattr(obj, attribute)   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/adapters.py", line 1025, in endpoints     int_port = ch_cluster.determine_api_port(   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 265, in determine_api_port     if https():   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 228, in https     cert_utils.get_requests_for_local_unit("certificates")   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/openstack/cert_utils.py", line 424, in get_requests_for_local_unit     'ca': data['ca'], KeyError: 'ca' During handling of the above exception, another exception occurred: Traceback (most recent call last):   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main     bus.dispatch(restricted=restricted_mode)   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch     _invoke(other_handlers)   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke     handler.invoke()   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke     self._action(*args)   File "/var/lib/juju/agents/unit-octavia-0/charm/reactive/octavia_handlers.py", line 272, in render     octavia_charm.render_with_interfaces(   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 827, in render_with_interfaces     self.render_configs(   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 814, in render_configs     _render('_'.join(conf.split(os.path.sep))[1:])   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render     charmhelpers.core.templating.render(   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 79, in render     raise e   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 74, in render     template = template_env.get_template(source)   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1000, in get_template     return self._load_template(name, globals)   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 959, in _load_template     template = self.loader.load(self, name, self.make_globals(globals))   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/loaders.py", line 575, in load     raise TemplateNotFound(name) jinja2.exceptions.TemplateNotFound: etc_apache2_sites-available_openstack_https_frontend.conf ================ It looks like the CA isn't found, but then a template is also missing from the charm. I'm not sure what the template would be for. Crashdumps and configs can be found here: https://oil-jenkins.canonical.com/artifacts/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/index.html
2023-07-25 20:10:02 Alex Kavanagh description [edit]: from comment #2: ====== maas - 3.2.8- juju - 2.9.44 cpe-foundation - 2.21 infra-ubuntu - focal ceph - quincy/stable openstack-charms - yoga/stable charmed-kubernetes - 1.27 cloud-init - 23.2.1-0ubuntu0~22.04.1 fce-container-image - ubuntu:jammy openstack - yoga sku - fcb-master-yoga-jammy legacy-lma - latest/candidate solutions-qa-ci - d0ba8ed0 landscape-server - 23.03+16-0landscape0 cos-lite - latest/stable:11 fcbtest - latest/beta ====== --- In test run https://solutions.qa.canonical.com/testruns/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/, the octavia units get into an error state: ================ octavia/0 error idle 3/lxd/9 10.246.164.228 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"   filebeat/77 active idle 10.246.164.228 Filebeat ready.   hacluster-octavia/1 active idle 10.246.164.228 Unit is ready and clustered   landscape-client/77 maintenance idle 10.246.164.228 Need computer-title and juju-info to proceed   logrotated/77 active idle 10.246.164.228 Unit is ready.   nrpe/85 active idle 10.246.164.228 icmp,5666/tcp Ready   octavia-mysql-router/2 active idle 10.246.164.228 Unit is ready   octavia-ovn-chassis/1 active idle 10.246.164.228 Unit is ready   prometheus-grok-exporter/77 active idle 10.246.164.228 9144/tcp Unit is ready   public-policy-routing/48 active idle 10.246.164.228 Unit is ready   telegraf/77 active idle 10.246.164.228 9103/tcp Monitoring octavia/0 (source version/commit 23.01-8-...)   ubuntu-advantage/77 active idle 10.246.164.228 Attached (esm-apps,esm-infra) octavia/1 error idle 4/lxd/9 10.246.167.116 9876/tcp hook failed: "ovsdb-cms-relation-changed"   filebeat/78 active idle 10.246.167.116 Filebeat ready.   hacluster-octavia/2 active idle 10.246.167.116 Unit is ready and clustered   landscape-client/78 maintenance idle 10.246.167.116 Need computer-title and juju-info to proceed   logrotated/78 active idle 10.246.167.116 Unit is ready.   nrpe/86 active idle 10.246.167.116 icmp,5666/tcp Ready   octavia-mysql-router/1 active idle 10.246.167.116 Unit is ready   octavia-ovn-chassis/2 active idle 10.246.167.116 Unit is ready   prometheus-grok-exporter/78 active idle 10.246.167.116 9144/tcp Unit is ready   public-policy-routing/49 active idle 10.246.167.116 Unit is ready   telegraf/78 active idle 10.246.167.116 9103/tcp Monitoring octavia/1 (source version/commit 23.01-8-...)   ubuntu-advantage/78 active idle 10.246.167.116 Attached (esm-apps,esm-infra) octavia/2* error idle 5/lxd/9 10.246.167.69 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"   filebeat/75 active idle 10.246.167.69 Filebeat ready.   hacluster-octavia/0* active idle 10.246.167.69 Unit is ready and clustered   landscape-client/75 maintenance idle 10.246.167.69 Need computer-title and juju-info to proceed   logrotated/75 active idle 10.246.167.69 Unit is ready.   nrpe/83 active idle 10.246.167.69 icmp,5666/tcp Ready   octavia-mysql-router/0* active idle 10.246.167.69 Unit is ready   octavia-ovn-chassis/0* active idle 10.246.167.69 Unit is ready   prometheus-grok-exporter/75 active idle 10.246.167.69 9144/tcp Unit is ready   public-policy-routing/46 active idle 10.246.167.69 Unit is ready   telegraf/75 active idle 10.246.167.69 9103/tcp Monitoring octavia/2 (source version/commit 23.01-8-...)   ubuntu-advantage/75 active idle 10.246.167.69 Attached (esm-apps,esm-infra) ================ In the debug log, we see the following message: ================ Traceback (most recent call last):   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 808, in render_configs     _render(os.path.basename(conf))   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render     charmhelpers.core.templating.render(   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 80, in render     content = template.render(context)   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1291, in render     self.environment.handle_exception()   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 926, in handle_exception     raise rewrite_traceback_stack(source=source)   File "templates/openstack_https_frontend.conf", line 1, in top-level template code     {% if options.endpoints -%}   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 475, in getattr     return getattr(obj, attribute)   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/adapters.py", line 1025, in endpoints     int_port = ch_cluster.determine_api_port(   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 265, in determine_api_port     if https():   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 228, in https     cert_utils.get_requests_for_local_unit("certificates")   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/openstack/cert_utils.py", line 424, in get_requests_for_local_unit     'ca': data['ca'], KeyError: 'ca' During handling of the above exception, another exception occurred: Traceback (most recent call last):   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main     bus.dispatch(restricted=restricted_mode)   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch     _invoke(other_handlers)   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke     handler.invoke()   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke     self._action(*args)   File "/var/lib/juju/agents/unit-octavia-0/charm/reactive/octavia_handlers.py", line 272, in render     octavia_charm.render_with_interfaces(   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 827, in render_with_interfaces     self.render_configs(   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 814, in render_configs     _render('_'.join(conf.split(os.path.sep))[1:])   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render     charmhelpers.core.templating.render(   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 79, in render     raise e   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 74, in render     template = template_env.get_template(source)   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1000, in get_template     return self._load_template(name, globals)   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 959, in _load_template     template = self.loader.load(self, name, self.make_globals(globals))   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/loaders.py", line 575, in load     raise TemplateNotFound(name) jinja2.exceptions.TemplateNotFound: etc_apache2_sites-available_openstack_https_frontend.conf ================ It looks like the CA isn't found, but then a template is also missing from the charm. I'm not sure what the template would be for. Crashdumps and configs can be found here: https://oil-jenkins.canonical.com/artifacts/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/index.html [edit]: from comment #2: ====== maas - 3.2.8- juju - 2.9.44 cpe-foundation - 2.21 infra-ubuntu - focal ceph - quincy/stable openstack-charms - yoga/stable charmed-kubernetes - 1.27 cloud-init - 23.2.1-0ubuntu0~22.04.1 fce-container-image - ubuntu:jammy openstack - yoga sku - fcb-master-yoga-jammy legacy-lma - latest/candidate solutions-qa-ci - d0ba8ed0 landscape-server - 23.03+16-0landscape0 cos-lite - latest/stable:11 fcbtest - latest/beta ====== --- The problem is that the Octavia units go into an error state during deployment after Vault is initialized but before Octavia is initialized. The expected behaviour is that the Octavia units do not go into an error state. A more detailed overview of the symptoms can be found in the first comment. In test run https://solutions.qa.canonical.com/testruns/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/, the octavia units get into an error state: ================ octavia/0 error idle 3/lxd/9 10.246.164.228 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"   filebeat/77 active idle 10.246.164.228 Filebeat ready.   hacluster-octavia/1 active idle 10.246.164.228 Unit is ready and clustered   landscape-client/77 maintenance idle 10.246.164.228 Need computer-title and juju-info to proceed   logrotated/77 active idle 10.246.164.228 Unit is ready.   nrpe/85 active idle 10.246.164.228 icmp,5666/tcp Ready   octavia-mysql-router/2 active idle 10.246.164.228 Unit is ready   octavia-ovn-chassis/1 active idle 10.246.164.228 Unit is ready   prometheus-grok-exporter/77 active idle 10.246.164.228 9144/tcp Unit is ready   public-policy-routing/48 active idle 10.246.164.228 Unit is ready   telegraf/77 active idle 10.246.164.228 9103/tcp Monitoring octavia/0 (source version/commit 23.01-8-...)   ubuntu-advantage/77 active idle 10.246.164.228 Attached (esm-apps,esm-infra) octavia/1 error idle 4/lxd/9 10.246.167.116 9876/tcp hook failed: "ovsdb-cms-relation-changed"   filebeat/78 active idle 10.246.167.116 Filebeat ready.   hacluster-octavia/2 active idle 10.246.167.116 Unit is ready and clustered   landscape-client/78 maintenance idle 10.246.167.116 Need computer-title and juju-info to proceed   logrotated/78 active idle 10.246.167.116 Unit is ready.   nrpe/86 active idle 10.246.167.116 icmp,5666/tcp Ready   octavia-mysql-router/1 active idle 10.246.167.116 Unit is ready   octavia-ovn-chassis/2 active idle 10.246.167.116 Unit is ready   prometheus-grok-exporter/78 active idle 10.246.167.116 9144/tcp Unit is ready   public-policy-routing/49 active idle 10.246.167.116 Unit is ready   telegraf/78 active idle 10.246.167.116 9103/tcp Monitoring octavia/1 (source version/commit 23.01-8-...)   ubuntu-advantage/78 active idle 10.246.167.116 Attached (esm-apps,esm-infra) octavia/2* error idle 5/lxd/9 10.246.167.69 9876/tcp hook failed: "ovsdb-subordinate-relation-changed"   filebeat/75 active idle 10.246.167.69 Filebeat ready.   hacluster-octavia/0* active idle 10.246.167.69 Unit is ready and clustered   landscape-client/75 maintenance idle 10.246.167.69 Need computer-title and juju-info to proceed   logrotated/75 active idle 10.246.167.69 Unit is ready.   nrpe/83 active idle 10.246.167.69 icmp,5666/tcp Ready   octavia-mysql-router/0* active idle 10.246.167.69 Unit is ready   octavia-ovn-chassis/0* active idle 10.246.167.69 Unit is ready   prometheus-grok-exporter/75 active idle 10.246.167.69 9144/tcp Unit is ready   public-policy-routing/46 active idle 10.246.167.69 Unit is ready   telegraf/75 active idle 10.246.167.69 9103/tcp Monitoring octavia/2 (source version/commit 23.01-8-...)   ubuntu-advantage/75 active idle 10.246.167.69 Attached (esm-apps,esm-infra) ================ In the debug log, we see the following message: ================ Traceback (most recent call last):   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 808, in render_configs     _render(os.path.basename(conf))   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render     charmhelpers.core.templating.render(   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 80, in render     content = template.render(context)   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1291, in render     self.environment.handle_exception()   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 926, in handle_exception     raise rewrite_traceback_stack(source=source)   File "templates/openstack_https_frontend.conf", line 1, in top-level template code     {% if options.endpoints -%}   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 475, in getattr     return getattr(obj, attribute)   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/adapters.py", line 1025, in endpoints     int_port = ch_cluster.determine_api_port(   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 265, in determine_api_port     if https():   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/hahelpers/cluster.py", line 228, in https     cert_utils.get_requests_for_local_unit("certificates")   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/contrib/openstack/cert_utils.py", line 424, in get_requests_for_local_unit     'ca': data['ca'], KeyError: 'ca' During handling of the above exception, another exception occurred: Traceback (most recent call last):   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main     bus.dispatch(restricted=restricted_mode)   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch     _invoke(other_handlers)   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke     handler.invoke()   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke     self._action(*args)   File "/var/lib/juju/agents/unit-octavia-0/charm/reactive/octavia_handlers.py", line 272, in render     octavia_charm.render_with_interfaces(   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 827, in render_with_interfaces     self.render_configs(   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 814, in render_configs     _render('_'.join(conf.split(os.path.sep))[1:])   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charms_openstack/charm/core.py", line 797, in _render     charmhelpers.core.templating.render(   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 79, in render     raise e   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/charmhelpers/core/templating.py", line 74, in render     template = template_env.get_template(source)   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1000, in get_template     return self._load_template(name, globals)   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 959, in _load_template     template = self.loader.load(self, name, self.make_globals(globals))   File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.10/site-packages/jinja2/loaders.py", line 575, in load     raise TemplateNotFound(name) jinja2.exceptions.TemplateNotFound: etc_apache2_sites-available_openstack_https_frontend.conf ================ It looks like the CA isn't found, but then a template is also missing from the charm. I'm not sure what the template would be for. Crashdumps and configs can be found here: https://oil-jenkins.canonical.com/artifacts/1d6752e4-e03a-476f-9f5b-d4de9e1e172d/index.html
2023-07-26 17:55:22 Alex Kavanagh charm-octavia: status New Triaged
2023-07-26 17:55:24 Alex Kavanagh charm-octavia: importance Undecided Critical
2023-07-26 19:11:11 Alex Kavanagh charm-octavia: assignee Alex Kavanagh (ajkavanagh)
2023-07-27 05:26:45 Nobuto Murata bug added subscriber Nobuto Murata
2023-07-27 12:02:29 Alex Kavanagh charm-octavia: status Triaged In Progress
2023-07-27 13:27:18 Alex Kavanagh bug task added charm-ironic-api
2023-07-27 13:27:41 Alex Kavanagh bug task added charm-helpers
2023-07-27 13:27:47 Alex Kavanagh charm-helpers: status New In Progress
2023-07-27 13:27:51 Alex Kavanagh charm-helpers: importance Undecided Critical
2023-07-27 13:27:54 Alex Kavanagh charm-helpers: assignee Alex Kavanagh (ajkavanagh)
2023-07-27 13:29:06 Alex Kavanagh nominated for series charm-octavia/victoria
2023-07-27 13:29:06 Alex Kavanagh bug task added charm-octavia/victoria
2023-07-27 13:29:06 Alex Kavanagh nominated for series charm-octavia/yoga
2023-07-27 13:29:06 Alex Kavanagh bug task added charm-octavia/yoga
2023-07-27 13:29:06 Alex Kavanagh nominated for series charm-octavia/2023.1
2023-07-27 13:29:06 Alex Kavanagh bug task added charm-octavia/2023.1
2023-07-27 13:29:06 Alex Kavanagh nominated for series charm-octavia/xena
2023-07-27 13:29:06 Alex Kavanagh bug task added charm-octavia/xena
2023-07-27 13:29:06 Alex Kavanagh nominated for series charm-octavia/ussuri
2023-07-27 13:29:06 Alex Kavanagh bug task added charm-octavia/ussuri
2023-07-27 13:29:06 Alex Kavanagh nominated for series charm-octavia/trunk
2023-07-27 13:29:06 Alex Kavanagh bug task added charm-octavia/trunk
2023-07-27 13:29:06 Alex Kavanagh nominated for series charm-octavia/wallaby
2023-07-27 13:29:06 Alex Kavanagh bug task added charm-octavia/wallaby
2023-07-27 13:29:06 Alex Kavanagh nominated for series charm-octavia/zed
2023-07-27 13:29:06 Alex Kavanagh bug task added charm-octavia/zed
2023-07-27 13:29:33 Alex Kavanagh nominated for series charm-ironic-api/victoria
2023-07-27 13:29:33 Alex Kavanagh bug task added charm-ironic-api/victoria
2023-07-27 13:29:33 Alex Kavanagh nominated for series charm-ironic-api/2023.1
2023-07-27 13:29:33 Alex Kavanagh bug task added charm-ironic-api/2023.1
2023-07-27 13:29:33 Alex Kavanagh nominated for series charm-ironic-api/yoga
2023-07-27 13:29:33 Alex Kavanagh bug task added charm-ironic-api/yoga
2023-07-27 13:29:33 Alex Kavanagh nominated for series charm-ironic-api/ussuri
2023-07-27 13:29:33 Alex Kavanagh bug task added charm-ironic-api/ussuri
2023-07-27 13:29:33 Alex Kavanagh nominated for series charm-ironic-api/xena
2023-07-27 13:29:33 Alex Kavanagh bug task added charm-ironic-api/xena
2023-07-27 13:29:33 Alex Kavanagh nominated for series charm-ironic-api/trunk
2023-07-27 13:29:33 Alex Kavanagh bug task added charm-ironic-api/trunk
2023-07-27 13:29:33 Alex Kavanagh nominated for series charm-ironic-api/zed
2023-07-27 13:29:33 Alex Kavanagh bug task added charm-ironic-api/zed
2023-07-27 13:29:33 Alex Kavanagh nominated for series charm-ironic-api/wallaby
2023-07-27 13:29:33 Alex Kavanagh bug task added charm-ironic-api/wallaby
2023-08-04 17:47:35 OpenStack Infra charm-ironic-api: status New In Progress
2023-08-07 22:03:03 OpenStack Infra charm-ironic-api: status In Progress Fix Committed
2023-08-07 22:04:19 OpenStack Infra charm-octavia: status In Progress Fix Committed
2023-08-14 08:50:41 Yoshi Kadokawa bug added subscriber Yoshi Kadokawa
2023-08-15 14:16:19 OpenStack Infra charm-ironic-api/yoga: status New In Progress
2023-08-15 21:24:20 OpenStack Infra charm-ironic-api/2023.1: status New In Progress
2023-08-17 12:58:06 OpenStack Infra charm-ironic-api/xena: status New In Progress
2023-08-17 15:31:18 OpenStack Infra charm-ironic-api/wallaby: status New In Progress
2023-08-17 19:45:10 OpenStack Infra charm-ironic-api/victoria: status New In Progress
2023-08-18 11:16:11 OpenStack Infra charm-ironic-api/ussuri: status New In Progress
2023-08-18 16:41:11 OpenStack Infra charm-ironic-api/zed: status New In Progress
2023-08-21 16:49:11 Jeffrey Chang bug added subscriber Jeffrey Chang
2023-08-22 18:01:58 OpenStack Infra charm-ironic-api/2023.1: status In Progress Fix Committed
2023-09-08 15:48:03 OpenStack Infra charm-ironic-api/zed: status In Progress Fix Committed
2023-09-08 15:48:08 OpenStack Infra charm-ironic-api/yoga: status In Progress Fix Committed
2023-09-08 15:48:14 OpenStack Infra charm-ironic-api/xena: status In Progress Fix Committed
2023-09-08 15:48:19 OpenStack Infra charm-ironic-api/wallaby: status In Progress Fix Committed
2023-09-24 00:45:33 Alan Baghumian bug added subscriber Alan Baghumian
2023-09-24 16:01:21 Alex Kavanagh charm-octavia/ussuri: status New Fix Released
2023-09-24 16:01:33 Alex Kavanagh charm-octavia/wallaby: status New Fix Released
2023-09-24 16:01:43 Alex Kavanagh charm-octavia/xena: status New Fix Released
2023-09-24 16:01:53 Alex Kavanagh charm-octavia/yoga: status New Fix Released
2023-09-24 16:02:04 Alex Kavanagh charm-octavia/zed: status New Fix Released
2023-09-24 16:02:17 Alex Kavanagh charm-octavia/2023.1: status New Fix Released
2023-09-24 16:02:38 Alex Kavanagh charm-helpers: status In Progress Fix Released
2023-09-24 16:03:34 Alex Kavanagh charm-octavia/victoria: status New In Progress