action managed upgrade fails with "jinja2.exceptions.UndefinedError: 'storage_ceph' is undefined"

Bug #1908267 reported by Alex Kavanagh
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Gnocchi Charm
Fix Released
Undecided
Unassigned
OpenStack Principle Layer
Fix Released
Undecided
Unassigned

Bug Description

The action managed upgrade fails as the interfaces don't get added to the context for the Jinja2 renderer:

      Traceback (most recent call last):
        File "/var/lib/juju/agents/unit-gnocchi-0/charm/actions/openstack-upgrade", line 54, in main
          action(args)
        File "/var/lib/juju/agents/unit-gnocchi-0/charm/actions/openstack-upgrade", line 35, in openstack_upgrade_action
          charm_instance.run_upgrade()
        File "/var/lib/juju/agents/unit-gnocchi-0/.venv/lib/python3.8/site-packages/charms_openstack/charm/core.py", line 1030, in run_upgrade
          target_charm.do_openstack_upgrade_config_render(interfaces_list)
        File "/var/lib/juju/agents/unit-gnocchi-0/.venv/lib/python3.8/site-packages/charms_openstack/charm/core.py", line 1112, in do_openstack_upgrade_config_render
          self.render_all_configs()
        File "/var/lib/juju/agents/unit-gnocchi-0/.venv/lib/python3.8/site-packages/charms_openstack/charm/core.py", line 794, in render_all_configs
          self.render_configs(self.full_restart_map.keys(),
        File "/var/lib/juju/agents/unit-gnocchi-0/.venv/lib/python3.8/site-packages/charms_openstack/charm/core.py", line 873, in render_configs
          _render(os.path.basename(conf))
        File "/var/lib/juju/agents/unit-gnocchi-0/.venv/lib/python3.8/site-packages/charms_openstack/charm/core.py", line 862, in _render
          charmhelpers.core.templating.render(
        File "/var/lib/juju/agents/unit-gnocchi-0/.venv/lib/python3.8/site-packages/charmhelpers/core/templating.py", line 85, in render
          content = template.render(context)
        File "/var/lib/juju/agents/unit-gnocchi-0/.venv/lib/python3.8/site-packages/jinja2/asyncsupport.py", line 76, in render
          return original_render(self, *args, **kwargs)
        File "/var/lib/juju/agents/unit-gnocchi-0/.venv/lib/python3.8/site-packages/jinja2/environment.py", line 1008, in render
          return self.environment.handle_exception(exc_info, True)
        File "/var/lib/juju/agents/unit-gnocchi-0/.venv/lib/python3.8/site-packages/jinja2/environment.py", line 780, in handle_exception
          reraise(exc_type, exc_value, tb)
        File "/var/lib/juju/agents/unit-gnocchi-0/.venv/lib/python3.8/site-packages/jinja2/_compat.py", line 37, in reraise
          raise value.with_traceback(tb)
        File "templates/queens/gnocchi.conf", line 29, in <module>
          {% if options.storage_backend == 'ceph' and storage_ceph.key -%}
        File "/var/lib/juju/agents/unit-gnocchi-0/.venv/lib/python3.8/site-packages/jinja2/environment.py", line 430, in getattr
          return getattr(obj, attribute)
      jinja2.exceptions.UndefinedError: 'storage_ceph' is undefined

The reason is related to this code in charms.openstack: (charms_openstack/charm/core.py +896):

    def render_configs(self, configs, adapters_instance=None):
        """Render the configuration files identified in the list passed as
        configs.

        Configs may not only be loaded via OpenStack loaders but also via
        string templates passed via config options or from relation data.
        This must be explicitly declared via string_templates dict of a given
        derived charm class by using a relation name that identifies a relation
        adapter or config option adapter and a property to be used from that
        adapter instance.

        :param configs: list of strings, the names of the configuration files.
        :param adapters_instance: [optional] the adapters_instance to use.
        """
        if adapters_instance is None:
            interfaces = []
            for f in flags.get_flags():
                ep_from_f = relations.endpoint_from_flag(f)
                if ep_from_f:
                    interfaces.append(ep_from_f)
            try:
                adapters_instance = self.adapters_class(interfaces,
                                                        charm_instance=self)
            except TypeError:
                adapters_instance = self.adapters_class(interfaces)

During an action, the adapters_interface passed to the function is None AND the flags.get_flags() doesn't result in any interfaces. Therefore, no context gets passed to the charm-helpers renderer and thus the rendering fails.

A workaround is to NOT use the action managed 'openstack-upgrade' function, and instead, use the config option with "action-managed-upgrade" set to False. This will kick the upgrade during a config-changed hook which WILL have the available context.

Temporary fix is to remove the action-managed-upgrade. Permanent fix is to probably override the render function and provide the adapters_instance directly.

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Part of the problem is in charms.openstack in the run_upgrade() function in core.py which tries to do the rendering (which is good), but fails completely if the rendering fails. This would mean that an upgrade can't proceed if, say, the charm is not related to the components that the config files require rendering; remember this is being called from an action and that means that it may not have the full context; c.f. with normally rendering, where the handlers will not run if the interfaces are not available, avoiding the whole "rendering" issue.

I think the right thing to do is to wrap the rendering and the upgrade in a try-except. This would allow the action to complete, but may result in the configs not being rendered and the db migration not happening. I think this should be gated on the interfaces_list being empty. i.e. don't worry about a failing render or db migration if the interfaces_list is empty as the unit may not actually be related to anything. If it is, then it's up to the charm to provide the interfaces list (and thus probably override "run_upgrade()" to provide the minimal list of interfaces if they are configured.

no longer affects: charms.openstack
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

The problem turns out not to be in charms.openstack (although it's all related), but in the layer-openstack-principle layer. The issue is that the storage-ceph interface has been switched over to an Endpoint class and these are not discovered at runtime unless the start-up script (in this case actions/openstack-upgrade) explicitly calls "hookenv._run_atstart()". The RelationBase classes can still be found without this as the value (a 'pointer' to the instance) is stored along with the flag and can thus be instantiated with "from_flag". This isn't possible with endpoints, and so the discovery call (_run_atstart()) needs to be performed first.

This is necessary for any action where the endpoint data is needed as part of the action code. This bug will only address the openstack-upgrade action.

Changed in layer-openstack-principle:
assignee: nobody → Alex Kavanagh (ajkavanagh)
Changed in charm-gnocchi:
assignee: nobody → Alex Kavanagh (ajkavanagh)
Changed in layer-openstack-principle:
status: New → In Progress
Changed in charm-gnocchi:
status: New → In Progress
Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

The review (https://review.opendev.org/c/openstack/charm-layer-openstack-principle/+/767522) fixes the problem in layer-openstack-principle

Changed in charm-gnocchi:
milestone: none → 21.01
Changed in layer-openstack-principle:
milestone: none → 21.01
Changed in charm-gnocchi:
status: In Progress → Fix Committed
Changed in layer-openstack-principle:
status: In Progress → Fix Committed
Changed in layer-openstack-principle:
assignee: Alex Kavanagh (ajkavanagh) → nobody
Changed in charm-gnocchi:
assignee: Alex Kavanagh (ajkavanagh) → nobody
David Ames (thedac)
Changed in charm-gnocchi:
status: Fix Committed → Fix Released
Changed in layer-openstack-principle:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.