1/3 hacluster units is in "Resource: res_manila_share_manila_share not yet configured"

1) hacluster/0 has never lost leadership during its lifetime (leader-elected got executed once and only on that unit);

2) the hacluster interface is container-scoped, therefore, the fact that manila-ganesha/1 has not exposed resource configuration does not matter;

3) the "Configuring Resources" log message appeared after the second ha-relation-changed got logged

* hacluster/0: 05 Aug 2020 17:37:47Z juju-unit executing running ha-relation-changed hook

* manila-ganesha/0:

2020-08-05 17:42:32 INFO juju-log ha:42: Invoking reactive handler: reactive/manila_ganesha.py:108:cluster_connected

* hacluster/0:
05 Aug 2020 17:42:51Z juju-unit executing running ha-relation-changed hook
2020-08-05 17:43:07 DEBUG juju-log ha:42: Configuring Resources: # ...

Chronologically, hacluster/0 learned about the resources exposed by manila-ganesha/0 at 05 Aug 2020 17:42:51Z. There weren't any -changed events associated with that relation after that.

juju show-status-log hacluster/0
Time Type Status Message
05 Aug 2020 17:36:14Z juju-unit executing running leader-elected hook
05 Aug 2020 17:36:20Z juju-unit executing running config-changed hook
05 Aug 2020 17:36:22Z workload maintenance Setting up corosync
05 Aug 2020 17:37:08Z juju-unit executing running start hook
05 Aug 2020 17:37:19Z workload active Unit is ready and clustered
05 Aug 2020 17:37:26Z juju-unit executing running ha-relation-joined hook
05 Aug 2020 17:37:47Z juju-unit executing running ha-relation-changed hook
05 Aug 2020 17:38:07Z juju-unit executing running hanode-relation-joined hook
05 Aug 2020 17:38:27Z juju-unit executing running hanode-relation-changed hook
05 Aug 2020 17:38:40Z workload blocked Insufficient peer units for ha cluster (require 3)
05 Aug 2020 17:38:48Z juju-unit executing running hanode-relation-joined hook
05 Aug 2020 17:39:08Z juju-unit executing running hanode-relation-changed hook
05 Aug 2020 17:41:39Z juju-unit idle
05 Aug 2020 17:42:51Z juju-unit executing running ha-relation-changed hook
05 Aug 2020 17:43:26Z juju-unit idle

juju show-status-log hacluster/1 --days 2
Time Type Status Message
05 Aug 2020 17:35:02Z workload waiting waiting for machine
05 Aug 2020 17:35:02Z juju-unit allocating
05 Aug 2020 17:35:02Z workload waiting installing agent
05 Aug 2020 17:35:06Z workload waiting agent initializing
05 Aug 2020 17:35:19Z workload maintenance installing charm software
05 Aug 2020 17:35:19Z juju-unit executing running install hook
05 Aug 2020 17:35:20Z workload maintenance Installin...

The manila-share resource is only exposed by manila-ganesha/2, not manila-ganesha/0 which is the one the leader hacluster unit is taking the resource configuration data from - only a vip resource got exposed by manila-ganesha/0:

sudo juju-run -r 42 --remote-unit manila-ganesha/0 hacluster/0 'JUJU_HOOK_NAME=ha-relation-changed hooks/ha-relation-changed'

rlwrap telnet localhost 4444

ubuntu@juju-dedf8c-zaza-a6cee85c0b7f-11:~$ rlwrap telnet localhost 4444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
> /var/lib/juju/agents/unit-hacluster-0/charm/hooks/ha-relation-changed(290)ha_relation_changed()
-> if not get_corosync_conf():

(Pdb) l
285 def ha_relation_changed():
286 import rpdb
287 rpdb.set_trace()
288 # Check that we are related to a principle and that
289 # it has already provided the required corosync configuration
290 -> if not get_corosync_conf():
291 log('Unable to configure corosync right now, deferring configuration',
292 level=INFO)
293 return
294
295 if relation_ids('hanode'):

(Pdb) n
> /var/lib/juju/agents/unit-hacluster-0/charm/hooks/ha-relation-changed(295)ha_relation_changed()
-> if relation_ids('hanode'):
(Pdb) n
> /var/lib/juju/agents/unit-hacluster-0/charm/hooks/ha-relation-changed(296)ha_relation_changed()
-> log('Ready to form cluster - informing peers', level=DEBUG)
(Pdb) n
> /var/lib/juju/agents/unit-hacluster-0/charm/hooks/ha-relation-changed(297)ha_relation_changed()
-> relation_set(relation_id=relation_ids('hanode')[0], ready=True)
(Pdb) n
> /var/lib/juju/agents/unit-hacluster-0/charm/hooks/ha-relation-changed(305)ha_relation_changed()
-> if len(get_cluster_nodes()) < int(config('cluster_count')):
(Pdb) n
> /var/lib/juju/agents/unit-hacluster-0/charm/hooks/ha-relation-changed(310)ha_relation_changed()
-> relids = relation_ids('ha') or relation_ids('juju-info')
(Pdb) n
> /var/lib/juju/agents/unit-hacluster-0/charm/hooks/ha-relation-changed(311)ha_relation_changed()
-> if len(relids) == 1: # Should only ever be one of these
(Pdb) n
> /var/lib/juju/agents/unit-hacluster-0/charm/hooks/ha-relation-changed(313)ha_relation_changed()
-> relid = relids[0]
(Pdb) n
> /var/lib/juju/agents/unit-hacluster-0/charm/hooks/ha-relation-changed(314)ha_relation_changed()
-> units = related_units(relid)
(Pdb) n
> /var/lib/juju/agents/unit-hacluster-0/charm/hooks/ha-relation-changed(315)ha_relation_changed()
-> if len(units) < 1:

(Pdb) l
310 relids = relation_ids('ha') or relation_ids('juju-info')
311 if len(relids) == 1: # Should only ever be one of these
312 # Obtain relation information
313 relid = relids[0]
314 units = related_units(relid)
315 -> if len(units) < 1:
316 log('No principle unit found, deferring configuration',
317 level=INFO)
318 return
319
320 unit = units[0]
(Pdb) relid
'ha:42'
(Pdb) units
['manila-ganesha/0']

(Pdb) n
> /var/lib/juju/agents/unit-hacluster-0/charm/hooks/ha-relation-changed(320)ha_relation_changed()
-> unit = units[0]
(Pdb) n
> /var/lib/juju/agents/unit-hacluster-0/charm/hooks/ha-relation-chan...

juju show-status-log manila-ganesha/0 --days 2
...
05 Aug 2020 17:42:19Z juju-unit executing running ha-relation-joined hook
05 Aug 2020 17:42:40Z juju-unit executing running ha-relation-changed hook

$ juju ssh manila-ganesha/0 'grep "cluster_connected" /var/log/juju/unit-manila-ganesha-0.log'
tracer: ++ queue handler reactive/manila_ganesha.py:108:cluster_connected
2020-08-05 17:42:32 INFO juju-log ha:42: Invoking reactive handler: reactive/manila_ganesha.py:108:cluster_connected

$ juju ssh manila-ganesha/0 'grep "juju-log ha:42: tracer: set flag ha.available" /var/log/juju/unit-manila-ganesha-0.log'
2020-08-05 17:42:42 DEBUG juju-log ha:42: tracer: set flag ha.available

Two code-paths set relation data on the HA relation.

1) cluster_connected -> this_charm.configure_ha_resources -> configure_ha_resources -> hacluster.bind_resources
2) cluster_connected -> hacluster.manage_resources(crm)

I thought that the second call with a custom CRM object (where crm.resources == {}) might be a overriding the relation data but it turns out not to be the case base of "if v" here:

https://github.com/openstack/charm-interface-hacluster/blob/9ea447c296466ba9fdca1eb8e9752bbd6a75cc59/requires.py#L98
        relation_data = {
            'json_{}'.format(k): json.dumps(v, sort_keys=True)
            for k, v in crm.items() if v
        }

charms_openstack/charm/classes.py

    def configure_ha_resources(self, hacluster):
        """Inform the ha subordinate about each service it should manage. The
        child class specifies the services via self.ha_resources

        @param hacluster instance of interface class HAClusterRequires
        """
        RESOURCE_TYPES = {
            'vips': self._add_ha_vips_config,
            'haproxy': self._add_ha_haproxy_config,
            'dnsha': self._add_dnsha_config,
        }
        if self.ha_resources:
            for res_type in self.ha_resources:
                RESOURCE_TYPES[res_type](hacluster)
            hacluster.bind_resources(iface=self.config[IFACE_KEY]) # <-------- this

requires.py

    def manage_resources(self, crm): # <-------- this
        """
        Request for the hacluster to manage the resources defined in the
        crm object.

            res = CRM()
            res.primitive('res_neutron_haproxy', 'lsb:haproxy',
                          op='monitor interval="5s"')
            res.init_services('haproxy')
            res.clone('cl_nova_haproxy', 'res_neutron_haproxy')

            hacluster.manage_resources(crm)

        :param crm: CRM() instance - Config object for Pacemaker resources
        :returns: None
        """
        relation_data = {
            'json_{}'.format(k): json.dumps(v, sort_keys=True)
            for k, v in crm.items() if v
        }
        if data_changed('hacluster-manage_resources', relation_data):
            self.set_local(**relation_data)
            self.set_remote(**relation_data)

    def bind_resources(self, iface=None, mcastport=None):
        """Inform the ha subordinate about each service it should manage. The
        child class specifies the services via self.ha_resources

        :param iface: string - Network interface to bind to
        :param mcastport: int - Multicast port corosync should use for cluster
                                management traffic
        """
        if mcastport is None:
            mcastport = 4440
        resources_dict = self.get_local('resources')
        self.bind_on(iface=iface, mcastport=mcastport)
        if resources_dict:
            resources = relations.hacluster.common.CRM(**resources_dict) # <-------- this
            self.manage_resources(resources) # <-------- this

@reactive.when('ha.connected', 'ganesha-pool-configured',
        ...

https://review.opendev.org/#/c/743212 patch set 7 addresses the issue but there is a different problem blocking it.

The issue in question is https://bugs.launchpad.net/charm-manila-ganesha/+bug/1890401/