In test run https://solutions.qa.canonical.com/testruns/10057975-86d9-4c24-b124-ee6934f077a6/, the installation of mysql-innodb-cluster fails no the command leader-get. In the crashdump we see:
========
2023-10-11 02:20:28 INFO unit.mysql-innodb-cluster/1.juju-log server.go:325 Reactive main running for hook install
2023-10-11 02:20:28 WARNING unit.mysql-innodb-cluster/1.install logger.go:60 ERROR cannot read leadership settings: cannot read settings: failed to call leadership api: connection is shut down
2023-10-11 02:20:28 ERROR unit.mysql-innodb-cluster/1.juju-log server.go:325 Hook error:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 71, in main
bus.discover()
File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 426, in discover
_register_handlers_from_file(search_path, filepath)
File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 466, in _register_handlers_from_file
_load_module(root, filepath)
File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 444, in _load_module
return importlib.import_module(package + module)
File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/charm/reactive/coordinator.py", line 18, in <module>
from charms.coordinator import coordinator, log
File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/charm/lib/charms/coordinator.py", line 144, in <module>
coordinator = _instantiate()
File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/charm/lib/charms/coordinator.py", line 137, in _instantiate
return class_(peer_relation_name='coordinator')
File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.10/site-packages/charmhelpers/coordinator.py", line 240, in __call__
cls._instances[cls] = super(Singleton, cls).__call__(*args,
File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/charm/lib/charm/openstack/delayed_action_coordinator.py", line 41, in __init__
self.get_released_locks_counts_from_leader_settings())
File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/charm/lib/charm/openstack/delayed_action_coordinator.py", line 50, in get_released_locks_counts_from_leader_settings
hookenv.leader_get(self.released_locks_count_key) or
File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.10/site-packages/charmhelpers/core/hookenv.py", line 1180, in inner_translate_exc2
return f(*args, **kwargs)
File "/var/lib/juju/agents/unit-mysql-innodb-cluster-1/.venv/lib/python3.10/site-packages/charmhelpers/core/hookenv.py", line 1225, in leader_get
return json.loads(subprocess.check_output(cmd).decode('UTF-8'))
File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['leader-get', '--format=json', 'coordinator-released-locks-counts']' returned non-zero exit status 1.
========
I believe we have a different bug open with the same "connection is shut down" message. Maybe we should add a retry there?
Configs and crashdumps can be found here: https://oil-jenkins.canonical.com/artifacts/10057975-86d9-4c24-b124-ee6934f077a6/index.html
