As seen in testrun: https://oil-jenkins.canonical.com/job/fce_openstackupgrade/41//console
Openstack upgrade ussuri to victoria of keystone fails with:
------------------------------------------------------------------------------
zaza.model.ActionFailed: Run of action "openstack-upgrade" with parameters "<not-set>" on "keystone/1" failed with "exit status 1" (id=112 status=failed enqueued=2022-02-04T04:13:50Z started=2022-02-04T04:13:50Z completed=2022-02-04T04:15:39Z output={'Code': '1', 'Stderr': 'ERROR no relation id specified
Failed to start apache2.service: Unit apache2.service is masked.
Unit /etc/systemd/system/keystone.service is masked, ignoring.
ERROR no relation id specified
Traceback (most recent call last):
File "./hooks/config-changed-postupgrade", line 937, in <module>
main()
File "./hooks/config-changed-postupgrade", line 930, in main
hooks.execute(sys.argv)
File "/var/lib/juju/agents/unit-keystone-1/charm/charmhelpers/core/hookenv.py", line 962, in execute
self._hooks[hook_name]()
File "/var/lib/juju/agents/unit-keystone-1/charm/charmhelpers/contrib/openstack/utils.py", line 1857, in wrapped_f
return f(*args, **kwargs)
File "/var/lib/juju/agents/unit-keystone-1/charm/charmhelpers/contrib/hardening/harden.py", line 93, in _harden_inner2
return f(*args, **kwargs)
File "./hooks/config-changed-postupgrade", line 298, in config_changed_postupgrade
update_all_domain_backends()
File "./hooks/config-changed-postupgrade", line 354, in update_all_domain_backends
domain_backend_changed(relation_id=rid, unit=unit)
File "./hooks/config-changed-postupgrade", line 661, in domain_backend_changed
create_or_show_domain(domain_name)
File "/var/lib/juju/agents/unit-keystone-1/charm/hooks/keystone_utils.py", line 1122, in create_or_show_domain
domain_id = manager.resolve_domain_id(name)
File "/var/lib/juju/agents/unit-keystone-1/charm/hooks/keystone_utils.py", line 1226, in __call__
return _proxy_manager_call(self._path, self.api_version, args, kwargs)
File "/var/lib/juju/agents/unit-keystone-1/charm/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
return f(*args, **kwargs)
File "/var/lib/juju/agents/unit-keystone-1/charm/hooks/keystone_utils.py", line 1270, in _proxy_manager_call
raise e
File "/var/lib/juju/agents/unit-keystone-1/charm/hooks/keystone_utils.py", line 1264, in _proxy_manager_call
raise RuntimeError(s)
RuntimeError: The call within manager.py failed with the error: \'Unable to establish connection to http://localhost:35337/v3/auth/tokens: HTTPConnectionPool(host=\'localhost\', port=35337): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7fbd735d4880>: Failed to establish a new connection: [Errno 111] Connection refused\'))\'. The call was: path=[\'resolve_domain_id\'], args=(\'cdoqa\',), kwargs={}, api_version=None
------------------------------------------------------------------------------
Crashdump collection for openstack upgrade is still in the works, but information about the deployment can be found here:
https://oil-jenkins.canonical.com/artifacts/586ad7a0-b732-47a6-9fd8-980157776299/index.html
The latest occurrence has log collecting working:
https:/
We have crashdumps from before and after the upgrade, so look at the newer ones.
The openstack-upgrade action on keystone/0 gives a confusing output:
output={'Code': '1', 'Stderr': <big error in attachment>, 'outcome': 'success, upgrade completed.'})
Unit keystone/0 is not in error state after the upgrade, so maybe its just wrong reporting of the upgrade action.