| 2022-05-17 16:27:08 |
Peter Matulis |
description |
While attempting to validate the openstack-telemetry bundle for focal-yoga I ran
juju run-action --wait ceilometer/0 ceilometer-upgrade
Doing so led to failure:
unit-ceilometer-0:
UnitId: ceilometer/0
id: "10"
message: 'ceilometer-upgrade resulted in an unexpected error: Command ''[''ceilometer-upgrade'',
''--debug'', ''--retry'', ''10'']'' returned non-zero exit status 1.'
results:
Stdout: "2022-05-16 22:06:54.760 45489 DEBUG ceilometer.cmd.storage [-] Upgrading
Gnocchi resource types upgrade /usr/lib/python3/dist-packages/ceilometer/cmd/storage.py:42\e[00m\n2022-05-16
22:06:59.706 45489 CRITICAL ceilometer [-] Unhandled error: gnocchiclient.exceptions.ClientException:
{\"message\": \"The server is currently unavailable. Please try again at a later
time.<br /><br />\\nThe Keystone service is temporarily unavailable.\\n\\n\",
\"code\": \"503 Service Unavailable\", \"title\": \"Service Unavailable\"} (HTTP
503)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer Traceback (most recent
call last):\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/bin/ceilometer-upgrade\",
line 10, in <module>\n2022-05-16 22:06:59.706 45489 ERROR ceilometer sys.exit(upgrade())\n2022-05-16
22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/ceilometer/cmd/storage.py\",
line 49, in upgrade\n2022-05-16 22:06:59.706 45489 ERROR ceilometer tenacity.Retrying(\n2022-05-16
22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/tenacity/__init__.py\",
line 423, in __call__\n2022-05-16 22:06:59.706 45489 ERROR ceilometer do
= self.iter(retry_state=retry_state)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer
\ File \"/usr/lib/python3/dist-packages/tenacity/__init__.py\", line 360, in
iter\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return fut.result()\n2022-05-16
22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3.8/concurrent/futures/_base.py\",
line 437, in result\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return
self.__get_result()\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3.8/concurrent/futures/_base.py\",
line 389, in __get_result\n2022-05-16 22:06:59.706 45489 ERROR ceilometer raise
self._exception\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/tenacity/__init__.py\",
line 426, in __call__\n2022-05-16 22:06:59.706 45489 ERROR ceilometer result
= fn(*args, **kwargs)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File
\"/usr/lib/python3/dist-packages/ceilometer/gnocchi_client.py\", line 266, in
upgrade_resource_types\n2022-05-16 22:06:59.706 45489 ERROR ceilometer gnocchi.resource_type.get(name=name)\n2022-05-16
22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/gnocchiclient/v1/resource_type.py\",
line 43, in get\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return self._get(self.url
+ name,\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/gnocchiclient/v1/base.py\",
line 37, in _get\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return
self.client.api.get(*args, **kwargs)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer
\ File \"/usr/lib/python3/dist-packages/keystoneauth1/adapter.py\", line 395,
in get\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return self.request(url,
'GET', **kwargs)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/gnocchiclient/client.py\",
line 52, in request\n2022-05-16 22:06:59.706 45489 ERROR ceilometer raise
exceptions.from_response(resp, method)\n2022-05-16 22:06:59.706 45489 ERROR
ceilometer gnocchiclient.exceptions.ClientException: {\"message\": \"The server
is currently unavailable. Please try again at a later time.<br /><br />\\nThe
Keystone service is temporarily unavailable.\\n\\n\", \"code\": \"503 Service
Unavailable\", \"title\": \"Service Unavailable\"} (HTTP 503)\n2022-05-16 22:06:59.706
45489 ERROR ceilometer \e[00m\n"
outcome: ceilometer-upgrade failed, see traceback.
traceback: |
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-ceilometer-0/charm/actions/ceilometer_utils.py", line 669, in ceilometer_upgrade_helper
ceilometer_upgrade(action=True)
File "/var/lib/juju/agents/unit-ceilometer-0/charm/actions/ceilometer_utils.py", line 691, in ceilometer_upgrade
subprocess.check_call(cmd)
File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ceilometer-upgrade', '--debug', '--retry', '10']' returned non-zero exit status 1.
status: failed
There appears to be a Keystone issue. I verified that I could create and connect to an instance (via SSH).
A juju crashdump file is attached. |
While attempting to validate the openstack-telemetry bundle for focal-yoga I ran
juju run-action --wait ceilometer/0 ceilometer-upgrade
Doing so led to failure:
unit-ceilometer-0:
UnitId: ceilometer/0
id: "10"
message: 'ceilometer-upgrade resulted in an unexpected error: Command ''[''ceilometer-upgrade'',
''--debug'', ''--retry'', ''10'']'' returned non-zero exit status 1.'
results:
Stdout: "2022-05-16 22:06:54.760 45489 DEBUG ceilometer.cmd.storage [-] Upgrading
Gnocchi resource types upgrade /usr/lib/python3/dist-packages/ceilometer/cmd/storage.py:42\e[00m\n2022-05-16
22:06:59.706 45489 CRITICAL ceilometer [-] Unhandled error: gnocchiclient.exceptions.ClientException:
{\"message\": \"The server is currently unavailable. Please try again at a later
time.<br /><br />\\nThe Keystone service is temporarily unavailable.\\n\\n\",
\"code\": \"503 Service Unavailable\", \"title\": \"Service Unavailable\"} (HTTP
503)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer Traceback (most recent
call last):\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/bin/ceilometer-upgrade\",
line 10, in <module>\n2022-05-16 22:06:59.706 45489 ERROR ceilometer sys.exit(upgrade())\n2022-05-16
22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/ceilometer/cmd/storage.py\",
line 49, in upgrade\n2022-05-16 22:06:59.706 45489 ERROR ceilometer tenacity.Retrying(\n2022-05-16
22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/tenacity/__init__.py\",
line 423, in __call__\n2022-05-16 22:06:59.706 45489 ERROR ceilometer do
= self.iter(retry_state=retry_state)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer
\ File \"/usr/lib/python3/dist-packages/tenacity/__init__.py\", line 360, in
iter\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return fut.result()\n2022-05-16
22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3.8/concurrent/futures/_base.py\",
line 437, in result\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return
self.__get_result()\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3.8/concurrent/futures/_base.py\",
line 389, in __get_result\n2022-05-16 22:06:59.706 45489 ERROR ceilometer raise
self._exception\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/tenacity/__init__.py\",
line 426, in __call__\n2022-05-16 22:06:59.706 45489 ERROR ceilometer result
= fn(*args, **kwargs)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File
\"/usr/lib/python3/dist-packages/ceilometer/gnocchi_client.py\", line 266, in
upgrade_resource_types\n2022-05-16 22:06:59.706 45489 ERROR ceilometer gnocchi.resource_type.get(name=name)\n2022-05-16
22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/gnocchiclient/v1/resource_type.py\",
line 43, in get\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return self._get(self.url
+ name,\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/gnocchiclient/v1/base.py\",
line 37, in _get\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return
self.client.api.get(*args, **kwargs)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer
\ File \"/usr/lib/python3/dist-packages/keystoneauth1/adapter.py\", line 395,
in get\n2022-05-16 22:06:59.706 45489 ERROR ceilometer return self.request(url,
'GET', **kwargs)\n2022-05-16 22:06:59.706 45489 ERROR ceilometer File \"/usr/lib/python3/dist-packages/gnocchiclient/client.py\",
line 52, in request\n2022-05-16 22:06:59.706 45489 ERROR ceilometer raise
exceptions.from_response(resp, method)\n2022-05-16 22:06:59.706 45489 ERROR
ceilometer gnocchiclient.exceptions.ClientException: {\"message\": \"The server
is currently unavailable. Please try again at a later time.<br /><br />\\nThe
Keystone service is temporarily unavailable.\\n\\n\", \"code\": \"503 Service
Unavailable\", \"title\": \"Service Unavailable\"} (HTTP 503)\n2022-05-16 22:06:59.706
45489 ERROR ceilometer \e[00m\n"
outcome: ceilometer-upgrade failed, see traceback.
traceback: |
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-ceilometer-0/charm/actions/ceilometer_utils.py", line 669, in ceilometer_upgrade_helper
ceilometer_upgrade(action=True)
File "/var/lib/juju/agents/unit-ceilometer-0/charm/actions/ceilometer_utils.py", line 691, in ceilometer_upgrade
subprocess.check_call(cmd)
File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ceilometer-upgrade', '--debug', '--retry', '10']' returned non-zero exit status 1.
status: failed
There appears to be a Keystone + SSL issue. I verified that I could create and connect to an instance (via SSH).
A juju crashdump file is attached. |
|
