OpenStack Ceilometer Charm

mojo Build #23040 specs/full_stack/next_designate_ha xenial-pike: upgrade action failed

Bug #1886968 reported by Alex Kavanagh on 2020-07-09

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Ceilometer Charm	Confirmed	Undecided	Unassigned

Bug Description

11:44:56 #############################################################################
11:44:56 Setup ceilometer
11:44:56 #############################################################################
11:44:56
11:44:56
11:44:56 2020-07-06 11:44:56 [INFO] Pulling secrets from /tmp/tmp.Q5zfnXUUye/LOCAL/mojo-openstack-specs/specs/full_stack/next_designate_ha/pike to /tmp/tmp.Q5zfnXUUye/mojo-openstack-specs/xenial/osci-mojo/local
11:44:56 2020-07-06 11:44:56 [WARNING] Automatic secrets phase ran but secrets directory /tmp/tmp.Q5zfnXUUye/LOCAL/mojo-openstack-specs/specs/full_stack/next_designate_ha/pike does not exist!
11:44:56 2020-07-06 11:44:56 [INFO] Running script ceilometer_setup.py
11:45:16 2020-07-06 11:45:16 [WARNING] No debug log matching debug-logs found. Using default.
11:45:19 2020-07-06 11:45:19 [ERROR] 2020-07-06 11:44:57 [INFO] Setting up ceilometer schema
11:45:19 Traceback (most recent call last):
11:45:19 File "/tmp/tmp.Q5zfnXUUye/mojo-openstack-specs/xenial/osci-mojo/spec/specs/full_stack/next_designate_ha/pike/ceilometer_setup.py", line 23, in <module>
11:45:19 sys.exit(main(sys.argv))
11:45:19 File "/tmp/tmp.Q5zfnXUUye/mojo-openstack-specs/xenial/osci-mojo/spec/specs/full_stack/next_designate_ha/pike/ceilometer_setup.py", line 19, in main
11:45:19 action.status))
11:45:19 Exception: ceilometer-upgrade action returned failed status

Revision history for this message

Aurelien Lourot (aurelien-lourot) wrote on 2020-07-28:

Happened again here: http://osci:8080/view/MojoMatrix/job/mojo_runner/23243/console

Changed in charm-ceilometer:
status:	New → Confirmed

Revision history for this message

Aurelien Lourot (aurelien-lourot) wrote on 2020-10-02:

Download full text (4.3 KiB)

Happened again here: http://osci:8080/job/mojo_runner/23871/console

2020-09-30 05:02:17 DEBUG ceilometer-upgrade none
2020-09-30 05:02:17 DEBUG ceilometer-upgrade none
2020-09-30 05:02:18 DEBUG juju-log Running ceilometer-upgrade: ceilometer-upgrade --debug
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.541 11628 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.543 11628 DEBUG ceilometer.cmd.storage [-] Upgrading metering database upgrade /usr/lib/python2.7/dist-packages/ceilometer/cmd/storage.py:48
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.543 11628 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.metering.storage' get_connection /usr/lib/python2.7/dist-packages/ceilometer/storage/__init__.py:87
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.588 11628 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('172.17.107.28', 27017)]
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.656 11628 DEBUG ceilometer.cmd.storage [-] Upgrading Gnocchi resource types upgrade /usr/lib/python2.7/dist-packages/ceilometer/cmd/storage.py:57
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 CRITICAL ceilometer [-] Unhandled error: ClientException: {"message": "The server is currently unavailable. Please try again at a later time. \n\n\n", "code": "503 Service Unavailable", "title": "Service Unavailable"} (HTTP 503)
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer Traceback (most recent call last):
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/bin/ceilometer-upgrade", line 10, in <module>
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer sys.exit(upgrade())
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/cmd/storage.py", line 59, in upgrade
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer gnocchi_client.upgrade_resource_types(conf)
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/gnocchi_client.py", line 194, in upgrade_resource_types
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer gnocchi.resource_type.get(name=name)
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/gnocchiclient/v1/resource_type.py", line 44, in get
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer headers={'Content-Type': "application/json"}).json()
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File ...

Happened again here: http://osci:8080/job/mojo_runner/23871/console

2020-09-30 05:02:17 DEBUG ceilometer-upgrade none
2020-09-30 05:02:17 DEBUG ceilometer-upgrade none
2020-09-30 05:02:18 DEBUG juju-log Running ceilometer-upgrade: ceilometer-upgrade --debug
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.541 11628 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.543 11628 DEBUG ceilometer.cmd.storage [-] Upgrading metering database upgrade /usr/lib/python2.7/dist-packages/ceilometer/cmd/storage.py:48
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.543 11628 DEBUG ceilometer.storage [-] looking for 'mongodb' driver in 'ceilometer.metering.storage' get_connection /usr/lib/python2.7/dist-packages/ceilometer/storage/__init__.py:87
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.588 11628 INFO ceilometer.storage.mongo.utils [-] Connecting to mongodb on [('172.17.107.28', 27017)]
2020-09-30 05:02:20 DEBUG ceilometer-upgrade 2020-09-30 05:02:20.656 11628 DEBUG ceilometer.cmd.storage [-] Upgrading Gnocchi resource types upgrade /usr/lib/python2.7/dist-packages/ceilometer/cmd/storage.py:57
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 CRITICAL ceilometer [-] Unhandled error: ClientException: {"message": "The server is currently unavailable. Please try again at a later time. \n\n\n", "code": "503 Service Unavailable", "title": "Service Unavailable"} (HTTP 503)
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer Traceback (most recent call last):
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/bin/ceilometer-upgrade", line 10, in <module>
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer sys.exit(upgrade())
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/cmd/storage.py", line 59, in upgrade
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer gnocchi_client.upgrade_resource_types(conf)
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/ceilometer/gnocchi_client.py", line 194, in upgrade_resource_types
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer gnocchi.resource_type.get(name=name)
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/gnocchiclient/v1/resource_type.py", line 44, in get
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer headers={'Content-Type': "application/json"}).json()
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/gnocchiclient/v1/base.py", line 37, in _get
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer return self.client.api.get(*args, **kwargs)
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 288, in get
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer return self.request(url, 'GET', **kwargs)
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer File "/usr/lib/python2.7/dist-packages/gnocchiclient/client.py", line 38, in request
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer raise exceptions.from_response(resp, method)
2020-09-30 05:03:05 DEBUG ceilometer-upgrade 2020-09-30 05:03:05.118 11628 ERROR ceilometer ClientException: {"message": "The server is currently unavailable. Please try again at a later time. \n\n\n", "code": "503 Service Unavailable", "title": "Service Unavailable"} (HTTP 503)

So it looks like Gnocchi is giving a 503 to ceilometer although it advertises as ready.

Revision history for this message

Aurelien Lourot (aurelien-lourot) wrote on 2020-10-02:

On gnocchi's side we see

2020-09-30 05:03:05.210 30713 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Unable to establish connection to http://172.17.107.232:35357/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",)): ConnectFailure: Unable to establish connection to http://172.17.107.232:35357/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",))

172.17.107.232 being keystone's VIP. I wonder if we should go for a tenacity/retry mechanism somewhere?

Revision history for this message

Alex Kavanagh (ajkavanagh) wrote on 2020-10-05:

So it looks like keystone was 'away' when these occurred. I can't tell from the logs; is this an "unstable-test" or causing gnocchi to die due to a charm bug in gnocchi?

Revision history for this message

Aurelien Lourot (aurelien-lourot) wrote on 2020-10-29:

Seen on xenial-queens as well: http://osci:8080/job/mojo_runner/24121/console

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.