OpenStack Octavia Charm

configure-resources actions ends in error

Bug #1892063 reported by Marian Gasparovic
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Octavia Charm
New
Undecided
Unassigned

Bug Description

Post deploy configure-resources action failed. It passed on at least two other runs.

2020-08-18-13:14:31 root DEBUG [localhost]: juju run-action -m foundations-maas:openstack --format=yaml octavia/leader configure-resources
2020-08-18-13:14:32 root DEBUG [localhost]: juju show-action-output -m foundations-maas:openstack --format=yaml --wait 600 34
2020-08-18-13:15:26 root DEBUG action output: {'UnitId': 'octavia/2', 'id': '34', 'message': "Unbable to set quotas to unlimited: ('nova', 'quotas', BadRequest())", 'results': {'Stderr': 'Warning: Stopping systemd-networkd.service, but it can still be activated by:\n systemd-networkd.socket\n', 'Stdout': 'active\nactive\nactive\nactive\n'}, 'status': 'failed', 'timing': {'completed': '2020-08-18 13:15:25 +0000 UTC', 'enqueued': '2020-08-18 13:14:32 +0000 UTC', 'started': '2020-08-18 13:14:32 +0000 UTC'}}
2020-08-18-13:15:26 root ERROR action output: {'UnitId': 'octavia/2', 'id': '34', 'message': "Unbable to set quotas to unlimited: ('nova', 'quotas', BadRequest())", 'results': {'Stderr': 'Warning: Stopping systemd-networkd.service, but it can still be activated by:\n systemd-networkd.socket\n', 'Stdout': 'active\nactive\nactive\nactive\n'}, 'status': 'failed', 'timing': {'completed': '2020-08-18 13:15:25 +0000 UTC', 'enqueued': '2020-08-18 13:14:32 +0000 UTC', 'started': '2020-08-18 13:14:32 +0000 UTC'}}

Tags: cdo-qa
Revision history for this message
Marian Gasparovic (marosg) wrote :
Revision history for this message
Marian Gasparovic (marosg) wrote :
Revision history for this message
Michael Skalka (mskalka) wrote :

Seen again in this release test run: https://solutions.qa.canonical.com/testruns/testRun/cefecb46-4fcb-439f-ab43-3c15c1928b28

Crashdump here: https://oil-jenkins.canonical.com/artifacts/cefecb46-4fcb-439f-ab43-3c15c1928b28/generated/generated/openstack/juju-crashdump-openstack-2020-10-10-06.21.08.tar.gz

tags: added: cdo-release-blocker
Michael Skalka (mskalka)
tags: removed: cdo-release-blocker
Revision history for this message
Bas de Bruijne (basdbruijne) wrote :
Download full text (3.2 KiB)

We are seeing this again here: https://solutions.qa.canonical.com/v2/testruns/c26f6331-7387-4938-8921-9c09302d8087/

Lots of python errors in the crashdump, e.g.:
```
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils Traceback (most recent call last):
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils File "/usr/lib/python3/dist-packages/cotyledon/_utils.py", line 95, in exit_on_exception
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils yield
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils File "/usr/lib/python3/dist-packages/cotyledon/_service.py", line 139, in _run
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils self.run()
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils File "/usr/lib/python3/dist-packages/octavia/controller/queue/v1/consumer.py", line 47, in run
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils self.message_listener.start()
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 267, in wrapper
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils states[state].run_once(lambda: fn(self, *args, **kwargs),
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 188, in run_once
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils post_fn = fn()
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 267, in <lambda>
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils states[state].run_once(lambda: fn(self, *args, **kwargs),
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils File "/usr/lib/python3/dist-packages/oslo_messaging/server.py", line 413, in start
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils self.listener = self._create_listener()
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 150, in _create_listener
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils return self.transport._listen(self._target, 1, None)
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 139, in _listen
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils raise exceptions.InvalidTarget('A server\'s target must have '
octavia/octavia-worker.log:2022-10-25 13:04:32.622 72424 ERROR cotyledon._utils oslo_messaging.exceptions.InvalidTarget: A server's target must have topic and server names specified:<Target server=juju-c4a296-4-lxd-9>
```

Logs can be found here:
https://oil-jenkins.canonical.com/artifacts/c26f6331-7...

Read more...

Revision history for this message
Felipe Reyes (freyes) wrote :
Download full text (6.6 KiB)

I was taking a look to the logs available at https://oil-jenkins.canonical.com/artifacts/c26f6331-7387-4938-8921-9c09302d8087/generated/generated/openstack/juju-crashdump-openstack-2022-10-25-14.15.19.tar.gz , and the error posted is misleading, this is being printed before config-changed has generated the configuration, after those errors it can be seen how octavia-worker started running successfully

2022-10-25 13:06:54.321 96392 ERROR cotyledon._utils oslo_messaging.exceptions.InvalidTarget: A server's target must have topic and server names specified:<Target server=juju-c4a296-3-lxd-8>
2022-10-25 13:06:54.321 96392 ERROR cotyledon._utils
2022-10-25 13:07:15.463 96579 INFO octavia.common.config [-] Logging enabled!
2022-10-25 13:07:15.463 96579 INFO octavia.common.config [-] /usr/bin/octavia-worker version 10.0.0
2022-10-25 13:07:15.543 97301 INFO octavia.controller.queue.v1.consumer [-] Starting consumer...
2022-10-25 13:07:15.556 97304 INFO octavia.controller.queue.v2.consumer [-] Starting V2 consumer...
2022-10-25 13:07:26.841 96579 INFO cotyledon._service_manager [-] Caught SIGTERM signal, graceful exiting of master process
2022-10-25 13:07:26.842 97301 INFO cotyledon._service [-] Caught SIGTERM signal, graceful exiting of service ConsumerService(0) [97301]
2022-10-25 13:07:26.842 97301 INFO octavia.controller.queue.v1.consumer [-] Stopping consumer...
2022-10-25 13:07:26.843 97304 INFO cotyledon._service [-] Caught SIGTERM signal, graceful exiting of service ConsumerService(0) [97304]
2022-10-25 13:07:26.844 97304 INFO octavia.controller.queue.v2.consumer [-] Stopping V2 consumer...
2022-10-25 13:07:26.908 97304 INFO octavia.controller.queue.v2.consumer [-] V2 Consumer successfully stopped. Waiting for final messages to be processed...
2022-10-25 13:07:26.919 97304 WARNING amqp [-] Received method (60, 30) during closing channel 1. This method will be ignored
2022-10-25 13:07:26.922 97304 INFO octavia.controller.queue.v2.consumer [-] Shutting down V2 endpoint worker executors...
2022-10-25 13:07:26.928 97301 INFO octavia.controller.queue.v1.consumer [-] Consumer successfully stopped. Waiting for final messages to be processed...
2022-10-25 13:07:26.937 97301 WARNING amqp [-] Received method (60, 30) during closing channel 1. This method will be ignored
2022-10-25 13:07:26.939 97301 INFO octavia.controller.queue.v1.consumer [-] Shutting down endpoint worker executors...
2022-10-25 13:07:33.040 97444 INFO octavia.common.config [-] Logging enabled!
2022-10-25 13:07:33.041 97444 INFO octavia.common.config [-] /usr/bin/octavia-worker version 10.0.0
2022-10-25 13:07:33.209 97654 INFO octavia.controller.queue.v1.consumer [-] Starting consumer...
2022-10-25 13:07:33.230 97662 INFO octavia.controller.queue.v2.consumer [-] Starting V2 consumer...
2022-10-25 13:09:01.537 97444 INFO cotyledon._service_manager [-] Caught SIGTERM signal, graceful exiting of master process
2022-10-25 13:09:01.538 97654 INFO cotyledon._service [-] Caught SIGTERM signal, graceful exiting of service ConsumerService(0) [97654]
2022-10-25 13:09:01.538 97662 INFO cotyledon._service [-] Caught SIGTERM signal, graceful exiting of service ConsumerService(0) [97662]
2022...

Read more...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.