OpenStack Octavia Charm

Tempest Octavia test timeout: load balancer stays offline

Bug #1953639 reported by Bas de Bruijne
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Octavia Charm
New
Undecided
Unassigned

Bug Description

Tempest test test_load_balancer_show_status fails with:

--------------------------------------------------------
Traceback (most recent call last):
  File "/snap/fcbtest/20/lib/python3.6/site-packages/octavia_tempest_plugin/tests/api/v2/test_load_balancer.py", line 844, in test_load_balancer_show_status
    CONF.load_balancer.check_timeout)
  File "/snap/fcbtest/20/lib/python3.6/site-packages/octavia_tempest_plugin/tests/waiters.py", line 96, in wait_for_status
    raise exceptions.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: (LoadBalancerAPITest:test_load_balancer_show_status) show_loadbalancer operating_status failed to update to ONLINE within the required time 300. Current status of show_loadbalancer: OFFLINE
--------------------------------------------------------

Sometimes this is fixed by increasing the ha timeout values, but this does not always work. This bug does not occur consistently.

Crashdump and artifacts:
https://oil-jenkins.canonical.com/artifacts/85c72c8d-05cc-4dc0-9f34-ce4f0188efcc/index.html
All occurances of this bug (crashdumps can be found under testrun id -> view full artifacts (bottom of the page)):
https://solutions.qa.canonical.com/bugs/bugs/bug/1946738
Edit: the new occurances are listed here:
https://solutions.qa.canonical.com/bugs/bugs/bug/1953639

See original description
Bas de Bruijne (basdbruijne)
description: updated
Revision history for this message
Bas de Bruijne (basdbruijne) wrote :

This is blocking all the bionic testruns and too many focal ones for SQA, so we will remove this test from the list until the bug is fixed.

Revision history for this message
Nobuto Murata (nobuto) wrote :

I've downloaded the following crashdump:
https://oil-jenkins.canonical.com/artifacts/9eae3037-69df-4413-801a-4c2e077f5c72/generated/generated/openstack/juju-crashdump-openstack-2022-02-11-03.06.01.tar.gz

And confirmed that octavia.amphorae.drivers.health.heartbeat_udp is using 127.0.0.1 (the default value if no config) instead of IPv6 address of o-hm0 prot.

I cannot be 100% sure since I couldn't find /etc/octavia/octavia.conf in the crashdump but this is likely the same with:
https://bugs.launchpad.net/charm-octavia/+bug/1961088

juju-crashdump-openstack-2022-02-11-03.06.01/4d84b389-7b51-4614-af77-a3150cd9ebe4$ tail -n5 octavia_*/var/log/octavia/octavia-health-manager.log
==> octavia_0/var/log/octavia/octavia-health-manager.log <==
2022-02-11 01:30:11.255 197854 INFO octavia.common.config [-] Logging enabled!
2022-02-11 01:30:11.255 197854 INFO octavia.common.config [-] /usr/bin/octavia-health-manager version 6.2.1
2022-02-11 01:30:11.384 197854 INFO octavia.cmd.health_manager [-] Health Manager listener process starts:
2022-02-11 01:30:11.387 197854 INFO octavia.cmd.health_manager [-] Health manager check process starts:
2022-02-11 01:30:11.389 197880 INFO octavia.amphorae.drivers.health.heartbeat_udp [-] attempting to listen on 127.0.0.1 port 5555

==> octavia_1/var/log/octavia/octavia-health-manager.log <==
2022-02-11 01:29:18.431 157382 INFO octavia.common.config [-] Logging enabled!
2022-02-11 01:29:18.432 157382 INFO octavia.common.config [-] /usr/bin/octavia-health-manager version 6.2.1
2022-02-11 01:29:18.557 157382 INFO octavia.cmd.health_manager [-] Health Manager listener process starts:
2022-02-11 01:29:18.560 157382 INFO octavia.cmd.health_manager [-] Health manager check process starts:
2022-02-11 01:29:18.561 157401 INFO octavia.amphorae.drivers.health.heartbeat_udp [-] attempting to listen on 127.0.0.1 port 5555

==> octavia_2/var/log/octavia/octavia-health-manager.log <==
2022-02-11 01:30:05.961 161287 INFO octavia.common.config [-] Logging enabled!
2022-02-11 01:30:05.961 161287 INFO octavia.common.config [-] /usr/bin/octavia-health-manager version 6.2.1
2022-02-11 01:30:06.094 161287 INFO octavia.cmd.health_manager [-] Health Manager listener process starts:
2022-02-11 01:30:06.096 161287 INFO octavia.cmd.health_manager [-] Health manager check process starts:
2022-02-11 01:30:06.099 161309 INFO octavia.amphorae.drivers.health.heartbeat_udp [-] attempting to listen on 127.0.0.1 port 5555

Billy Olsen (billy-olsen)
tags: added: cdo-qa
Bas de Bruijne (basdbruijne)
tags: added: cdo-tempest
Revision history for this message
Bas de Bruijne (basdbruijne) wrote (last edit ):

Most recent crashdumps available (ussuri-focal): https://oil-jenkins.canonical.com/artifacts/03e6c65a-8876-446b-9bbb-ce5e9e4884bb/index.html

I will see if I can get reproduce this on yoga focal too.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.