OpenStack Dashboard Charm

policy.d test is unstable

Bug #1882243 reported by Alex Kavanagh on 2020-06-05

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Dashboard Charm	Triaged	Medium	Unassigned

Bug Description

From: https://review.opendev.org/#/c/732673/
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-openstack-dashboard/732673/2/5962/test_charm_func_full_9103/func.txt

ERROR
2020-06-05 11:10:32 [INFO] ======================================================================
2020-06-05 11:10:32 [INFO] ERROR: test_003_test_overide_is_observed (zaza.openstack.charm_tests.openstack_dashboard.tests.OpenStackDashboardPolicydTests)
2020-06-05 11:10:32 [INFO] Test that the override is observed by the underlying service.
2020-06-05 11:10:32 [INFO] ----------------------------------------------------------------------
2020-06-05 11:10:32 [INFO] Traceback (most recent call last):
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 677, in urlopen
2020-06-05 11:10:32 [INFO] chunked=chunked,
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 426, in _make_request
2020-06-05 11:10:32 [INFO] six.raise_from(e, None)
2020-06-05 11:10:32 [INFO] File "<string>", line 3, in raise_from
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 421, in _make_request
2020-06-05 11:10:32 [INFO] httplib_response = conn.getresponse()
2020-06-05 11:10:32 [INFO] File "/usr/lib/python3.5/http/client.py", line 1225, in getresponse
2020-06-05 11:10:32 [INFO] response.begin()
2020-06-05 11:10:32 [INFO] File "/usr/lib/python3.5/http/client.py", line 307, in begin
2020-06-05 11:10:32 [INFO] version, status, reason = self._read_status()
2020-06-05 11:10:32 [INFO] File "/usr/lib/python3.5/http/client.py", line 276, in _read_status
2020-06-05 11:10:32 [INFO] raise RemoteDisconnected("Remote end closed connection without"
2020-06-05 11:10:32 [INFO] http.client.RemoteDisconnected: Remote end closed connection without response
2020-06-05 11:10:32 [INFO] During handling of the above exception, another exception occurred:
2020-06-05 11:10:32 [INFO] Traceback (most recent call last):
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/requests/adapters.py", line 449, in send
2020-06-05 11:10:32 [INFO] timeout=timeout
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 725, in urlopen
2020-06-05 11:10:32 [INFO] method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/urllib3/util/retry.py", line 403, in increment
2020-06-05 11:10:32 [INFO] raise six.reraise(type(error), error, _stacktrace)
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/urllib3/packages/six.py", line 734, in reraise
2020-06-05 11:10:32 [INFO] raise value.with_traceback(tb)
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 677, in urlopen
2020-06-05 11:10:32 [INFO] chunked=chunked,
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 426, in _make_request
2020-06-05 11:10:32 [INFO] six.raise_from(e, None)
2020-06-05 11:10:32 [INFO] File "<string>", line 3, in raise_from
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/urllib3/connectionpool.py", line 421, in _make_request
2020-06-05 11:10:32 [INFO] httplib_response = conn.getresponse()
2020-06-05 11:10:32 [INFO] File "/usr/lib/python3.5/http/client.py", line 1225, in getresponse
2020-06-05 11:10:32 [INFO] response.begin()
2020-06-05 11:10:32 [INFO] File "/usr/lib/python3.5/http/client.py", line 307, in begin
2020-06-05 11:10:32 [INFO] version, status, reason = self._read_status()
2020-06-05 11:10:32 [INFO] File "/usr/lib/python3.5/http/client.py", line 276, in _read_status
2020-06-05 11:10:32 [INFO] raise RemoteDisconnected("Remote end closed connection without"
2020-06-05 11:10:32 [INFO] urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
2020-06-05 11:10:32 [INFO] During handling of the above exception, another exception occurred:
2020-06-05 11:10:32 [INFO] Traceback (most recent call last):
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/policyd/tests.py", line 475, in test_003_test_overide_is_observed
2020-06-05 11:10:32 [INFO] self.get_client_and_attempt_operation(self.keystone_ips[0])
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/openstack_dashboard/tests.py", line 471, in get_client_and_attempt_operation
2020-06-05 11:10:32 [INFO] result = client.get(_url)
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/requests/sessions.py", line 543, in get
2020-06-05 11:10:32 [INFO] return self.request('GET', url, **kwargs)
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/requests/sessions.py", line 530, in request
2020-06-05 11:10:32 [INFO] resp = self.send(prep, **send_kwargs)
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/requests/sessions.py", line 665, in send
2020-06-05 11:10:32 [INFO] history = [resp for resp in gen] if allow_redirects else []
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/requests/sessions.py", line 665, in <listcomp>
2020-06-05 11:10:32 [INFO] history = [resp for resp in gen] if allow_redirects else []
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/requests/sessions.py", line 245, in resolve_redirects
2020-06-05 11:10:32 [INFO] **adapter_kwargs
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/requests/sessions.py", line 643, in send
2020-06-05 11:10:32 [INFO] r = adapter.send(request, **kwargs)
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/requests/adapters.py", line 498, in send
2020-06-05 11:10:32 [INFO] raise ConnectionError(err, request=request)
2020-06-05 11:10:32 [INFO] requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
2020-06-05 11:10:32 [INFO] During handling of the above exception, another exception occurred:
2020-06-05 11:10:32 [INFO] Traceback (most recent call last):
2020-06-05 11:10:32 [INFO] File "/tmp/tmp.ivuRRwYBQe/func/lib/python3.5/site-packages/zaza/openstack/charm_tests/policyd/tests.py", line 480, in test_003_test_overide_is_observed
2020-06-05 11:10:32 [INFO] .format(str(e)))
2020-06-05 11:10:32 [INFO] zaza.openstack.utilities.exceptions.PolicydError: Service action failed and should have passed after removing policy override: "('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))"

This probably means that the apache webserver wasn't back in time; perhaps some retries to allow it to come back up with help stabilise the test.

Aurelien Lourot (aurelien-lourot) on 2020-06-08

Changed in charm-openstack-dashboard:
status:	New → Triaged
importance:	Undecided → Medium

Aurelien Lourot (aurelien-lourot) on 2020-07-08

Changed in charm-openstack-dashboard:
status:	Triaged → In Progress
assignee:	nobody → Aurelien Lourot (aurelien-lourot)

Revision history for this message

Aurelien Lourot (aurelien-lourot) wrote on 2020-07-08:

Looking at the logs, I don't believe the webserver wasn't back and ready. Also the code that failed is:

def _login(dashboard_ip, domain, username, password):
# [...]
auth_url = 'http://{}/horizon/auth/login/'.format(dashboard_ip)

    # start session, get csrftoken
    client = requests.session()
    client.get(auth_url)

def get_client_and_attempt_operation(self, ip):
    # [...]
    client, response = _login(
        unit.public_address, domain, username, password)
    # now attempt to get the domains page
    _url = self.url.format(unit.public_address)
    result = client.get(_url)

_login() (and its client.get()) succeeded (so the webserver was ready) whereas the later client.get() failed.

Changed in charm-openstack-dashboard:
status:	In Progress → Confirmed

Aurelien Lourot (aurelien-lourot) on 2020-07-20

Changed in charm-openstack-dashboard:
status:	Confirmed → Triaged

Aurelien Lourot (aurelien-lourot) on 2020-07-28

Changed in charm-openstack-dashboard:
assignee:	Aurelien Lourot (aurelien-lourot) → nobody

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.