networking-odl

networking-odl-functional-oxygen fails with neutron.common.utils.WaitTimeout: Timed out after 5 seconds

Bug #1835885 reported by Lajos Katona on 2019-07-09

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	networking-odl	New	Undecided	Lajos Katona

Bug Description

networking-odl-functional-oxygen randomly fails (there are cases when just a few tests and cases when half of the suit fails) with the following exception:
...
  File "networking_odl/tests/functional/base.py", line 117, in get_odl_resource
    utils.wait_until_true(no_journal_rows, 5, 0.5)
  File "/home/zuul/src/opendev.org/openstack/networking-odl/.tox/functional/local/lib/python2.7/site-packages/neutron/common/utils.py", line 702, in wait_until_true
    raise WaitTimeout(_("Timed out after %d seconds") % timeout)
neutron.common.utils.WaitTimeout: Timed out after 5 seconds

Example:
http://logs.openstack.org/04/668904/1/check/networking-odl-functional-oxygen/06e6765/testr_results.html.gz

Logstash query:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22utils.wait_until_true%5C%5C(no_journal_rows%2C%205%2C%200.5%5C%22

Tags:

Lajos Katona (lajos-katona) on 2019-07-09

Changed in networking-odl:
assignee:	nobody → Lajos Katona (lajos-katona)

Revision history for this message

Lajos Katona (lajos-katona) wrote on 2019-07-09:

Download full text (5.6 KiB)

An interesting log entry:
http://logs.openstack.org/04/668904/1/check/networking-odl-functional-oxygen/06e6765/controller/logs/functional-logs/networking_odl.tests.functional.test_odl_dhcp_driver.TestOdlDhcpDriver.test_subnet_update_from_disable_to_enable.txt.gz

By this the connection towards ODL is timed out.

By this the connection towards ODL is timed out.

2019-07-04 15:23:57.809 31587 ERROR networking_odl.common.client [req-11836557-737d-4a05-8659-1905c97a68bc - - - - -] REST request ( post ) to url ( security-groups ) is failed. Request body : [{u'security_group': {'project_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b', 'name': u'default', 'security_group_rules': [{'project_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b', 'port_range_max': None, 'remote_group_id': '3483f322-c3c1-4d76-ae55-8e4ca7018c93', 'port_range_min': None, 'protocol': None, 'direction': u'ingress', 'security_group_id': '3483f322-c3c1-4d76-ae55-8e4ca7018c93', 'tenant_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b', 'remote_ip_prefix': None, 'id': 'a6cd8e0e-d2fe-4c24-b81f-2693529d41ea', 'ethertype': u'IPv4'}, {'project_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b', 'port_range_max': None, 'remote_group_id': None, 'port_range_min': None, 'protocol': None, 'direction': u'egress', 'security_group_id': '3483f322-c3c1-4d76-ae55-8e4ca7018c93', 'tenant_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b', 'remote_ip_prefix': None, 'id': '2e859b19-cbdf-4232-961b-8451de8253dc', 'ethertype': u'IPv4'}, {'project_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b', 'port_range_max': None, 'remote_group_id': '3483f322-c3c1-4d76-ae55-8e4ca7018c93', 'port_range_min': None, 'protocol': None, 'direction': u'ingress', 'security_group_id': '3483f322-c3c1-4d76-ae55-8e4ca7018c93', 'tenant_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b', 'remote_ip_prefix': None, 'id': '0e4582ba-75a1-4dae-bff8-f6de8ad5c413', 'ethertype': u'IPv6'}, {'project_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b', 'port_range_max': None, 'remote_group_id': None, 'port_range_min': None, 'protocol': None, 'direction': u'egress', 'security_group_id': '3483f322-c3c1-4d76-ae55-8e4ca7018c93', 'tenant_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b', 'remote_ip_prefix': None, 'id': '059a1481-8085-47a0-9cca-98086a4468ab', 'ethertype': u'IPv6'}], 'description': u'Default security group', 'id': '3483f322-c3c1-4d76-ae55-8e4ca7018c93', 'tenant_id': u'46f70361-ba71-4bd0-9769-3573fd227c4b'}}] service: ReadTimeout: HTTPConnectionPool(host='127.0.0.1', port=8181): Read timed out. (read timeout=10)
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal [req-11836557-737d-4a05-8659-1905c97a68bc - - - - -] Error while processing (Entry ID: 1) - create security_group 3483f322-c3c1-4d76-ae55-8e4ca7018c93 (Time stamp: 63697850637.8): ReadTimeout: HTTPConnectionPool(host='127.0.0.1', port=8181): Read timed out. (read timeout=10)
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal Traceback (most recent call last):
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal   File "networking_odl/journal/journal.py", line 280, in _sync_entry
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal     self.client.sendjson(method, urlpath, to_send)
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal   File "networking_odl/common/client.py", line 99, in sendjson
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal     'body': obj})
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal   File "/home/zuul/src/opendev.org/openstack/networking-odl/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal     self.force_reraise()
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal   File "/home/zuul/src/opendev.org/openstack/networking-odl/.tox/functional/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal     six.reraise(self.type_, self.value, self.tb)
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal   File "networking_odl/common/client.py", line 91, in sendjson
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal     self.request(method, urlpath, data))
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal   File "networking_odl/common/client.py", line 84, in request
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal     method, url=url, headers=headers, data=data, timeout=self.timeout)
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal   File "/home/zuul/src/opendev.org/openstack/networking-odl/.tox/functional/local/lib/python2.7/site-packages/requests/sessions.py", line 533, in request
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal     resp = self.send(prep, **send_kwargs)
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal   File "/home/zuul/src/opendev.org/openstack/networking-odl/.tox/functional/local/lib/python2.7/site-packages/requests/sessions.py", line 646, in send
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal     r = adapter.send(request, **kwargs)
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal   File "/home/zuul/src/opendev.org/openstack/networking-odl/.tox/functional/local/lib/python2.7/site-packages/requests/adapters.py", line 529, in send
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal     raise ReadTimeout(e, request=request)
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal ReadTimeout: HTTPConnectionPool(host='127.0.0.1', port=8181): Read timed out. (read timeout=10)
2019-07-04 15:23:57.810 31587 ERROR networking_odl.journal.journal

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.