openstack-ansible

neutron-ovn-metadata-agent logs fill disk

Bug #2027854 reported by James Denton
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
openstack-ansible
New
Undecided
Unassigned

Bug Description

If for some reason Neutron/OVN services are unavailable, the neutron-ovn-metadata-agent will continuously spit logs and eventually fill the disk via /var/log/syslog. Restarting services and clearing syslog gets things back in order.

--
Jul 15 00:37:20 lab-compute01 neutron-ovn-metadata-agent[1477]: 2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection [-] non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>: ValueError: non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection Traceback (most recent call last):
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection File "/openstack/venvs/neutron-27.1.0.dev22/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 107, in run
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection self.idl.run()
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection File "/openstack/venvs/neutron-27.1.0.dev22/lib/python3.10/site-packages/ovs/db/idl.py", line 433, in run
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection self._session.run()
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection File "/openstack/venvs/neutron-27.1.0.dev22/lib/python3.10/site-packages/ovs/jsonrpc.py", line 519, in run
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection error = self.stream.connect()
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection File "/openstack/venvs/neutron-27.1.0.dev22/lib/python3.10/site-packages/ovs/stream.py", line 817, in connect
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection retval = super(SSLStream, self).connect()
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection File "/openstack/venvs/neutron-27.1.0.dev22/lib/python3.10/site-packages/ovs/stream.py", line 300, in connect
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection self.__scs_connecting()
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection File "/openstack/venvs/neutron-27.1.0.dev22/lib/python3.10/site-packages/ovs/stream.py", line 268, in __scs_connecting
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection retval = self.check_connection_completion(self.socket)
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection File "/openstack/venvs/neutron-27.1.0.dev22/lib/python3.10/site-packages/ovs/stream.py", line 777, in check_connection_completion
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection return Stream.check_connection_completion(sock)
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection File "/openstack/venvs/neutron-27.1.0.dev22/lib/python3.10/site-packages/ovs/stream.py", line 137, in check_connection_completion
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection return ovs.socket_util.check_connection_completion(sock)
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection File "/openstack/venvs/neutron-27.1.0.dev22/lib/python3.10/site-packages/ovs/socket_util.py", line 181, in check_connection_completion
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection sock.send("\0".encode(), socket.MSG_DONTWAIT)
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection File "/openstack/venvs/neutron-27.1.0.dev22/lib/python3.10/site-packages/eventlet/green/ssl.py", line 194, in send
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection return self._call_trampolining(
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection File "/openstack/venvs/neutron-27.1.0.dev22/lib/python3.10/site-packages/eventlet/green/ssl.py", line 158, in _call_trampolining
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection return func(*a, **kw)
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection File "/usr/lib/python3.10/ssl.py", line 1203, in send
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection raise ValueError(
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection ValueError: non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>
2023-07-15 00:37:20.438 1477 ERROR ovsdbapp.backend.ovs_idl.connection
--

Revision history for this message
Dmitriy Rabotyagov (noonedeadpunk) wrote :

I'm inclined to think this is a duplicate of https://bugs.launchpad.net/openvswitch/+bug/1985062

Though I would expect it to be fixed already upstream for a while.

Revision history for this message
Dmitriy Rabotyagov (noonedeadpunk) wrote :

Same issue actually happens on OVS and it's neutron-server.service that's filling up the disk

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to openstack-ansible-openstack_hosts (stable/zed)

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/892563

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to openstack-ansible-openstack_hosts (stable/yoga)

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/892581

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on openstack-ansible-openstack_hosts (stable/yoga)

Change abandoned by "Dmitriy Rabotyagov <email address hidden>" on branch: stable/yoga
Review: https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/892581
Reason: This will most likely break EL8->EL9 upgrade path

Revision history for this message
Dmitriy Rabotyagov (noonedeadpunk) wrote (last edit ):

It's fixed in OVS python package 2.17.3. Though U-C wasn't updated for quite a while and still install affected version which is
2.17.1.post1

I've proposed patches there:
https://review.opendev.org/c/openstack/requirements/+/892891
https://review.opendev.org/c/openstack/requirements/+/892892

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to openstack-ansible-openstack_hosts (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/892563
Committed: https://opendev.org/openstack/openstack-ansible-openstack_hosts/commit/c102a8f43ec48c8abd9c98c1fc1ef9f1402375ce
Submitter: "Zuul (22348)"
Branch: stable/zed

commit c102a8f43ec48c8abd9c98c1fc1ef9f1402375ce
Author: Dmitriy Rabotyagov <email address hidden>
Date: Wed Aug 23 17:59:05 2023 +0200

    Release constraint for OVS/OVN versions for EL

    RDO has stopped building packages for older versions of OVS/OVN
    which means that upstream fixes are not ending up in deployments.

    We remove previously defined excludes as bugs related to
    new versions were sorted out. At the same time we struggle from
    old OVS bugs in CI.

    Change-Id: I5be67c3a08645050187bee61e6a46f6b6f738604
    Related-Bug: #2027854

tags: added: in-stable-zed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.