Open vSwitch (Version 2.9.2) goes into deadlocked state

Status changed to 'Confirmed' because the bug affects multiple users.

https://github.com/openvswitch/ovs/commit/2ed1c95e9c80a21a5d81cb760c872b62b32b0733

This seems to be commit for this.
and 2.9.3 has it.

@Camilo, Do you have reproducer for this issue?

From the discussion in https://github.com/openvswitch/ovs-issues/issues/175 it seems like this may actually be a glibc bug https://sourceware.org/bugzilla/show_bug.cgi?id=23844 , I'll try to get some tests done in order to confirm this.

bug ref for 2.9.5 SRU - bug 1854360

2.9.5 test packages in:

  https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3534/+packages

Status changed to 'Confirmed' because the bug affects multiple users.

I just came across this bug report and would like to share my expierence.

I've been having similar issues on 6 servers since we upgraded from 16.04 to 18.04 about 2 years ago with openvswitch.
Our biggest problem is our inabilty to reproduce it. We just see Openvswitch hanging from time to time. Sometimes it takes a day to get stuck, sometimes it takes months.
The only way to recover from it is to restart openvswitch.

Right now we are running with a backport of openvswitch from Disco (2.11.0-0ubuntu2) in Bionic. With that version backported we are having the same issues as with the previously installed 2.9.2-0ubuntu0.18.04.3 version that Bionic has.

I have gbd traces from both versions which I will attach.

Here a small portion from the ovs log and gdb trace of openvswitch 2.9.2-0ubuntu0.18.04.3:
Sun Aug 25 06:16:14 2019-2019-08-25T04:16:14.943Z|00001|ovs_rcu(urcu4)|WARN|blocked 1000 ms waiting for revalidator127 to quiesce
Sun Aug 25 06:16:15 2019-2019-08-25T04:16:15.943Z|00002|ovs_rcu(urcu4)|WARN|blocked 2000 ms waiting for revalidator127 to quiesce
Sun Aug 25 06:16:50 2019-2019-08-25T04:16:17.943Z|00003|ovs_rcu(urcu4)|WARN|blocked 4001 ms waiting for revalidator127 to quiesce

Small portion of the trace:
32 Thread 0x7f1bfa7fc700 (LWP 1461) "revalidator127" 0x00007f1c61aeb37b in futex_abstimed_wait (private=<optimized out>, abstime=0x0, expected=10, futex_word=0x55e4ed0aa800 <rwlock>) at ../sysdeps/unix/sysv/linux/futex-internal.h:172

The full trace is attached in gdbwrap.1566706577.log.gz (Openvswitch 2.9.2)

Here a similar portion from the ovs log and gdb trace running openvswitch 2.11.0-0ubuntu2:
Thu Oct 31 18:20:23 2019-2019-10-31T17:20:23.521Z|00001|ovs_rcu(urcu4)|WARN|blocked 1000 ms waiting for revalidator124 to quiesce
Thu Oct 31 18:20:24 2019-2019-10-31T17:20:24.521Z|00002|ovs_rcu(urcu4)|WARN|blocked 2000 ms waiting for revalidator124 to quiesce
Thu Oct 31 18:20:59 2019-2019-10-31T17:20:26.520Z|00003|ovs_rcu(urcu4)|WARN|blocked 4000 ms waiting for revalidator124 to quiesce

In the trace:
29 Thread 0x7f72f97fa700 (LWP 26608) "revalidator124" 0x00007f734aee237b in futex_abstimed_wait (private=<optimized out>, abstime=0x0, expected=10, futex_word=0x5577bab397c0 <rwlock>) at ../sysdeps/unix/sysv/linux/futex-internal.h:172

Full trace attached in gdbwrap.1572542426.log.gz

The traces are a bit hocus to me, I really don't have a clue whats going on there but I guess it might help you make sense of whats going on here.

So as documented in https://github.com/openvswitch/ovs-issues/issues/175 , this really seems to be a bug in glibc and not in openswitch. After installing libc6 and related pkgs at version 2.29-0ubuntu2 (from disco), the issue is resolved for us. The single patch from https://sourceware.org/bugzilla/show_bug.cgi?id=23844 did not fix the issue. So we would either need to make an effort to get the version from disco into bionic-updates or someone would need to locate a set of possible patches and backport them.

I would also like to share our experiences:
We have had problems with this bug for a hole year. "ovs is dead..."
First we suspected a bug in openvswitch. But, this issue made us aware of glibc: https://github.com/openvswitch/ovs-issues/issues/175
We updated glibc to version 2.29-0ubuntu2 from the disco repository.
Now, for more than a month we don't have any problems on 6 compute nodes.
It would be nice if the glibc package updates can also be make available in the bionic repositories.

we are also affected by this issue and would appreciate if glibc would be updated to 2.29-0ubuntu2 in bionic repositories.

Is there glic upgrade coming on 18.04? We are hit by this problem very frequently and the only way out is to restart the ovs and neutron services.

And obviously that restarting of ovs and neutron services disrupt the network traffic massively.

@salmankh
you can solve this by installing this packages from eoan
```
libc6
libc6-dbg
libcap-ng0
libc-bin
libidn2-0
locales
openvswitch-common
openvswitch-switch
```
and don't forget to run `systemctl daemon-reexec` afterwards!

is this duplicate of https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1864864 ?

does new glibc from bionic-proposed https://launchpad.net/ubuntu/+source/glibc/2.27-3ubuntu1.3 help?

Likely it is not a duplicate, see my comment in #10 and in the bugzilla. So if branch-pthread_rwlock_trywrlock-hang-23844.patch is the patch from the bugzilla, I tested that earlier and it did not solve the issue. The issue here seems to be with pthread_rwlock_rdlock hanging, see the tracebacks in the ovs github issue.

I have done some extensive research on this and I am able to confirm that while Bug #1864864 does solve a class of issues, it does not solve the one affecting Open vSwitch, as it does rely on PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP:

https://github.com/openvswitch/ovs/blob/0de1b425962db073ebbaa3ddbde445580afda840/lib/ovs-thread.c#L247-L248

And when running the reproducer here:

https://sourceware.org/bugzilla/show_bug.cgi?id=23861

By doing the following:

```
curl https://sourceware.org/bugzilla/attachment.cgi?id=11382 -o bug23861.c
sed -i 's/do_exit = 0/do_exit(0)/' bug23861.c
g++ bug23861.c -lpthread -o bug23861
for ((x=1;x<100;x++)) ; do echo $x;date;./bug23861 --prefer-writer-nonrecursive;done
```

I will add that in my initial tests of doing this inside a small VM with a Docker container, it did not hang after running for 10-15 minutes, but it immediately hung on a much more powerful machine. I will file a bug to try and get the patch above SRU'd into glibc as well and hopefully that will be the end of it.

So, it turns out that 2.27-3ubuntu1.3 could fix this because of the fact that not only does it include that patch, but a full rebase to 2.27 branch of upstream glibc, which does include the other fix mentioned above (I found this out when I tried to rebuild locally and found out that quilt was complaining of an already built patch).

Upon running the code above on the 2.27-3ubuntu1.2 system, it freezes but works fine under 2.27-3ubuntu1.3.

For context - 2.27-3ubuntu1.3 is in bionic-proposed

Thanks for the research @mnaser - interesting that this is only seen on much more powerful machines which I think has been the key blocker to reproduction of this issue.

2.27-3ubuntu1.3 has been released to bionic-updates. @mnaser do I understand correctly that this bug is fully fixed by it?

Also I assume the bug is fixed in Ubuntu 20.04 and later as well. Could you please confirm that?