Segfault in svnserve on UB 16.04 LTS, patch available most likely.

Hi all,

to make a long story short: There's a segfault in svnserve 1.9.3 distributed by UB 16.04 LTSm which is very likely available in 1.9.7 distributed with UB 18.04 LTS as well and has been fixed upstream by the following patch in version 1.9.9:

https://svn.apache.org/viewvc?view=revision&revision=1818584
https://lists.apache.org/thread.html/a60674fd3169cc66a73dd2fd12b60b56fc7d1332c6baa264fb0ae91a%40%3Cdev.subversion.apache.org%3E

The more detailed background can be found on Launchpad and the users@ and dev@ mailing lists of SVN:

https://answers.launchpad.net/ubuntu/+question/404322
https://lists.apache.org/thread.html/d51cd58613c1033a0e53210346f901cb264d7da77723e4a80412ebc8%40%3Cusers.subversion.apache.org%3E
https://lists.apache.org/thread.html/6271afd4bd30f20512d371fc9483338f76996f907d5f9a0d904bbad7%40%3Cdev.subversion.apache.org%3E

Some more details from the links:

Some days ago I recognized a segfault in svnserve which seems to have
been documented for UB 16.04 LTS already:

> Apr 12 09:58:55 [...] kernel: [214930.125762] svnserve[556]: segfault at 7f5f75994f00 ip 00007f5f74ea1065 sp 00007ffddc1353f0 error 4 in libsvn_subr-1.so.1.0.0[7f5f74e43000+d3000]
> Apr 12 10:11:41 [...] kernel: [215695.854475] svnserve[3769]: segfault at 7f5f75994f00 ip 00007f5f74ea1065 sp 00007ffddc1353f0 error 4 in libsvn_subr-1.so.1.0.0[7f5f74e43000+d3000]

https://answers.launchpad.net/ubuntu/+question/404322

In all cases the version seems to be the default one distributed by
UB, 1.9.3, and one additional thing in common seems to be the usage
of hooks at least in some repos. The thread starter e.g. sends mails,
while in one of my repos I'm distributing commits using svnsync.

The problem is that it seems a bit difficult to reproduce what
triggers the segfault: After recognizing it, I committed in some test
repo without any hook and in the repo with the hook and in both cases
wasn't able to trigger the problem. Notice that I dind't restart
svnserve or the whole system before the tests, just as is. Looking at
the past logs, the problem somtimes happens every few minutes,
somtimes a few times a day and sometimes even not at all.

The interesting thing in my case is that I have some repo to which
things get committed multiple times a day automatically. That's as
well the repo with the hook. So there shouldn't be any day without any
commit to at least one repo, but I still have days without the problem.

If the segfaults happen, it seems to be exactly the same error
message until the main process gets restarted. My logs contained only
one exception from the rule: 7f5f75994640

> Apr 9 12:50:14 [...] kernel: [1677281.956916] svnserve[30221]: segfault at 7f0258833640 ip 00007f0257d40065 sp 00007ffe29fd1de0 error 4 in libsvn_subr-1.so.1.0.0[7f0257ce2000+d3000]
> Apr 9 12:55:09 [...] kernel: [1677577.574064] svnserve[31221]: segfault at 7f0258833640 ip 00007f0257d40065 sp 00007ffe29fd1de0 error 4 in libsvn_subr-1.so.1.0.0[7f0257ce2000+d3000]
[...]
> Apr 10 09:04:14 [...] kernel: [38850.248311] svnserve[19575]: segfault at 7fe569bcbc30 ip 00007fe5690d7065 sp 00007ffd951fdad0 error 4 in libsvn_subr-1.so.1.0.0[7fe569079000+d3000]
> Apr 10 09:06:11 [...] kernel: [38967.469929] svnserve[19596]: segfault at 7f5f75994640 ip 00007f5f74ea1065 sp 00007ffddc1353f0 error 4 in libsvn_subr-1.so.1.0.0[7f5f74e43000+d3000]
> Apr 10 09:13:25 [...] kernel: [39401.078386] svnserve[20429]: segfault at 7fe569bcbc30 ip 00007fe5690d7065 sp 00007ffd951fdad0 error 4 in libsvn_subr-1.so.1.0.0[7fe569079000+d3000]
> Apr 10 09:15:44 [...] kernel: [39539.710149] svnserve[21583]: segfault at 7fe569bcbc30 ip 00007fe5690d7065 sp 00007ffd951fdad0 error 4 in libsvn_subr-1.so.1.0.0[7fe569079000+d3000]
[...]
> Apr 11 12:07:13 [...] kernel: [136227.892728] svnserve[10466]: segfault at 7fe569bcbc30 ip 00007fe5690d7065 sp 00007ffd951fdad0 error 4 in libsvn_subr-1.so.1.0.0[7fe569079000+d3000]
> Apr 11 12:08:38 [...] kernel: [136313.617840] svnserve[10515]: segfault at 7fe569bcbc30 ip 00007fe5690d7065 sp 00007ffd951fdad0 error 4 in libsvn_subr-1.so.1.0.0[7fe569079000+d3000]
> Apr 11 12:08:38 [...] kernel: [136313.619646] svnserve[10512]: segfault at 7fe569bcbc30 ip 00007fe5690d7065 sp 00007ffd951fdad0 error 4 in libsvn_subr-1.so.1.0.0[7fe569079000+d3000]
> Apr 11 12:09:41 [...] kernel: [136376.041439] svnserve[10591]: segfault at 7fe569bcbc30 ip 00007fe5690d7065 sp 00007ffd951fdad0 error 4 in libsvn_subr-1.so.1.0.0[7fe569079000+d3000]
> Apr 11 12:10:10 [...] kernel: [136404.999582] svnserve[10629]: segfault at 7fe569bcbc30 ip 00007fe5690d7065 sp 00007ffd951fdad0 error 4 in libsvn_subr-1.so.1.0.0[7fe569079000+d3000]
> Apr 11 12:11:25 [...] kernel: [136480.403404] svnserve[11437]: segfault at 7fe569bcbc30 ip 00007fe5690d7065 sp 00007ffd951fdad0 error 4 in libsvn_subr-1.so.1.0.0[7fe569079000+d3000]
> Apr 11 12:11:25 [...] kernel: [136480.414366] svnserve[11438]: segfault at 7fe569bcbc30 ip 00007fe5690d7065 sp 00007ffd951fdad0 error 4 in libsvn_subr-1.so.1.0.0[7fe569079000+d3000]
> Apr 11 12:12:05 [...] kernel: [136520.407709] svnserve[11470]: segfault at 7fe569bcbc30 ip 00007fe5690d7065 sp 00007ffd951fdad0 error 4 in libsvn_subr-1.so.1.0.0[7fe569079000+d3000]
[...]
> Apr 12 09:58:55 [...] kernel: [214930.125762] svnserve[556]: segfault at 7f5f75994f00 ip 00007f5f74ea1065 sp 00007ffddc1353f0 error 4 in libsvn_subr-1.so.1.0.0[7f5f74e43000+d3000]
> Apr 12 10:11:41 [...] kernel: [215695.854475] svnserve[3769]: segfault at 7f5f75994f00 ip 00007f5f74ea1065 sp 00007ffddc1353f0 error 4 in libsvn_subr-1.so.1.0.0[7f5f74e43000+d3000]

After posting the problem to the user mailing list, I was instructed
to install debug symbols and get a core dump with some stacktrace and
did so.

https://lists.apache.org/thread.html/d51cd58613c1033a0e53210346f901cb264d7da77723e4a80412ebc8%40%3Cusers.subversion.apache.org%3E

> Stacktrace:
> #0 object_ref_cleanup (baton=0x7f5f75994f00) at /build/subversion-8E3yhQ/subversion-1.9.3/subversion/libsvn_subr/object_pool.c:148
> object = 0x7f5f75994f00
> object_pool = <optimized out>
> #1 0x00007f5f747e4e3e in apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> No symbol table info available.
> #2 0x00007f5f747e4e15 in apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> No symbol table info available.
> #3 0x0000000000406e4e in main (argc=<optimized out>, argv=0x7ffddc135568) at /build/subversion-8E3yhQ/subversion-1.9.3/subversion/svnserve/svnserve.c:1368
> pool = 0x7f5f759d6028
> exit_code = 0
> err = <optimized out>
> StacktraceAddressSignature: /usr/bin/svnserve:11:/usr/lib/x86_64-linux-gnu/libapr-1.so.0.5.2+1ae3e:/usr/lib/x86_64-linux-gnu/libapr-1.so.0.5.2+1ae15:/usr/bin/svnserve+6e4e
> StacktraceSource:
> #0 object_ref_cleanup (baton=0x7f5f75994f00) at /build/subversion-8E3yhQ/subversion-1.9.3/subversion/libsvn_subr/object_pool.c:148
> 143: */
> 144: static apr_status_t
> 145: object_ref_cleanup(void *baton)
> 146: {
> 147: object_ref_t *object = baton;
> 148: svn_object_pool__t *object_pool = object->object_pool;
> 149:
> 150: /* If we released the last reference to object, there is one more
> 151: unused entry.
> 152:
> 153: Note that unused_count does not need to be always exact but only
> #1 0x00007f5f747e4e3e in apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2 0x00007f5f747e4e15 in apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #3 0x0000000000406e4e in main (argc=<optimized out>, argv=0x7ffddc135568) at /build/subversion-8E3yhQ/subversion-1.9.3/subversion/svnserve/svnserve.c:1368
> 1363: if (threads)
> 1364: apr_thread_pool_destroy(threads);
> 1365: #endif
> 1366:
> 1367: /* this will also close the server's socket */
> 1368: svn_pool_destroy(pool);
> 1369: return exit_code;
> 1370: }
>
> StacktraceTop:
> object_ref_cleanup (baton=0x7f5f75994f00) at /build/subversion-8E3yhQ/subversion-1.9.3/subversion/libsvn_subr/object_pool.c:148
> apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> main (argc=<optimized out>, argv=0x7ffddc135568) at /build/subversion-8E3yhQ/subversion-1.9.3/subversion/svnserve/svnserve.c:1368
> ThreadStacktrace:
> .
> Thread 1 (Thread 0x7f5f759c9780 (LWP 3769)):
> #0 object_ref_cleanup (baton=0x7f5f75994f00) at /build/subversion-8E3yhQ/subversion-1.9.3/subversion/libsvn_subr/object_pool.c:148
> object = 0x7f5f75994f00
> object_pool = <optimized out>
> #1 0x00007f5f747e4e3e in apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> No symbol table info available.
> #2 0x00007f5f747e4e15 in apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> No symbol table info available.
> #3 0x0000000000406e4e in main (argc=<optimized out>, argv=0x7ffddc135568) at /build/subversion-8E3yhQ/subversion-1.9.3/subversion/svnserve/svnserve.c:1368
> pool = 0x7f5f759d6028
> exit_code = 0
> err = <optimized out>

That stacktrace lead to the following answer on the dev@ mailing list:

> This stack trace looks very much like the issue fixed with this change:
> https://svn.apache.org/r1818584
> The fix was released as part of SVN 1.9.9 on 20 July 2018.