[eglibc] process shared mutex's fail on armel v7 (thumb)

for the apr report see bug 599874

It's critical that we investigate this and understand it.

code should be fixed to ensure the use of the atomic primitives provided by gcc, and gcc should ensure not to use the expensive routines when in v7 mode.

Note that there are upstream patches that improve the GCC _sync_*
primitives. If APR uses those, or via GLIBC uses those, then it might
be related.

Confirmed on maverick with gcc-4.4 4.4.4-9ubuntu2, eglibc 2.12.1-0ubuntu1, and apr 1.4.2-3ubuntu1

The test halts while running the pthread based APR_LOCK_PROC_PTHREAD variant. All other variants pass.

The test spawns a number of processes and then randomly locks, writes to a shared variable, waits, and unlocks on a process mutex across the different processes. When using APR_LOCK_PROC_PTHREAD, what actually happens is the first process locks and unlocks while all others are stalled.

LP: #491342 is probably related. According to that pthread spinlocks are broken on eglibc on ARM due to still using the old swp/swpb instructions.

Next step would be to replace these with __sync_* as per sysdeps/ia64/bits/atomic.h and similar.

I'm opening an apr task to remember reverting the changes from bug #599874 once this is fixed in eglibc.

Given that the APR task is just for our own development usage, I'm setting the status to Confirmed, and the Importance to Wishlist. When eglibc is fixed, the changes should be reverted, but at the very least, this bug can change to status Triaged at that point.

I had a quick look into the apr sources. Their locks/unix/proc_mutex.c can map the proc_mutex functionality to various methods:
 1) Posix sem implemenation (see mutex_posixsem_methods)
 2) SysV sem implementation (see mutex_sysv_methods)
 3) pthread implementation (see mutex_proc_pthread_methods)
 4) fcntl implementation (see mutex_fcntl_methods)
 5) flock implementation (see mutex_flock_methods)

I think only the pthread implemenation (3) is of interest here. The creation of the pthread_mutex happens at:
  locks/unix/proc_mutex.c:proc_mutex_proc_pthread_create
that calls pthread_mutex_init with the following attributes:
  PTHREAD_PROCESS_SHARED, PTHREAD_MUTEX_ROBUST_NP, PTHREAD_PRIO_INHERIT
The apr-phthread-mutex functionality relies on pthread_mutex_lock and pthread_mutex_trylock. The eglibc implemenation for such a mutex calls atomic_compare_and_exchange_val_acq (see also lll_lock/lll_trylock at ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.h) which goes down to __arch_compare_and_exchange_val_32_acq. Using (e)glibc (>= 2.12.1-0ubuntu11) and GCC (>=4.5) this expands to __sync_val_compare_and_swap (ports/sysdeps/unix/sysv/linux/arm/nptl/bits/atomic.h) which should be safe even on a SMP system.

Some infos regarding the atomic memory operations can be found at:
https://wiki.linaro.org/WorkingGroups/ToolChain/AtomicMemoryOperations#implementation%20details

The next step would be to check whether the testcase fails using a decent GCC and eglibc (Ubuntu Natty for example).

I ran the testprocmutex test on a panda board using ubuntu natty alpha 2 and it passed:
$ ./testall testprocmutex
testprocmutex : SUCCESS
All tests passed.

$ apt-cache show libc6 gcc|grep Version
Version: 4:4.5.1-1ubuntu3
Version: 2.12.1-0ubuntu16

I ran the apr testsuite (svn revision 1071306) several times and I notice that the testlock:test_timeoutcond test fails sometimes because the timer returned to late. Not sure sure if this is related.

I think this the issue has been fixed by #643171. I do not see this testcase failing using a recent (e)glibc and GCC. Could anyone of the original reporters check if it works for them? Please make sure that you use at least glibc 2.12.1-0ubuntu11 and GCC 4.5.

apr without workaround applied passes tests on current natty so glibc is fixed.
We still need to revert the workaround in apr (that is part of Debian packaging now as well)

This bug was fixed in the package apr - 1.4.2-7ubuntu1

---------------
apr (1.4.2-7ubuntu1) natty; urgency=low

  * debian/rules: Reenable robust pthread mutexes on armel, now that eglibc
    process shared mutexes were fixed to use gcc atomic sync builtins.
    (LP: #604753)
 -- Jani Monoses <email address hidden> Fri, 18 Mar 2011 18:37:44 +0200

Reverted:

apr (1.4.2-7ubuntu2) natty; urgency=low

  * Revert previous change. I forgot the build servers have an older
    kernel on which the testsuite fails to pass. Reopens LP: #604753

 -- Jani Monoses <email address hidden> Mon, 21 Mar 2011 10:14:43 +0200

Need to reapply once buildds' kernels are upgraded

Do the builders have newer kernels now? If so the testsuite in apr may now pass and the workaround could be dropped.