all openoffice apps die in 'com::sun::star::ucb::InteractiveAugmentedIOException' on armel in karmic

@Chris - I'm probably not the right person to take assignment of this bug. The reason I targetted it at alpha5 came out of a discussion with ogra on IRC. Thanks for the vote of confidence though!

Hopefully someone can fix it, I won't be able to as I am only at 20% time on OOo which is barely enough time to even do bug triage, not to mention upload new versions of the package or fix bugs.

bumping to A6, we need to find someone to take that bug first, will be discussed with the desktop team

Assigning to doko to see if he can help.

I have asked Rene about the issue and he hasn't heard anything about it yet. I also emailed the ooo-build mailing list to see if anyone else sees the problem. Also the person who did the initial arm port reads that list so perhaps he will have some insight on the issue. If nothing comes from the mailing list post I will get a babbage system sent to me to do testing.

Thanks,

Chris

This is his response:

"
Have you got workspace.cmcfixes59.patch applied ? That's got some fixes
for IA64 and ARM to ensure the desired alignment, i.e. without it an ARM
OOo built on qemu or in kernel lenient mode will auto-detect laxer
alignment that is desirable on arm. There would probably be a kernel
message about bad alignment in this case.

There was some spurious -march in unxlngr.mk in the past, but I think
this is gone now.

It does look like it might be the very first uno exception throw getting
busted (experiment with gdb, break main, catch throw, cont, and see if
its "death on first excecption" which might point to experimenting with
commenting out the #define USE_DOUBLE_MMAP in
bridges/inc/bridges/cpp_uno/shared/vtablefactory.hxx
and seeing if that makes a difference.

Otherwise it might be the the exception code in
bridges/source/cpp_uno/gcc3_linux_arm/except.cxx is wrong, e.g. the bit
inside __ARM_EABI__ but I've no real idea, just some possible clues.
"

I will look into seeing about the patch he references in the comment.u

This is his response:

"
Have you got workspace.cmcfixes59.patch applied ? That's got some fixes
for IA64 and ARM to ensure the desired alignment, i.e. without it an ARM
OOo built on qemu or in kernel lenient mode will auto-detect laxer
alignment that is desirable on arm. There would probably be a kernel
message about bad alignment in this case.

There was some spurious -march in unxlngr.mk in the past, but I think
this is gone now.

It does look like it might be the very first uno exception throw getting
busted (experiment with gdb, break main, catch throw, cont, and see if
its "death on first excecption" which might point to experimenting with
commenting out the #define USE_DOUBLE_MMAP in
bridges/inc/bridges/cpp_uno/shared/vtablefactory.hxx
and seeing if that makes a difference.

Otherwise it might be the the exception code in
bridges/source/cpp_uno/gcc3_linux_arm/except.cxx is wrong, e.g. the bit
inside __ARM_EABI__ but I've no real idea, just some possible clues.
"

I will look into seeing about the patch he references in the comment.

A similar patch to the one referenced seems to already be applied.

Oli,

Does any of the stuff Caolan mentioned make sense to you? I have no prior experience with arm at all.

Chris

they surely do since he is the expert, i have as much experience with OO.o as you have with arm :)

So Oliver points out we have a slightly different patch than the referenced one. If the patch is only about fixing alignment issues then it's not a big deal since I confirmed that the bug reported here doesn't trigger any alignment traps (echo 3 >/proc/cpu/alignment + watching dmesg and cat /proc/cpu/alignment).

I did various research over the day and confirmed that it's not USE_DOUBLE_MMAP (a strace already indicated that it was unlikely the issue).

It might be related to recent libstdc++ changes in gcc-4.4 but it's not clear.

I'm copying the log of today's conversations with caolan. It looks like we need to rebuild an old ooo with our latest toolchain or rebuild latest ooo with older toolchains.

11:56 < lool> I'm trying to debug an early startup issue of soffice.bin on Ubuntu armel
11:57 < lool> (gdb) run -norestore -writer
11:57 < lool> Starting program: /usr/lib/openoffice/program/soffice.bin -norestore -writer
11:57 < lool> terminate called after throwing an instance of 'com::sun::star::ucb::InteractiveAugmentedIOException'
11:57 < lool> During startup program exited with code 80.
11:57 < lool> I tried breaking on various functions but I dont seem to reach them
11:57 < lool> Oddly strace shows a SIGABRT which I dont see in gdb
11:59 < lool> I read through http://wiki.services.openoffice.org/wiki/Debugging and http://www.skynet.ie/~caolan/TechTexts/OpenOfficeHacking.html but it seems it's breaking earlier
12:07 <@caolan> lool: are you able to... gdb) break main, (gdb) run -norestore -writer, gdb) cont, gdb) catch throw
12:09 < lool> caolan: If I try to break main it just does the same output
12:10 <@caolan> lool: i.e. the terminate called... ?
12:10 < lool> Yes
12:10 < lool> caolan: Oh you're the author of the EABI patch?
12:10 <@caolan> does gdb offer any sort of a bt ?
12:10 < lool> No, I just drop back to gdb
12:11 <@caolan> wonder if its dying before main, or if gdb is just uselessly broken
12:11 < lool> Perhaps it's more fragile on armel and needs debug symbols
12:12 < lool> I tried breaking on things like __libc_start_main too and that didn't work either
12:12 < lool> So it might be gdb being broken
12:15 <@caolan> lool: btw, is this under a real arm, qemu-system-arm, or qemu-arm ?
12:17 < lool> caolan: Real arm
12:17 < lool> caolan: imx51 soc
12:17 < lool> v7 but v5/v6 userspace
12:18 < lool> We just changed our toolchain from v5 + soft vfp to v6 + softfp vfp
12:18 <@caolan> lool: version is 3.1.1 right ?
12:18 < lool> caolan: BTW( I'd be happy to offer you access if you like)
12:19 < lool> caolan: yes
12:19 <@caolan> lool: and did any earlier versions of OOo work previously ?
12:19 < lool> The jaunty one worked
12:19 < lool> that was 1:3.0.1-9ubuntu3
12:20 <@caolan> hmmm
12:20 < lool> we're not sure of whether a new v5 toolchain broke it or a new oo.o upstream release
12:20 < lool> Oliver tells me a new oo.o upstream release broke it
12:21 < lool> It could really be either toolchain or oo.o
12:21 <@caolan> does e.g. commenting out USE_DOUBLE_MMAP in bridges/inc/bridges/cpp_uno/shared/vtablefactory.hxx make any difference ?
12:22 < lool> It will take me some to build but I'll try that out, thanks
12:24 <@caolan> lool: a str...

This might help debugging:
http://wiki.services.openoffice.org/wiki/Debugging

This bug might explain a gcc regression but not sure:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=38732

And these are instructions for building:
http://wiki.services.openoffice.org/wiki/Documentation/Building_Guide/Building_on_Linux#setting_the_environment
http://wiki.services.openoffice.org/wiki/Ubuntu_Build_Instructions

I have a backported version of 3.1.1 in the openoffice-pkgs ppa for jaunty that you could try out on arm I suppose. Building the OOo from jaunty on karmic would take a lot of work since a lot of supporting packages have changed and many are no longer available in karmic.

https://launchpad.net/~openoffice-pkgs/+archive/ppa

Chris

Note that the build on jaunty would not be 100% identical due to build depends package version differences, but may help determine if the issue is the toolchain.

Chris

trying to build the PPA packages in a jaunty armel system i end up with the attached error

Loic,

From Oliver's build error it looks like it is possible it could be a compiler issue. The same version of OOo on the previous arm toolchain fails to build entirely due to toolchain problems. What is the next step in your opinion? I can try to make OOo 3.0.1 compilable on Karmic but it probably take quite a bit of work to do so.

g++ -D_REENTRANT -I. -I../common -DU_I18N_IMPLEMENTATION -O -fvisibility=hidden -c -o rematch.ao rematch.cpp
rematch.cpp: In member function 'void icu_4_0::RegexMatcher::MatchAt(int32_t, UBool, UErrorCode&)':
rematch.cpp:2785: error: unrecognizable insn:
(insn 4314 1673 1674 182 rematch.cpp:1949 (set (reg:SI 2 r2)
        (plus:SI (reg:SI 3 r3 [1111])
            (mult:SI (reg/v:SI 12 ip [orig:254 opValue.1479 ] [254])
                (const_int 32 [0x20])))) -1 (nil))
rematch.cpp:2785: internal compiler error: in extract_insn, at recog.c:2001
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-4.3/README.Bugs> for instructions.

bug #427829 has info on teh compiler, i'll try a build with lower optimization

Chris, we want to workaround the toolchain issue by using smaller opts for now and report/track/fix the toolchain issue in the mean time.

building the backport from chris ppa in an armel jaunty chroot still fails ( Bug #430525 ), but gets far enough to build bridges/unxlngr.pro/lib/libgcc3_uno.so.
we tried to copy that into a karmic system which has a standard oo.o install, firing up oo.o still fails with the same error.

this seems to indicate it is not a toolchain induced error but changes in the oo.o code causing the failure on startup.

moving on with testing, replacing the packaged karmic versions of uno-libs3 and ure with the packages from jaunty makes the error go away, chris is that info sufficient to find (and fix) the code changes that made it stop working in karmic ?

copying just libgcc3_uno.so from the jaunty package into /usr/lib/ure/lib/ on karmic makes OO.o start as well

I emailed upstream to see if they have any idea what would break in that library to cause the problem.

Caolan's response was the following:

"The OOo source was the same in each case right ? Or was it one version
of OOo vs another ?

Do similar things to the uno bridge (i.e. libgcc3.uno.so) work in both
toolchains, e.g. libffi and mozilla have similar things. So does firefox
exist and work on the new toolchain. How about gij, does that work ? On
the other hand libffi doesn't concern itself with exceptions, and I have
no idea if firefox does either."

When looking to see obvious changes between jaunty and karmic one of the things is the version of boost used, in jaunty it is 1.35 and in karmic it is 1.38.

Also it appears boost 1.38 got some big patches about a month ago which might also have broken something:

"11:19 < ScottK> ccheney: It's a month ago, but a lot more recent than when we switched to it (in May): https://launchpad.net/ubuntu/karmic/+source/boost1.38/1.38.0-6ubuntu3 "

The only problem with testing this is you will need to grab the boost 1.35 from jaunty to test with as it has been transitioned and removed from Karmic.

To be clear about the above comment, I am talking about what is different about OOo 3.1.1 when built on jaunty vs karmic. We can't get an exact duplicate build due to package versions changing, and the most obvious of these other than the toolchain itself is the version of boost used as noted above.

So Matthias did various test builds and Oliver tried various resulting binaries:
- karmic's oo.o source built under karmic and running with jaunty's libgcc3_uno.so works
- running with libgcc3_uno.so from a jaunty oo.o build of karmic's oo.o source works
- karmic's oo.o source built under karmic with gcc-4.3 does not work
- karmic's oo.o source built under karmic with jaunty's eglibc does not work

Caolan who did the initial armel EABI port of oo.o told me to check whether firefox and gij work fine; firefox works fine in our testing in karmic and he proposed using http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21285 as a test case for gij.

I built Test.java with gcj and gcj -C --main=Test and it ran in both cases and reported the stack trace at runtime (both java byte code .class with gij and a.out).

- karmic's oo.o source built under karmic with jaunty's eglibc does not work

that should rather be:
- karmic's oo.o source built under karmic with eglibc built with jauntys toolchain and gcc-4.3 does not work

On 22.09.2009 18:05, Loïc Minier wrote:
> Caolan who did the initial armel EABI port of oo.o told me to check
> whether firefox and gij work fine; firefox works fine in our testing in
> karmic and he proposed using
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=21285 as a test case for
> gij.
>
> I built Test.java with gcj and gcj -C --main=Test and it ran in both
> cases and reported the stack trace at runtime (both java byte code
> .class with gij and a.out).

hmm, however, some stacktrace tests seem to be broken, but not with -O2.

Native configuration is arm-unknown-linux-gnueabi

  === libjava tests ===

Running target unix
FAIL: natevents.cc compilation
FAIL: natgetallthreads.cc compilation
FAIL: natgeterrorname.cc compilation
FAIL: natgetmethodname.cc compilation
FAIL: StackTrace2 execution - source compiled test
FAIL: StackTrace2 -findirect-dispatch execution - source compiled test
FAIL: StackTrace2 -O3 execution - source compiled test
FAIL: StackTrace2 -O3 -findirect-dispatch execution - source compiled test
FAIL: Throw_2 execution - source compiled test
FAIL: Throw_2 -findirect-dispatch execution - source compiled test
FAIL: Throw_2 -O3 execution - source compiled test
FAIL: Throw_2 -O3 -findirect-dispatch execution - source compiled test
FAIL: Throw_3 execution - source compiled test
FAIL: Throw_3 -findirect-dispatch execution - source compiled test
FAIL: Throw_3 -O3 execution - source compiled test
FAIL: Throw_3 -O3 -findirect-dispatch execution - source compiled test
FAIL: stacktrace execution - source compiled test
FAIL: stacktrace -findirect-dispatch execution - source compiled test
FAIL: stacktrace -O3 execution - source compiled test
FAIL: stacktrace -O3 -findirect-dispatch execution - source compiled test

  === libjava Summary for unix ===

# of expected passes 2530
# of unexpected failures 20
# of untested testcases 16

I just heard about a possibly relevant issue. Although I don't have any direct evidence that this is the cause, could this be related to the interaction between exception handling and the switch to VFP? :

> > Codesourcery found and fixed a couple of bugs in glibc's longjmp
> > implementation which could lead to a corrupt FPSCR on systems with
> > VFP or to crashes depending on the value of an uninitialized word in
> > the jumpbuf.

Since this is a recent issue it is likely not to be fixed in mainline yet— I will try to get patches and post links to patches when I have them available.

here is the uno2cpp.cxx from OOo for arm; I'm unsure about the __SOFTFP__ conditional. Shouldn't that read

#if defined(__ARM_EABI__) && (!defined(__SOFTFP__) || defined(__ARM_NEON__) || defined(__VFP_FP__))

Isn't -mfloat-abi=softfp supposed not to change the ABI?

Note there is a similar issue with libffi (see http://gcc.gnu.org/PR41443).

@dave: the original breakage sadly predates the switch to VFP by default

Matthias, we changed the toolchain defaults after this bug had been witnessed.

Also I change oo.o to use -mfloat-abi=soft since I thought uno2cpp.cxx needed porting in case softfp used fpu regs to pass args in some cases (e.g. function calls within the same binary). On second thought this doesn't make any sense since that means functions would have to be modified during link. Calaon confirmed that softfp never used fpu regs to pass args. So it seems uno2cpp.cxx should be changed to allow softpfp and reject -mfloat-abi=hard which is hard to test for. But this is unrelated to our oo.o startup issue sadly.

> #34 Oliver Grawert wrote 7 minutes ago:
>
> @dave: the original breakage sadly predates the switch to VFP by default

Shame :( I'll flag up the toolchain patches when I have them anyway, but I guess they may not solve this problem with OOo.

> #35 Loïc Minier wrote 40 seconds ago:
>
> Calaon confirmed that softfp never used fpu regs to pass
> args.

I think that's right in general, but it's possible that this may not always be true when functions are inlined.

I think that when functions are inlined they don't appear in the backtraces/dont change the stack like a function, so don't need to be handled by the exception handlers/stack parsers.

this is due to bug #436617, introduced with a binutils update in 2.19.51.20090515-0ubuntu1

the broken build with vfp is unrelated, fix in progress as well

Well done Matthias! \o/

Great job Matthias!

Thanks,

Chris

upload waiting for approval since 2009-09-26

Latest attempt to work around this FTBFS on armel due to a missing build-dependency. Pushing the milestone back to 9.10.

supposed to be fixed, see test packages at
https://launchpad.net/~doko/+archive/toolchain

> supposed to be fixed, see test packages at
> https://launchpad.net/~doko/+archive/toolchain
>

I tried installing 1:3.1.1-2ubuntu5~ppa1, but this does not fix the problem
for me.

OOo still dies with the same exception error after the splash screen.

--
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Issue still persists for me with the PPA packages as well.

Well .jaunty works (default) but .old and .new .sos dont.

> Well .jaunty works (default) but .old and .new .sos dont.

Are you referring to the PPA? I didn't find any .jaunty there, only .old and .new, with .new being the default. Again, neither seems to work (in -2ubuntu5~ppa1)

There's a .jaunty is the ppa version:
lool@dove-y1:~$ apt-cache policy ure
ure:
  Installed: 1.5.1+OOo3.1.1-2ubuntu5~ppa1
  Candidate: 1.5.1+OOo3.1.1-2ubuntu5~ppa1
  Version table:
 *** 1.5.1+OOo3.1.1-2ubuntu5~ppa1 0
        500 http://ppa.launchpad.net karmic/main Packages
        100 /var/lib/dpkg/status
     1.5.1+OOo3.1.1-2ubuntu4 0
        500 http://mirror.dooz.org karmic/main Packages
        500 http://ports.ubuntu.com karmic/main Packages

lool@dove-y1:~$ dpkg -L ure|grep /usr/lib/ure/lib/libgcc3_uno
/usr/lib/ure/lib/libgcc3_uno.so.old
/usr/lib/ure/lib/libgcc3_uno.so.new
/usr/lib/ure/lib/libgcc3_uno.so.jaunty
/usr/lib/ure/lib/libgcc3_uno.so

Ah... it looks like this package did not get upgraded when I installed from the PPA... I will try again.

OK, I can now confirm lool's observation - soffice works with libgcc3_uno.so.jaunty, but libgcc3_uno.so.{old,new} seem to suffer from the familiar exception problem.

> > 1) A "filesystem last mounted in future" error is still treated as
> > serious enough to interrupt the boot process and require
> interaction
> > via mountall-shell. My view is that this is not appopriate because
> > this may well happen when Windows fiddles with the clock on a
> > dual-boot system, or when there's some other RTC problem.
> >
> The filesystem upstream author disagrees unfortunately; which
> is why that error is there. Apparently the inconsistency
> causes problems for journalled filesystems.
>
> Now, you might say that this is one of those errors that fsck
> -a should fix automatically - but this issue has become a bit
> politically charged and the ext3/4 upstream is currently
> holding out having it a boot-critical error.

OK... I don't know enough about the fs design to comment, but it sounds plausible.

Would it be possible simply to do a full filesystem check instead of the maintenance shell? I can foresee problems with portable devices where the console text console is not easily to access or use. If the full fs check fails, then dropping to a shell is more appropriate.

This will slow down boot in most circumstances when RTC problems occur, rather than exploding every time.

[...]

> For the Windows case - the hardware clock is in localtime,
> and our installer automatically sets that up when it detects
> the partition.

OK, fair enough. Otherwise, it seemed a good argument :)

>
> You can get the jaunty behaviour by creating an
> /etc/e2fsck.conf file with the contents:
>
> [options]
> buggy_init_scripts = 1
>

If there's a workaround that may be good enough for me. Is this documented somewhere? Otherwise, I'd have no clue to enable this option to work around a buggy RTC :)

[...]

> > for various non-fatal situations anyway? It's necessary to
> quit the
> > shell with "exit 0" to work around this... again, the
> novice user will
> > have no clue about that.
> >
> Good catch, I've committed a fix for that.

OK, thanks. I'll try it... is it in the PPA?

Argh! Please ignore that last comment from me. It relates to a completely different bug— pasted it in the wrong tab X@

Still an issue in karmic, but we have a workaround in place (using the jaunty binary).

I've been doing research on this problem, and making some notes here on my work debugging this. I'm somewhat limited that I'm working off jocote and dyfet's imx51 boards, but I've made some progress.

Reading an interview (http://www.openoffice.org/editorial/peter_naulls.html) with the Peter Nallus who did the initial OOo port to ARM, a few sentence jump out at me, specifically:

"In the end, I was able to write some static assembler that performed the same job, although did rely a bit on specific (but unlikely to change) layout of ARM instructions by the compiler." - Given the breakage is cased by gcc 4.4 and disappears with gcc 4.3, this suggests that the problem may be with OOo over the toolchain.

However, while setting up to do debugging builds, I noticed the following output from configure:
checking if gcc is -fvisibility-inlines-hidden safe with STL headers... yes
checking if gcc has a visibility bug with class-level attributes (GCC bug 26905)... yes
configure: WARNING: Your gcc is not -fvisibility=hidden safe. Disabling visibility

The GCC bug referenced: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=26905 was fixed in 4.2, so its unclear to me if this is a bug in GCC on ARM, a bug in OOo test code, or something else; I'm not familiar enough with GCC C++ visibility to determine if the bug is actually occurring. This test fails in jaunty however where it builds a successful uno library so it might be an unrelated anomaly.

If my understanding is correct changing the visibility of the function will affect how that is called (direct call versus through the PLT) and if it shows up in the PLT at all. If the disability of visibility in the compiler affects the PLT, then it may be throwing off this snipit of ARM assemby code:

        add r1, sp, #16 @ r1 points to this and params
        bl cpp_vtable_call(PLT)

Can a toolchain expert look at the test case, and determine if we have a bug with visibilities? I'll continue to see if the problem is on the OOo side or not, but if we can determine the issue with visibilities on ARM, it might lead us closer to solving the root cause of the problem.

No, afaics the visibility bug has nothing to do with it. wether the configure test is fixed (lucid) or not (karmic), the outcome for this report doesn't change.

After an intense round of debugging, toolchain bisecting, and other crazy things, I've finally managed to pinpoint the exact location of the crash to bridges/source/cpp_uno/gcc3_linux_arm/except.cxx:284

The code in question is the following:

        if (! rtti)
        {
            throw RuntimeException(
                OUString( RTL_CONSTASCII_USTRINGPARAM("no rtti for type ") ) +
                *reinterpret_cast< OUString const * >( &pUnoExc->pType->pTypeName ),
                Reference< XInterface >() );
        }
        }

 __cxa_throw( pCppExc, rtti, deleteException );
    }

On jaunty built versions of the library, when the code hits the __cxa_throw, it would call into the deleteException method in the same class, and continue on its merry way. On karmic and lucid, when the code hits __cxa_throw, it causes __cxa_throw to abort with a std:termiate call. Looking at the ARM C++ EABI, the only way __cxa_throw will call terminate is if there's been a failure in the phase2 unwinding of the stack. Its possible that something is being unwound that shouldn't be, or via verus, but I haven't managed to get a clear stack trace before the __cxa_throw to see the condition of the stack, and due to the nature of UNO, its quite possible that stack trace I may generate may not match what's being unwound by __cxa_throw.

I'm attempting to do a full build of OOo with debug symbols on lucid to debug this, and will post back with my results as I get them.

As requested by ARM toolchain engineers, I got a stacktrace of just before the crash in question (when the PC is stopped just before the __cxa_throw)

Breakpoint 5, gcc3::raiseException (pUnoExc=0xbed9652c, pUno2Cpp=0x1742c4)
    at /home/mcasadevall/src/ooo-build/build/ooo320-m12/bridges/source/cpp_uno/gcc3_linux_arm/except.cxx:284
284 __cxa_throw( pCppExc, rtti, deleteException );
(gdb) bt
#0 gcc3::raiseException (pUnoExc=0xbed9652c, pUno2Cpp=0x1742c4)
    at /home/mcasadevall/src/ooo-build/build/ooo320-m12/bridges/source/cpp_uno/gcc3_linux_arm/except.cxx:284
#1 0x4419cad2 in cpp2uno_call (pThis=0x174980, pMemberTypeDescr=0x1749a0, pReturnTypeRef=0x83b28, nParams=1,
    pParams=0x174748, pCallStack=0xbed96648, pRegisterReturn=0xbed96628)
    at /home/mcasadevall/src/ooo-build/build/ooo320-m12/bridges/source/cpp_uno/gcc3_linux_arm/cpp2uno.cxx:211
#2 0x4419cff2 in cpp_mediate (nFunctionIndex=3, nVtableOffset=0, pCallStack=0xbed96648, pRegisterReturn=0xbed96628)
    at /home/mcasadevall/src/ooo-build/build/ooo320-m12/bridges/source/cpp_uno/gcc3_linux_arm/cpp2uno.cxx:388
#3 0x4419d12c in cpp_vtable_call (pFunctionAndOffset=0x4408d024, pCallStack=0xbed96648)
    at /home/mcasadevall/src/ooo-build/build/ooo320-m12/bridges/source/cpp_uno/gcc3_linux_arm/cpp2uno.cxx:417
#4 0x441a202c in privateSnippetExecutor () from /home/mcasadevall/tmp/OOO/ure/lib/libgcc3_uno.so
#5 0x40626f54 in cppu::throwException (exc=...)
    at /home/mcasadevall/src/ooo-build/build/ooo320-m12/cppuhelper/source/exc_thrower.cxx:242
#6 0x41204656 in ucbhelper::cancelCommandExecution (eError=<value optimized out>, rArgs=<value optimized out>,
    xEnv=<value optimized out>, rMessage=..., xContext=...)
    at /home/mcasadevall/src/ooo-build/build/ooo320-m12/ucbhelper/source/provider/cancelcommandexecution.cxx:127
#7 0x46d6ea24 in fileaccess::throw_handler (errorCode=<value optimized out>, minorCode=<value optimized out>, xEnv=...,
    aUncPath=<value optimized out>, pContent=0x16dbc0, isHandled=true)
    at /home/mcasadevall/src/ooo-build/build/ooo320-m12/ucb/source/ucp/file/filglob.cxx:396
#8 0x46d61a38 in fileaccess::TaskManager::endTask (this=0x1543a4, CommandId=2, aUncPath=<value optimized out>,
    pContent=<value optimized out>) at /home/mcasadevall/src/ooo-build/build/ooo320-m12/ucb/source/ucp/file/filtask.cxx:105
#9 0x46d54736 in fileaccess::BaseContent::endTask (this=0x4408d01c, CommandId=-1093048640)
    at /home/mcasadevall/src/ooo-build/build/ooo320-m12/ucb/source/ucp/file/bc.cxx:1312
#10 0x46d57774 in fileaccess::BaseContent::execute (this=0x16dbc0, aCommand=..., CommandId=<value optimized out>,
    Environment=...) at /home/mcasadevall/src/ooo-build/build/ooo320-m12/ucb/source/ucp/file/bc.cxx:446
#11 0x411e3486 in ucbhelper::Content_Impl::executeCommand (this=<value optimized out>, rCommand=<value optimized out>)
    at /home/mcasadevall/src/ooo-build/build/ooo320-m12/ucbhelper/source/client/content.cxx:1809
#12 0x411e3d92 in ucbhelper::Content::executeCommand (this=0x4408d01c, rCommandName=<value optimized out>,
    rCommandArgument=...) at /home/mcasadevall/src/ooo-build/build/ooo320-m12/ucbhelper/source/client/content.cxx:832
#13 0x4127b25a in...

As another round of debugging passes, I've been trying to isolate the specific changes between karmic and jaunty which caused the regression in the first place. I haven't been testing on lucid since it stands to reason the cause of the UNO failure between karmic and lucid is unchanged. The test results have been baffling ...

For reference purposes:

Control Tests:
jaunty chroot + jaunty toolchain = PASS
karmic chroot + karmic toolchain = FAIL

karmic chroot + karmic gcc-4.3 = FAIL
karmic chroot + jaunty binutils = FAIL
karmic chroot + jaunty binutils + karmic gcc-4.3 = FAIL

karmic chroot + karmic binutils + karmic gcc-4.3 + jaunty glibc = XPASS
jaunty chroot + karmic glibc = XFAIL

I think we're dealing with a very bazaar interaction between OOo, the toolchain, and glibc. I can repeat the above tests with jaunty's gcc-4.3 grafted onto karmic, or via versus, but I'm questioning if it would make a realistic difference as there isn't a lot of changes between jaunty->karmic gcc-4.3 except a new minor vesion.

I'm honestly stumped at the root cause at the moment, and may have to look at disassembling the binaries to determine the root differences between them.

Based on IRC chatter, I realized I forgot to note that the X in XPASS/XFAIL stands for unexpected :-). Sorry about that.

More information, and another piece of the puzzle:

I began tearing apart the headers of the libgcc3_uno.so built on karmic, and jaunty, and noticed an interesting anomaly with the program headers between the jaunty and karmic/lucid versions of this library, the karmic version doesn't have an .eh_frame_hdr

readelf output on jaunty:
mcasadevall@dawn:/usr/lib/ure/lib$ readelf -l libgcc3_uno.so.jaunty

Elf file type is DYN (Shared object file)
Entry point 0x22a0
There are 8 program headers, starting at offset 52

Program Headers:
  Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
  EXIDX 0x010534 0x00010534 0x00010534 0x00a60 0x00a60 R 0x4
  LOAD 0x000000 0x00000000 0x00000000 0x10fd4 0x10fd4 R E 0x8000
  LOAD 0x011e98 0x00019e98 0x00019e98 0x003a8 0x003d8 RW 0x8000
  DYNAMIC 0x011ed8 0x00019ed8 0x00019ed8 0x00128 0x00128 RW 0x4
  NOTE 0x000134 0x00000134 0x00000134 0x00024 0x00024 R 0x4
  GNU_EH_FRAME 0x010f94 0x00010f94 0x00010f94 0x00014 0x00014 R 0x4
  GNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RWE 0x4
  GNU_RELRO 0x011e98 0x00019e98 0x00019e98 0x00168 0x00168 R 0x1

 Section to Segment mapping:
  Segment Sections...
   00 .ARM.exidx
   01 .note.gnu.build-id .hash .dynsym .dynstr .gnu.version .gnu.version_d .gnu.version_r .rel.dyn .rel.plt .init .plt .text .fini .rodata .ARM.extab .ARM.exidx .eh_frame_hdr .eh_frame
   02 .init_array .fini_array .jcr .data.rel.ro .dynamic .got .data .bss
   03 .dynamic
   04 .note.gnu.build-id
   05 .eh_frame_hdr
   06
   07 .init_array .fini_array .jcr .data.rel.ro .dynamic

readelf output on karmic version:
mcasadevall@dawn:/usr/lib/ure/lib$ readelf -l libgcc3_uno.so.karmic

Elf file type is DYN (Shared object file)
Entry point 0x2550
There are 7 program headers, starting at offset 52

Program Headers:
  Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
  EXIDX 0x00bb7c 0x0000bb7c 0x0000bb7c 0x00748 0x00748 R 0x4
  LOAD 0x000000 0x00000000 0x00000000 0x0c2c8 0x0c2c8 R E 0x8000
  LOAD 0x00ce88 0x00014e88 0x00014e88 0x003bc 0x003ec RW 0x8000
  DYNAMIC 0x00cec8 0x00014ec8 0x00014ec8 0x00138 0x00138 RW 0x4
  NOTE 0x000114 0x00000114 0x00000114 0x00024 0x00024 R 0x4
  GNU_STACK 0x000000 0x00000000 0x00000000 0x00000 0x00000 RW 0x4
  GNU_RELRO 0x00ce88 0x00014e88 0x00014e88 0x00178 0x00178 R 0x1

 Section to Segment mapping:
  Segment Sections...
   00 .ARM.exidx
   01 .note.gnu.build-id .hash .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_d .gnu.version_r .rel.dyn .rel.plt .init .plt .text .fini .rodata .ARM.extab .ARM.exidx .eh_frame
   02 .init_array .fini_array .jcr .data.rel.ro .dynamic .got .data .bss
   03 .dynamic
   04 .note.gnu.build-id
   05
   06 .init_array .fini_array .jcr .data.rel.ro .dynamic

eh_frame and eh_frame are parts of the ELF file that deal with exception handling, which seems to be related to our issues with the unwinder failures. I managed to force inclusion of .eh_frame_hdr in a karmic build, but it didn't produce a ...

A little more poking with readelf reveals that we don't actually have any data in the .eh_frame that exists on karmic/lucid:

mcasadevall@dawn:/usr/lib/ure/lib$ readelf -x.eh_frame ./libgcc3_uno.so.karmic

Hex dump of section '.eh_frame':
  0x0000c2c4 00000000 ....

mcasadevall@dawn:/usr/lib/ure/lib$ readelf -x.eh_frame ./libgcc3_uno.so.jaunty

Hex dump of section '.eh_frame':
  0x00010fa8 10000000 00000000 017a5200 027c0e01 .........zR..|..
  0x00010fb8 1b0c0d00 10000000 18000000 ecddffff ................
  0x00010fc8 28000000 00000000 00000000 (...........

I believe this is the root cause of our exceptions failing to unwind. The next step is to determine what data is in the eh_frame specifically in jaunty.

Turns out that readelf natively speaks .eh_frame

mcasadevall@dawn:/usr/lib/ure/lib$ readelf -Wwf ./libgcc3_uno.so.jaunty
Contents of the .eh_frame section:

00000000 00000010 00000000 CIE
  Version: 1
  Augmentation: "zR"
  Code alignment factor: 2
  Data alignment factor: -4
  Return address column: 14
  Augmentation data: 1b

  DW_CFA_def_cfa: r13 ofs 0

00000014 00000010 00000018 FDE cie=00000000 pc=0000edb0..0000edd8
  DW_CFA_nop
  DW_CFA_nop
  DW_CFA_nop

00000028 ZERO terminator

mcasadevall@dawn:/usr/lib/ure/lib$ readelf -Wwf ./libgcc3_uno.so.karmic
Contents of the .eh_frame section:

00000000 ZERO terminator

@Michael: I think there are two types of stack unwinding frames, .eh_frame and .debug_frame; relatively recently, gcc and gas were patched to generate only .debug_frame instead of both .eh_frame and .debug_frame, perhaps that's triggering the problem?

See http://gcc.gnu.org/ml/gcc-patches/2009-10/msg00818.html

So what might hit us in this case is presence of .eh_frame instead of .debug_frame, or perhaps a mix.

So scratch the last two comments; ARM uses .ARM.exidx and .ARM.extable instead of .eh_frame and friends; we should however be concerned that .eh_frame is present -- it shouldn't be.

Posting output from some investigations yesterday

> -----Original Message-----
> From: Matthew Gretton-Dann

[...]

> After some work with NCommander on IRC yesterday afternoon we
> identified a problem with privateSnippetExecutor being marked
> CANTUNWIND.
>
> Examination of the source shows that this is a piece of
> handwritten assembly which did not have unwind directives
> added to it (source snippet available at
> http://paste.ubuntu.com/391906/. NCommander tried to add
> unwind directives to it (see http://paste.ubuntu.com/391919/
> however this did not work.

[continues below]

[From http://paste.ubuntu.com/391906/:]

> ~/chroot/karmic/src/OOO320_m12/bridges/source/cpp_uno/gcc3_linux_arm$ cat armhelper.s
> @ ARM support code for OpenOffice C++/UNO bridging
> @
> @ Written by Peter Naulls <email address hidden>
> @ Modified by Caolan McNamara <email address hidden>
> .file "armhelper.s"
> .text
> .align 4
> .global privateSnippetExecutor
> .type privateSnippetExecutor, %function
> privateSnippetExecutor:
> stmfd sp!, {r0-r3} @ follow other parameters on stack
> mov r0, ip @ r0 points to functionoffset/vtable
> mov ip, sp @ fix up the ip
> stmfd sp!, {fp,ip,lr,pc} @ 8 x 4 => stack remains 8 aligned
> sub fp, ip, #4 @ set frame pointer
>
> add r1, sp, #16 @ r1 points to this and params
> bl cpp_vtable_call(PLT)
>
> add sp, sp, #32 @ restore stack
> ldr fp, [sp, #-32] @ restore frame pointer
> ldr pc, [sp, #-24] @ return

> src/OOO320_m12/bridges# cat source/cpp_uno/gcc3_linux_arm/armhelper.s
> @ ARM support code for OpenOffice C++/UNO bridging
> @
> @ Written by Peter Naulls <email address hidden>
> @ Modified by Caolan McNamara <email address hidden>
> .file "armhelper.s"
> .text
> .align 4
> .global privateSnippetExecutor
> .type privateSnippetExecutor, %function
> privateSnippetExecutor:
> .fnstart
> .save {r0, r1, r2, r3}
> stmfd sp!, {r0-r3} @ follow other parameters on stack
> mov r0, ip @ r0 points to functionoffset/vtable
> mov ip, sp @ fix up the ip
>
> .save {fp,ip,lr,pc}
> stmfd sp!, {fp,ip,lr,pc} @ 8 x 4 => stack remains 8 aligned
> sub fp, ip, #4 @ set frame pointer
>
> add r1, sp, #16 @ r1 points to this and params
> bl cpp_vtable_call(PLT)
>
> add sp, sp, #32 @ restore stack
> ldr fp, [sp, #-32] @ restore frame pointer
> ldr pc, [sp, #-24] @ return
> .fnend

[From http://paste.ubuntu.com/391919/:]

> src/OOO320_m12/bridges# cat source/cpp_uno/gcc3_linux_arm/armhelper.s
> @ ARM support code for OpenOffice C++/UNO bridging
> @
> @ Written by Peter Naulls <email address hidden>
> @ Modified by Caolan McNamara <email address hidden>
> .file "armhelper.s"
> .text
> .align 4
> .global privateSnippetExecutor
> .type privateSnippetExecutor, %function
> privateSnippetExecutor:
> .fnstart
> .save {r0, r1, r2, r3}
> stmfd sp!, {r0-r3} @ follow other parameters on stack
> mov r0, ip @ r0 points to functionoffset/vtable
> mov ip, sp @ fix up the ip
>
> .save {fp,ip,lr,pc}
> stmfd sp!, {fp,ip,lr,pc} @ 8 x 4 => stack remains 8 aligned
> sub fp, ip, #4 @ set frame pointer
>
> add r1, sp, #16 @ r1 points to this and params
> bl cpp_vtable_call(PLT)
>
> add sp, sp, #32 @ restore stack
> ldr fp, [sp, #-32] @ restore frame pointer
> ldr pc, [sp, #-24] @ return...

After yesterdays discussions on determining the CANTUNWIND status of parts of the OOo stack, I tried to build a version of libgcc that ignores CANTUNWIND by commenting out the following code segment from gcc-4.4-4.4.3/src/gcc/config/arm/unwind-arm.c.

  /* Can this frame be unwound at all? */
  if (eitp->content == EXIDX_CANTUNWIND)
    {
      UCB_PR_ADDR (ucbp) = 0;
      return _URC_END_OF_STACK;
    }

I successfully managed to build libgcc with this modification in place, but I still experience libuno crashes. In addition, I had earlier built UNO on karmic with jaunty's binutils (2.19 based) which predates the CANTUNWIND changes that landed in karmic. Now that readelf can read unwinder information, I'll rebuild karmic with binutils 2.19 and confirm that we don't have CANTUNWIND segments.

While I'm still sure we're dealing with an unwinder issue, I don't believe the issue alone is being caused by the CANTUNWIND segments.

ARM folks came up with this patch which seems to fix the issue:
--- armhelper.s 2007-12-12 15:35:44.000000000 +0000
+++ armhelper.s 2010-03-11 16:22:29.000000000 +0000
@@ -10,13 +10,5 @@
 privateSnippetExecutor:
         stmfd sp!, {r0-r3} @ follow other parameters on stack
  mov r0, ip @ r0 points to functionoffset/vtable
- mov ip, sp @ fix up the ip
- stmfd sp!, {fp,ip,lr,pc} @ 8 x 4 => stack remains 8 aligned
- sub fp, ip, #4 @ set frame pointer
-
- add r1, sp, #16 @ r1 points to this and params
- bl cpp_vtable_call(PLT)
-
- add sp, sp, #32 @ restore stack
- ldr fp, [sp, #-32] @ restore frame pointer
- ldr pc, [sp, #-24] @ return
+ mov r1, sp @ r1 points to this and params
+ b cpp_vtable_call(PLT)

Patch is in ooo-build, and should be in the next OOo upload. Marking fix commited for Lucid.

Unfortunately, we took a look at this again and the fix is obviously wrong: some arguments are pushed on the stack by privateSnippetExecutor but never removed; this is almost certain to lead to incorrect behaviour.

It looks like unwind support is needed for this after all.

Mandriva's Arnaud Patard proposed the following function body to me on IRC:
@
@ Written by Peter Naulls <email address hidden>
@ Modified by Caolan McNamara <email address hidden>
        .file "armhelper.s"
        .text
        .align 4
        .global privateSnippetExecutor
        .type privateSnippetExecutor, %function
privateSnippetExecutor:
        .fnstart
        stmfd sp!, {r0-r3, lr } @ follow other parameters on stack
        .save {r0-r3, lr}
        mov r0, ip @ r0 points to functionoffset/vtable
        mov r1, sp @ r1 points to this and params
        bl cpp_vtable_call(PLT)
        ldmfd sp!, {r0-r3, lr}
        bx lr
        .fnend

http://pastebin.mandriva.com/17521

NCommander gave me the patch at http://pastebin.ubuntu.com/395620/ and it's working here too.

The patch on http://pastebin.ubuntu.com/395620/ is probably the one to go for, since it preserves 64-bit stack alignment and will work for target functions which take more than 4 arguments including this (they end up in a contiguous slab on the stack).

The exception unwinder is also told not to bother restore r0-r3 since this isn't necessary to conform to the ABI.

New patch tested and commited in ooo-build. Just need an upload now.

there seem to be still problems (quote from the debian maintainer):

but I fear the change broke extensions:

Enabling: Presentation Minimizer
 Enabling: SunPresentationMinimizer.xcs
 Enabling: SunPresentationMinimizer.uno.so

ERROR: (com.sun.star.deployment.DeploymentException) { { Message = "An error
occurred while enabling: SunPresentationMinimizer.uno.so", Context =
(com.sun.star.uno.XInterface) @111730 }, Cause = (any) {
(com.sun.star.registry.CannotRegisterImplementationException) { { Message =
"ImplementationRegistration::registerImplementation() InvalidRegistryException
during registration (destination registry is read-only! cannot merge!)",
Context = (com.sun.star.uno.XInterface) @0 } } } }
 rollback...
  Disabling: SunPresentationMinimizer.xcs
  rollback finished.
An error occurred while enabling: SunPresentationMinimizer.uno.so

arch-indep extensions (e.g. wiki publisher) seem to work, so it can't be word
for word what it says...

@NCommander, can you confirm that the patch merged in ooo-build was this one http://pastebin.ubuntu.com/395620/ ? The patch in post #72 would likely barf on functions with more than 4 args.

Is there a debdiff or equivalent somewhere I can take a look at?

Dave Martin: Talked with _rene_ (OOo maintainer for Debian), and he confirms that its a problem on Debian with the latest patch but I couldn't reproduce on Ubuntu off a self-built tree. Will investigate further, but the patch should be up to date in git.

Michael,

The latest patch that you sent me which is in ooo-build went into the build uploaded today. Let me know if you have any new patches to add, I will be doing another upload in the next week before Beta 2 freeze and could add it then.

Thanks,

Chris

Note that oo.o failed to build on armel

Is this bug fixed now? I noticed that it built fine on armel after it was retried.

checked on current lucid. starting oowriter and working with it works

while the old jaunty copy isn't installed anymore, the copyright still refers to it.

Matthias,

Can you quote the part and what file it is in? I thought I removed it and I can't find it myself when looking for it just now.

Also for some reason the first time OOo 4ubuntu1 was attempted it failed and then it passed on retry. I didn't get to see how it failed, but when trying 4ubuntu2 it failed also. It hasn't been retried and I am not sure if it failed in the same manner. Is one of the buildds having trouble?

Compiling: i18npool/unxlngr/misc/localedata_others_version.c
: && LD_LIBRARY_PATH=${LD_LIBRARY_PATH+${LD_LIBRARY_PATH}:}/build/buildd/openoffice.org-3.2.0/ooo-build-3-2-0-7/build/OOO320_m12/solver/320/unxlngr.pro/lib ../../../unxlngr.pro/bin/saxparser en_AU en_AU.xml ../../../unxlngr.pro/misc/localedata_en_AU.cxx ../../../unxlngr.pro/bin/localedata_en_AU.rdb /build/buildd/openoffice.org-3.2.0/ooo-build-3-2-0-7/build/OOO320_m12/solver/320/unxlngr.pro/bin/types.rdb
/bin/bash: line 1: 15144 Segmentation fault LD_LIBRARY_PATH=${LD_LIBRARY_PATH+${LD_LIBRARY_PATH}:}/build/buildd/openoffice.org-3.2.0/ooo-build-3-2-0-7/build/OOO320_m12/solver/320/unxlngr.pro/lib ../../../unxlngr.pro/bin/saxparser en_AU en_AU.xml ../../../unxlngr.pro/misc/localedata_en_AU.cxx ../../../unxlngr.pro/bin/localedata_en_AU.rdb /build/buildd/openoffice.org-3.2.0/ooo-build-3-2-0-7/build/OOO320_m12/solver/320/unxlngr.pro/bin/types.rdb
dmake: Error code 139, while making '../../../unxlngr.pro/misc/localedata_en_AU.cxx'

Marking back as Fix Released since Matthias says he sees the copyright change now. I'm not sure why it wasn't showing up for him before.

If this same patch works for Karmic I can prepare a SRU. I have another unrelated fix that should go into a SRU as well.

@Chris: would love seeing it fixed in SRU, but please don't SRU oo.o just for that, it's a very high load on the archive/mirrors.

If another SRU is pending, fixing this armel uglyness would be very welcome! :-)