LTE: ISST:leeklp4 no mpath choices for install ubuntu 15.10

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1489379/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

------- Comment From <email address hidden> 2015-09-02 22:11 EDT-------
Today's work on this..

This is not so simple to "fix".

libchecktur.so really requires a dynamic libgcc because it uses pthreads (namely pthread_cancel).
it can go without a link-time dependency on libgcc *if* it's not built with -fexceptions.

it will run, but crash when pthread_cancel() is called.
I think this is already happening w/ other pieces of the code (libmultipath, iirc) - it's just luck it didn't crash yet.

Some testing/experiments.

$ cat test.c
#include <pthread.h>
#include <stdlib.h>

void * loop(void *arg) {
}

int main() {
pthread_t thread;
pthread_create(&thread, NULL, &loop, NULL);
pthread_cancel(thread); // this requires libgcc_s.so
return 0;
}

$ gcc -pthread -o test test.c

$ ./test; echo $?
0

$ ldd test
linux-vdso64.so.1 => (0x00003fff81730000)
libpthread.so.0 => /lib/powerpc64le-linux-gnu/libpthread.so.0 (0x00003fff816e0000)
libc.so.6 => /lib/powerpc64le-linux-gnu/libc.so.6 (0x00003fff81500000)
/lib64/ld64.so.2 (0x0000000050130000)

^ notice there's no dependency on libgcc_s at link-time, it's a run-time dlopen() in pthread_cancel_init()

$ find /lib -name libgcc_s.so.1
/lib/powerpc64le-linux-gnu/libgcc_s.so.1

On installer:

~ # ./test
libgcc_s.so.1 must be installed for pthread_cancel to work
Aborted

~ # find /lib -name libgcc_s.so.1
~ #

To be continued.

------- Comment From <email address hidden> 2015-09-03 10:18 EDT-------
Some developer-backed reference for this as well:

(In reply to comment #15)
> libchecktur.so really requires a dynamic libgcc because it uses pthreads
> (namely pthread_cancel).

"""
> Wonder if someone can explain why pthread_cancel_init() dlopens
> libgcc_s.so.1?

Thread cancellation requires unwinding which is implemented by
compiler helper routines.
"""
-- https://sourceware.org/ml/libc-help/2009-10/msg00023.html

------- Comment From <email address hidden> 2015-09-03 11:05 EDT-------
How to reproduce with QEMU/KVM (2 arguments as "-drive file=test.qcow2,serial=helloworld")

Force the path checker to tur:

~ # cat /etc/multipath.conf
defaults {
user_friendly_names yes
path_checker tur
}

blacklist_exceptions {
property ID_SERIAL
}

Notice the dlopen() error, and paths got no uuids, thus blacklisted:

~ # multipath -v3
...
<...> | sda: udev property ID_SERIAL whitelisted
...
<...> | sda: vendor = QEMU
<...> | sda: product = QEMU HARDDISK
<...> | sda: rev = 2.2.
<...> | sda: h:b:t:l = 0:0:1:0
...
<...> | sda: serial = helloworld
<...> | sda: get_state
<...> | loading /lib/multipath/libchecktur.so checker
<...> | A dynamic linking error occurred: (libgcc_s.so.1: cannot open shared object file: No such file or directory)
<...> | unloading tur checker
<...> | sda: path checker = (config file default)
<...> | sda: checker timeout = 30 ms (internal default)
<...> | sda: No checker selected
<...> | sdb: udev property ID_SERIAL whitelisted
...
<...> | sdb: vendor = QEMU
<...> | sdb: product = QEMU HARDDISK
<...> | sdb: rev = 2.2.
<...> | sdb: h:b:t:l = 0:0:2:0
...
<...> | sdb: serial = helloworld
<...> | sdb: get_state
<...> | loading /lib/multipath/libchecktur.so checker
<...> | A dynamic linking error occurred: (libgcc_s.so.1: cannot open shared object file: No such file or directory)
<...> | unloading tur checker
<...> | sdb: path checker = (config file default)
<...> | sdb: checker timeout = 30 ms (internal default)
<...> | sdb: No checker selected
<...> | loop0: blacklisted, udev property missing
...
===== paths list =====
uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev dev_st
0:0:1:0 sda 8:0 -1 undef faulty QEMU,QEMU HARDDISK running
0:0:2:0 sdb 8:16 -1 undef faulty QEMU,QEMU HARDDISK running
<...> | sda: orphan path, wwid blacklisted
<...> | sdb: orphan path, wwid blacklisted
...
~ #

------- Comment From <email address hidden> 2015-09-03 11:10 EDT-------
And if you manually put libgcc_s.so.1 (from the libgcc1 deb package), it works fine.

~ # ls /lib/powerpc64le-linux-gnu/libgcc_s.so.1
/lib/powerpc64le-linux-gnu/libgcc_s.so.1

~ # multipath -v3
...
<...> | sda: udev property ID_SERIAL whitelisted
...
<...> | sda: serial = helloworld
<...> | sda: get_state
<...> | loading /lib/multipath/libchecktur.so checker
<...> | sda: path checker = tur (config file default)
<...> | sda: checker timeout = 30 ms (internal default)
<...> | sda: state = up
<...> | sda: uid_attribute = ID_SERIAL (internal default)
<...> | sda: uid = 0QEMU_QEMU_HARDDISK_helloworld (udev)
<...> | sda: detect_prio = 1 (config file default)
<...> | sda: prio = const (internal default)
<...> | sda: prio args = (internal default)
<...> | sda: const prio = 1
<...> | sdb: udev property ID_SERIAL whitelisted
...
<...> | sdb: serial = helloworld
<...> | sdb: get_state
<...> | sdb: path checker = tur (config file default)
<...> | sdb: checker timeout = 30 ms (internal default)
<...> | sdb: state = up
<...> | sdb: uid_attribute = ID_SERIAL (internal default)
<...> | sdb: uid = 0QEMU_QEMU_HARDDISK_helloworld (udev)
<...> | sdb: detect_prio = 1 (config file default)
<...> | sdb: prio = co...

------- Comment From <email address hidden> 2015-09-03 11:11 EDT-------
How to put libgcc_s.so.1 (from the udeb package) in the installer environment,
for testing/work-around purposes:

1) Go to https://launchpad.net/ubuntu/wily/ppc64el/libgcc1/
2) Click the latest (top-most) "Version" link (e.g., 1:5.2.1-15ubuntu5)
from the Release "Pocket" (unless you know you should use Proposed)
3) Copy the link address of the ".deb" file in the "Downloadable files" section
(e.g., http://.../libgcc1_5.2.1-15ubuntu5_ppc64el.deb)

4) On the installer, right after network configuration (username configuration stage),
select "Go Back", and "Execute a shell"
5) Run the following commands:
~ # wget http://launchpadlibrarian.net/215539827/libgcc1_5.2.1-15ubuntu5_ppc64el.deb
~ # ar x libgcc1_*.deb
~ # xzcat data.tar.xz | tar x ./lib/powerpc64le-linux-gnu/libgcc_s.so.1 -C /
~ # exit

~ # ls /lib/powerpc64le-linux-gnu/libgcc_s.so.1
/lib/powerpc64le-linux-gnu/libgcc_s.so.1

The result with the previous test binary as well (multipath in previous comment):

~ # ./test
~ #

------- Comment From <email address hidden> 2015-09-03 11:13 EDT-------
Oops

> How to put libgcc_s.so.1 (from the udeb package) in the installer

s/udeb/libgcc1 deb/

------- Comment From <email address hidden> 2015-09-03 11:26 EDT-------
Fortunately enough, the *only* ELF (binary/shared-library) in the multipath-udeb that requires libgcc_s is... libchecktur.so
(C'mon.. otherwise we wouldn't be doing this :-)

$ dpkg-buildpackage
...
$ find debian/tmp-multipath-udeb/ \
| while read file; do \
file $file | grep -q ELF || continue
objdump -x $file | grep -q 'NEEDED.*libgcc_s' || continue
echo "$file needs libgcc_s"
done
debian/tmp-multipath-udeb/lib/multipath/libchecktur.so needs libgcc_s
$

------- Comment From <email address hidden> 2015-09-03 11:56 EDT-------
Okay, using a custom libchecktur.so built without -fexceptions..
It works fine in the installer environment, for its purposes.

This should be OK as there's no multipathd (which exercises the path checkers more) running in the installer.
I'll formalize/support this in another comment.

~ # ls /lib/powerpc64le-linux-gnu/libgcc_s.so.1
ls: /lib/powerpc64le-linux-gnu/libgcc_s.so.1: No such file or directory

~ # grep libgcc_s /lib/multipath/libchecktur.so
~ #

~ # grep libgcc_s /lib/multipath/libchecktur.so.orig
libgcc_s.so.1
~ #

~ # multipath -v3
...
<...> | sda: get_state
<...> | loading /lib/multipath/libchecktur.so checker
<...> | sda: path checker = tur (config file default)
...
<...> | sdb: get_state
<...> | sdb: path checker = tur (config file default)
...
===== paths list =====
uuid hcil dev dev_t pri dm_st chk_st vend/prod/re
0QEMU_QEMU_HARDDISK_helloworld 0:0:1:0 sda 8:0 1 undef ready QEMU,QEMU HA
0QEMU_QEMU_HARDDISK_helloworld 0:0:2:0 sdb 8:16 1 undef ready QEMU,QEMU HA
...
create: mpatha (0QEMU_QEMU_HARDDISK_helloworld) undef QEMU,QEMU HARDDISK
size=128G features='0' hwhandler='0' wp=undef
|-+- policy='round-robin 0' prio=1 status=undef
| `- 0:0:1:0 sda 8:0 undef ready running
`-+- policy='round-robin 0' prio=1 status=undef
`- 0:0:2:0 sdb 8:16 undef ready running
<...> | tur checker refcount 2
<...> | const prioritizer refcount 2
<...> | tur checker refcount 1
<...> | const prioritizer refcount 1
<...> | unloading const prioritizer
<...> | unloading tur checker
<...> | unloading directio checker

~ # echo $?
0

------- Comment From <email address hidden> 2015-09-03 12:25 EDT-------
> Okay, using a custom libchecktur.so built without -fexceptions..
> It works fine in the installer environment, for its purposes.

> This should be OK as there's no multipathd (which exercises the path checkers more) running in the installer.
> I'll formalize/support this in another comment.

The only places that pthread_cancel() is called in libchecktur.so (tur.c + libsg.c) are 2 functions in tur.c:
1) libcheck_free()
2) libcheck_check()

And their calls to pthread_cancel() are /not/ exercised when /not/ in daemon mode
(i.e., running as multipath, not multipathd; as in the installer).

It's only in daemon mode that the checker goes into async mode -- via get_state() -> checker_set_async().
- And it's only in async mode that libcheck_check() might call pthread_cancel(),
- And it's only in async mode that libcheck_check() might set 'holders' to a value
for which libcheck_free() might call pthread_cancel() (holders > 1).

file libmultipath/discovery.c:

get_state (struct path * pp, int daemon)
{
...
condlog(3, "%s: get_state", pp->dev);

if (!checker_selected(c)) {
...
select_checker(pp);
if (!checker_selected(c)) {
condlog(3, "%s: No checker selected", pp->dev);
return PATH_UNCHECKED;
}
...
if (daemon)
checker_set_async(c);

file libmultipath/checkers.c:

void checker_set_async (struct checker * c)
{
if (!c)
return;
c->sync = 0;
}

file libmultipath/checkers/tur.c:

extern int
libcheck_check (struct checker * c)
{
...
if (c->sync)
return tur_check(c->fd, c->timeout, c->message);

/*
* Async mode
*/
...
if (ct->running) {
...
if (tur_check_async_timeout(c)) {
condlog(3, "%d:%d: tur checker timeout",
TUR_DEVT(ct));
pthread_cancel(ct->thread);
...
/* Start new TUR checker */
...
ct->holders++;

int libcheck_init (struct checker * c)
{
...
ct->holders = 1;
...

void libcheck_free (struct checker * c)
...
ct->holders--;
...
if (holders)
pthread_cancel(thread);
...

------- Comment From <email address hidden> 2015-09-03 12:28 EDT-------
Given the supporting evidence provided, I'll work on a patch to build multipath-udeb without -fexceptions.

(In reply to comment #22)
> Fortunately enough, the *only* ELF (binary/shared-library) in the
> multipath-udeb that requires libgcc_s is... libchecktur.so

(In reply to comment #23)
> Okay, using a custom libchecktur.so built without -fexceptions..
> It works fine in the installer environment, for its purposes.
>
> This should be OK as there's no multipathd (which exercises the path
> checkers more) running in the installer.

(In reply to comment #24)
> The only places that pthread_cancel() is called in libchecktur.so (tur.c +
> libsg.c) are 2 functions in tur.c:
...
> And their calls to pthread_cancel() are /not/ exercised when /not/ in daemon
> mode
> (i.e., running as multipath, not multipathd; as in the installer).

I'm attaching a patch that solves the issue; verification steps below.
It disables -fexceptions when building multipath-udeb.

(I had in mind something more elegant, like only adding -fno-exceptions from debian/rules into DEB_CFLAGS_MAINT_APPEND as done w/ -static-libgcc, but that way it doesn't override the post-added -fexceptions in Makefile.inc).. and found no other variable that would override it w/ less changes. So it's adding a patch to Makefile.inc and it does the job.)

Some comments with more details/explanation of the issue were not mirrored yet; should be on their way.

Build verification:
------------------

Checking the build log, the 2 tur.o objects are built correctly:
1) multipath-udeb: with -static-libgcc and without -fexceptions
2) multipath deb: without -static-libgcc and with -fexceptions (default)

 $ dpkg-buildpackage 2>&1 | tee ../dpkg-buildpackage.log

 $ grep -- '-o tur.o' ../dpkg-buildpackage.log
 cc -g -O3 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -static-libgcc -D_FORTIFY_SOURCE=2 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector --param=ssp-buffer-size=4 -Wunused -Wstrict-prototypes -fPIC -DLIB_STRING=\"lib\" -I.. -c -o tur.o tur.c
 cc -g -O3 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -Wunused -Wstrict-prototypes -fPIC -DLIB_STRING=\"lib\" -I.. -c -o tur.o tur.c

Checking the build output, the ELF objects (don't) dyn-link to libgcc_s as appropriate:
1) multipath-udeb: ELF objects don't dyn-link with libgcc_s
2) multipath deb: ELF objects dyn-link with libgcc_s

 $ find debian/tmp-multipath-udeb/ | while read file; do file $file | grep -q ELF && objdump -x $file | grep -q 'NEEDED.*libgcc_s' && echo $file needs libgcc_s; done
 $

 $ find debian/tmp/ | while read file; do file $file | grep -q ELF && objdump -x $file | grep -q 'NEEDED.*libgcc_s' && echo $file needs libgcc_s; done
 debian/tmp/lib/libmultipath.so.0 needs libgcc_s
 debian/tmp/lib/multipath/libchecktur.so needs libgcc_s
 debian/tmp/sbin/multipathd needs libgcc_s
 $

Installer verification:
-----------------------

Configure multipath.conf for using tur with QEMU/KVM SCSI disks:

 ~ # cat <<EOF >/etc/multipath.conf <<EOF
 defaults {
   user_friendly_names yes
   path_checker tur
 }
 blacklist_exceptions {
   property ID_SERIAL
 }
 EOF

Test (Not OK):

 ~ # multipath -v3
 ...
 <...> | A dynamic linking error occurred: (libgcc_s.so.1: cannot open shared object file: No such file or directory)
 ...
 <...> | A dynamic linking error occurred: (libgcc_s.so.1: cannot open shared object file: No such file or directory)
 uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev dev_st
      0:0:1:0 sda 8:0 -1 undef faulty QEMU,QEMU HARDDISK running
      0:0:2:0 sdb 8:16 -1 undef faulty QEMU,QEMU HARDDISK running
 ...

Install the new udeb:

 ~ # udpkg -i multipath-udeb_0.5.0-7ubuntu3fexceptions1_ppc64el.udeb

Re-test (now OK):

 ~ # multipath -v3
 ...
 ===== paths list =====
 uuid hcil dev dev_t pri dm_st chk_st vend/prod/re
 0QEMU_QEMU_HARDDISK_hellowo...

The attachment "Disable -fexceptions when building multipath-udeb" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

Assigning to me and unsubscribing sponsors, I'll just do the review and upload the patch as necessary.

This bug was fixed in the package multipath-tools - 0.5.0-7ubuntu4

---------------
multipath-tools (0.5.0-7ubuntu4) wily; urgency=medium

  * Disable -fexceptions on multipath-udeb (LP: #1489379): the flag causes
    libchecktur.so to link with libgcc_s.so.1 (even with -static-libgcc),
    which is not available in the installer environment.
    - debian/patches/disable-fexceptions-udeb.patch: conditionally disable
      -fexceptions with CFLAGS_DISABLE_FEXCEPTIONS.
    - debian/rules: set CFLAGS_DISABLE_FEXCEPTIONS to build multipath-udeb.

 -- Mauricio Faria de Oliveira <email address hidden> Thu, 03 Sep 2015 07:29:59 -0400

------- Comment From <email address hidden> 2015-09-16 01:07 EDT-------
I can see the multipath choices on the installation menu and able to install the rootvg on one of the multipath disk using SAN storages. System boots up normal after all.

------- Comment From <email address hidden> 2015-09-18 21:27 EDT-------
*** Bug 129488 has been marked as a duplicate of this bug. ***