[Ubuntu 18.04.01][BostonLC][mpt3sas] installer does not detect any LSI based SAS/md raid device

Default Comment by Bridge

$ dpkg -c /home/lp_archive/ubuntu/pool/main/l/linux/scsi-modules-4.15.0-23-generic-di_4.15.0-23.25_ppc64el.udeb |grep mpt3sas
$

in bionic mpt3sas is in message-modules

debian.master$ find d-i/ -type f | xargs grep mpt3sas
d-i/modules/message-modules:mpt3sas ?

but message-modules is not in build/pkg-lists/netboot/ppc64el.cfg I believe d-i needs to include the udeb for message-modules and rebuilt?

Re-assigning to Foundations to comment on Manoj's proposal in comment #3. If a different resolution is preferred, could the Foundations team clarify what that would be?
Thanks.

Ok it should be possible to add message-modules to the netboot image, yes.

------- Comment From <email address hidden> 2018-07-16 00:55 EDT-------
Hi,

Is the fix available for the re-verification? Please suggest

Status changed to 'Confirmed' because the bug affects multiple users.

I'm investigating an issue we're seeing on a couple ARM servers, and it looks like this maybe the same issue here. What it looks like is that the mpt3sas driver is being downloaded/loaded just before the partitioner starts. Since the kernel is configured to scan for SCSI devices asynchronously, the partitioner ends up being displayed before disk detection has completed. When the reporter goes "Back" and re-enters the partitioner, the disks now appear because scanning has completed in the meantime.

The proposal in comment #3 would pre-load the mpt3sas driver into the initrd instead of fetching it from the mirror. This would make it available earlier in the installer which, for this case, may give mpt3sas enough time to complete its asynchronous scan before getting to the partitioner phase. But, I don't think it fixes the general problem. I suspect systems with lots of disks and/or automated (preseeded) installs would still be at risk of running into this race.

Back-in-the-day, there was a scsi_wait_scan module that you could load, and insmod would block until all asynchronous scans were complete. That no longer exists, and I haven't been able to find a reasonable userspace replacement. One thing we *could* do is add scsi_mod.scan=sync to d-i before the "---", which would force synchronous scanning during install time, but leave asynchronous scanning enabled for the runtime.

The following hack resolves the issue for me on arm64:

--- build/util/grub-gencfg.orig 2018-07-17 23:19:53.101127509 +0000
+++ build/util/grub-gencfg 2018-07-17 22:44:09.465555019 +0000
@@ -144,6 +144,7 @@
     push @cmdline, @OPTS;
     push @cmdline, "rescue/enable=true" if $xattr{Rescue};
     push @cmdline, "speakup.synth=soft" if $xattr{Speach};
+ push @cmdline, "scsi_mod.scan=sync";
     push @cmdline, "---";
     push @cmdline, "quiet" if $xattr{Quiet};

I've uploaded a fixed systemd to cosmic and bionic for this (xenial will follow when I sort out the best way to backport the fix). dannf should be in a position to test this once a new d-i is built using the new udev-udeb.

Hello bugproxy, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I prepared a ppa build of d-i (ppa:dannf/test) and booted w/ apt-setup/proposed=true. It resolves the issue I was seeing on a HiSilicon D05 system w/ disks attached to an mpt3sas controller.

Hello bugproxy, or anyone else affected,

Accepted debian-installer into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/debian-installer/20101020ubuntu543.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

------- Comment From <email address hidden> 2018-07-23 04:10 EDT-------
With the given fix, installer is able to detect SAS / MD device normally with out screen back n forth. we can close this bug as fixed and tested.

------- Comment From <email address hidden> 2018-07-23 04:12 EDT-------
I used this link to verify the bug http://ports.ubuntu.com/ubuntu-ports/dists/bionic-proposed/main/installer-ppc64el/20101020ubuntu543.1/images/netboot/ubuntu-installer/ppc64el/

abdhalee, Looks like you have successfully verified, could you please change the tag to verification-done-bionic ?

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package debian-installer - 20101020ubuntu543.1

---------------
debian-installer (20101020ubuntu543.1) bionic; urgency=medium

  * Move master kernels to 4.15.0-29 for the 18.04.1 release.
  * Rebuild against new systemd for SCSI sync (LP: #1779815)
  * Fix permissions on the netboot.tar kernel (LP: #1781036)

 -- Adam Conrad <email address hidden> Sun, 22 Jul 2018 03:19:55 -0600

This bug was fixed in the package systemd - 237-3ubuntu10.3

---------------
systemd (237-3ubuntu10.3) bionic; urgency=medium

  * debian/extra/start-udev: Set scsi_mod scan=sync even if it's builtin
    to the kernel (we previously only set it in modprobe.d) LP: #1779815

 -- Adam Conrad <email address hidden> Fri, 20 Jul 2018 11:13:58 -0600

Hello bugproxy, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu21.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello! Could you please perform the same validation for xenial? We would like to have this released for the upcoming 16.04.5 point-release. Thank you!

------- Comment From <email address hidden> 2018-07-27 00:33 EDT-------
(In reply to comment #28)
> Hello! Could you please perform the same validation for xenial? We would
> like to have this released for the upcoming 16.04.5 point-release. Thank you!

I would like to verify the fix on xenial, should I have to get the build from xenial-proposed ?

it this correct place to get the build ?
http://ports.ubuntu.com/ubuntu-ports/dists/xenial-proposed/main/installer-ppc64el/current/images/netboot/ubuntu-installer/ppc64el/

Thanks
Abdul

------- Comment From <email address hidden> 2018-07-27 01:40 EDT-------
(In reply to comment #29)
> (In reply to comment #28)
> > Hello! Could you please perform the same validation for xenial? We would
> > like to have this released for the upcoming 16.04.5 point-release. Thank you!
>
> I would like to verify the fix on xenial, should I have to get the build
> from xenial-proposed ?
>
> it this correct place to get the build ?
> http://ports.ubuntu.com/ubuntu-ports/dists/xenial-proposed/main/installer-
> ppc64el/current/images/netboot/ubuntu-installer/ppc64el/
>
> Thanks
> Abdul

Yes. You can use the above link to get the build. Thanks

------- Comment From <email address hidden> 2018-07-27 02:19 EDT-------
Abdul, Seeteena, actually no, the installer there is too old, it does not include the proper systemd fixes :

http://ports.ubuntu.com/ubuntu-ports/dists/xenial-proposed/main/installer-ppc64el/current/images/udeb.list

especially :

udev-udeb 229-4ubuntu21.1 ppc64el

and we should have this one : 229-4ubuntu21.3

https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu21.3

sil2100: shouldn't debian-installer be rebuilt to take the fixed xenial udev-udeb into account as
it was for bionic ? !
https://launchpad.net/ubuntu/+source/debian-installer/20101020ubuntu543.1

I don't see any such step on xenial :

https://launchpad.net/ubuntu/+source/debian-installer/

Thanks,

F.

------- Comment From <email address hidden> 2018-07-27 03:14 EDT-------
It seems this is what would be needed : a rebuild of debian-installer.
I rebuild it based on xenial,xenial-updates, xenial-proposed :
$ grep udev build/dest/udeb.list
libudev1-udeb 229-4ubuntu21.3 ppc64el
udev-udeb 229-4ubuntu21.3 ppc64el

Now, maybe the final verification should be done with Canonical-produced binaries.

F.

Hello bugproxy, or anyone else affected,

Accepted debian-installer into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/debian-installer/20101020ubuntu451.24 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello bugproxy, or anyone else affected,

Accepted debian-installer into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/debian-installer/20101020ubuntu451.25 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Could you proceed with testing the debian-installer bits? Apologies that those only landed recently but they were waiting for a formal review (also, we had to bump some size constants anyway). Thank you!

Note that this bug (and thus, the fix) probably only applies to the HWE variant of d-i with the 4.15 kernel.

------- Comment From <email address hidden> 2018-07-29 09:30 EDT-------
Hi Fred,

since few drivers had to enabled for installation to succeed. can you please help translate workaround done with the internal build?

http://stade.lab.toulouse-stg.fr.ibm.com/~fbonnard/LP1779815/hwe-netboot/ubuntu-installer/ppc64el/

Any news?

Not to delay the xenial 16.04.5 release candidate creation any longer, I'll be releasing both the systemd and debian-installer packages to xenial-updates without verification. Please test the packages regardless and, in case the bug is not fixed with the changes, please re-open this bug and we can iterate and re-spin the images if needed.

This bug was fixed in the package systemd - 229-4ubuntu21.3

---------------
systemd (229-4ubuntu21.3) xenial; urgency=medium

  [ Dimitri John Ledkov ]
  * udev-udeb: ship modprobe.d snippet to force scsi_mod.scan=sync in d-i.

  [ Adam Conrad ]
  * debian/extra/start-udev: Set scsi_mod scan=sync even if it's builtin
    to the kernel (we previously only set it in modprobe.d) LP: #1779815

 -- Adam Conrad <email address hidden> Thu, 26 Jul 2018 04:00:09 -0600

This bug was fixed in the package debian-installer - 20101020ubuntu451.25

---------------
debian-installer (20101020ubuntu451.25) xenial; urgency=medium

  * Bump FLOPPY_SIZE on amd64, i386 and powerpc for linux-firmware growth.

debian-installer (20101020ubuntu451.24) xenial; urgency=medium

  [ Łukasz 'sil2100' Zemczak ]
  * Move master kernels to 4.4.0-131.
  * Move hwe kernels to 4.15.0-29.
  * Rebuild against new systemd for SCSI sync (LP: #1779815)
  * build/config/amd64/hwe-*.cfg, build/config/ppc64el/hwe-*.cfg:
    - Strip off .efi.signed for signed hwe kernels.

  [ Adam Conrad ]
  * Fix permissions on the netboot.tar kernel (LP: #1781036)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Fri, 27 Jul 2018 10:38:40 +0200

Łukasz, Adam, thanks a lot for your help.

I don't know if Abdul tested Ubuntu's build but he did test successfully with a build I did last week, when debian-installer was no yet rebuilt on Ubuntu's side. Indeed that made sense on hwe which he tested ok.

Thanks for taking this pro-actively in 16.04.5 updates already.

F.

This bug was fixed in the package systemd - 239-7ubuntu4

---------------
systemd (239-7ubuntu4) cosmic; urgency=medium

  * Workaround broken meson copying symlinked data files, as dangling symlinks.

 -- Dimitri John Ledkov <email address hidden> Wed, 22 Aug 2018 14:11:35 +0100