Several blockprobe errors if trying to install the groovy daily live on LPAR

Btw. after clicking 'Continue' I am able to complete the installation, despite all the files/crashes in /var/crash.

For some reason, udev has not yet created partition node entries for
all of the member paths.

So, /dev/sda and /dev/sdc are two paths to the same lun, and when we dump
udev, we get:

/dev/sda
/dev/sdc

but no /dev/sda1 or /dev/sdc1;

Now, the partition exists, /dev/dm-1; however, there should also be
the /dev/sda1 device node.

I'm not sure why it's not there.

The current probe code assumes that we'll see both a /dev/sda1 as well as
the device-mapper disk for multipath partitions.

It'll take some to figure out if we can construct type:partitions entries
for multipath partitions if we're missing sda1 in the list of partitions
from udev.

Just a side not: this doesn't happen with 20.04.1 - only on 20.10 installation.

Even after upgrading to subiquity 20.09.1 (2027) I'm still running still into this situation.
The installer crashes, but it allows me to 'Continue' and I can complete the installation.
I've attached the /var/crash and /var/log content again (this time like they are with 20.09.1).

'+1' comment #3 this happens with zFCP/multipath only (normal DASD installations are fine afaiks).

It's still the case with the latest QA tracker image: Version: 20200927

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1893818

I was asked to try on 20.04.1 with latest subiquity from edge and can confirm that it still work in this case (on 20.04.1):

Subiquity help menu: 20.07.1+git2.5de9df3e
root@ubuntu-server:/# snap list | grep subiquity
subiquity 20.07.1+git2.5de9df3e 1969 latest/stable/… canonical* classic
root@ubuntu-server:/# snap refresh --edge subiquity
subiquity (edge) 20.09.1+git84.399e711b from Canonical* refreshed
root@ubuntu-server:/# snap list | grep subiquity
subiquity 20.09.1+git84.399e711b 2082 latest/stable/… canonical* classic
root@ubuntu-server:/#
Subiquity help menu: 20.09.1+git84.399e711b

So I tried now a 20.04.1 installation but upgraded to multipath-tools (incl. dependencies) from groovy.
It's obviously a bit ugly to upgrade to multipath-tools from groovy in a focal install, since further packages are pulled in and services need to be restarted (which didn't successfully work on first try) etc. -- anyway, finally I was able to proceed with the installation and it looks like it failed in a similar way compared to a groovy installation:
" 2020-10-09 13:06:10,740 ERROR root:39 finish: subiquity/Filesystem/_probe/probe_once: FAIL: Invalid dep_id (partition-sda1) not in storage config"

For the steps I took see the attached file - and /var/log and /var/crash in the attached tgz in the next comment.
(Please notice that several ui crash files are in /var/crash - this is only because I restarted the installer - but the block-probe-fail crash files are there, too.)

Subscribed ubuntu-server as now there is an indicator that this could be a regression in multipath-tools, and we need to discuss how to tackle it.

Some aspects of this don't make much sense to me. /dev/sda1 is not managed by udev, it's created by the kernel / devtmpfs. If it's not there on the filesystem something really wacky is going on. But it pretty clearly isn't in the udev database, which is almost as strange...

Status changed to 'Confirmed' because the bug affects multiple users.

OK, I was wrong /dev/sda1 (and similar) is really not present. If I run "partprobe /dev/sd*" things start behaving but that should in no way be needed...

OK some more clues: if i deactivate and reactivate the zdevs, the nodes for the partitions do not appear. If I remove multipath-tools and deactivate and reactivate the zdevs, the nodes for the partitions *do* appear. If I install multipath-tools and remove /lib/udev/rules.d/60-multipath.rules and deactivate and reactivate the zdevs, the nodes for the partitions still appear.

60-multipath.rules is the same between focal and groovy though. Good night for now!

Aaah so it turns out there has always been a udev rule that should have removed the /dev/sda1 etc nodes for a drive that is multipathed, but thanks to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=959727 it didn't work. So I guess probert / curtin need updates to cope with this new reality but given how close the release is perhaps we just break the udev rule again for compatibility and fix this early next cycle?

Good catch! Given where we are in the cycle I'm definitely for disabling the rule again in Groovy and fix this properly early in the HH cycle.

Per IRC discussion, the plan is to workaround the issue in Groovy by disabling the udev rule in the live installer environment *only*. This should minimize the overall risk of regressions.

This will require a change in the image build scripts.

This bug was fixed in the package livecd-rootfs - 2.691

---------------
livecd-rootfs (2.691) groovy; urgency=medium

  * Remove 68-del-part-nodes.rules from installer squashfs to work around it
    breaking curtin. (LP: #1893818)

 -- Michael Hudson-Doyle <email address hidden> Thu, 15 Oct 2020 08:56:11 +1300

I can confirm that it's fixed with image:
http://cdimage.ubuntu.com/ubuntu-server/daily-live/20201014.1/groovy-live-server-s390x.iso
I did two installations - on z/VM and on LPAR (both of course zFCP) - and the blockprobe issues no longer occurred.

This bug is fixed with commit e113b055 to curtin on branch master.
To view that commit see the following URL:
https://git.launchpad.net/curtin/commit/?id=e113b055

https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-rootfs/+merge/402232

As discussed with fheimes no extra task on probert needed.
But maybe the status of curtin "fix committed" is by now released or needs to be re-checked?

This bug is believed to be fixed in curtin in version 21.3. If this is still a problem for you, please make a comment and set the state back to New

Thank you.