Getting error "ipconfig: no devices to configure" while trying to autoinstall in a VLAN env on s390x

I just did an autoinstall attempt with having removed all config snippets from the parm-file that are required for the VLAN:

--------%<----------------%<----------------%<----------------%<----------------%<--------
A manual update of the initial RAM-disk is required.
QETH device 0.0.c000:0.0.c001:0.0.c002 configured
Note: The initial RAM-disk must be updated for these changes to take effect:
- QETH device 0.0.c000:0.0.c001:0.0.c002
IP-Config: encc000 hardware address 12:34:45:67:89:0a mtu 1500
IP-Config: encc000 guessed broadcast address 10.123.124.255
IP-Config: encc000 complete:
address: 10.123.124.14 broadcast: 10.123.124.255 netmask: 255.255.255.0

gateway: 10.123.124.1 dns0 : 10.123.124.1 dns1 : 0.0.0.0

host : hostname
rootserver: 0.0.0.0 rootpath:
filename :
Connecting to install-server:80 (installer-server:80)

BusyBox v1.30.1 (Ubuntu 1:1.30.1-6ubuntu2) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs) [6nwget: can't connect to remote host (install-server): Connection timed
out
Unable to find a live file system on the network
--------%<----------------%<----------------%<----------------%<----------------%<--------

and now the device c000 and the interface encc000 get activated:

encc000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
link/ether 26:e4:7c:23:c1:c9 brd ff:ff:ff:ff:ff:ff
inet 10.245.236.14/24 brd 10.245.236.255 scope global encc000
valid_lft forever preferred_lft forever
inet6 fe80::24e4:7cff:fe23:c1c9/64 scope link
valid_lft forever preferred_lft forever

The installation can of course not be completed in this case (since the VLAN config is now missing), but it shows that this is only a problem with handling potential VLAN config in the early boot stage (casper?).

The part of this that doesn't make sense is "ip: can't find device 'encc000'" -- is that device just not appearing in time? It should be chzdev activated just before the initramfs code tries to set up the vlan, maybe it's a race? (is this behaviour consistent?). It's strange that using the nic without a vlan works but there is a bit more time (and a wait for udev, hmm) between the device getting chzdev-ed and used in that case.

The code in initramfs-tools hasn't changed at all here between groovy and hirsute so I don't know what's changed. We can always blame the kernel I guess.

Yes, this behavior is quite consistent - and it happens on different systems that require vlan configurations (tested on two systems - each several times - happened every time).
I may try on another or more systems - maybe under less load conditions ...

Well, I'm not sure if looking back to groovy is enough, I have to admit that I mainly used autoinstall on focal, and tried it here and there on other releases too, but then w/o vlan for just doing a quick test (I should have taken more time for tests with vlan though).

One of the two systems that I used for the test is the same I used before with autoinstall (and vlan), and the environment didn't really changed - so I'm sure that it's not due to changes in the hw, setup or environment itself. (On the 2nd system autoinstall was used for the 1st time).

The z system got moved a few month ago to a different data center, but that didn't changed anything internally with the OSA network adapters, their ports or configuration - there might have been changes in the external network (different switches etc.), but that shouldn't have such an impact. And btw. doing a manual/interactive installation (incl. vlan) works fine - and that is what I do very often - on all releases.

And yes, the test with just dropping the vlan config is interesting. That make me think that it must have to do with the vlan config and setup itself. Well, maybe a race - but that race got for some reason introduced at some point in time ...

I found an older (known to work) config in my notes for 20.04.1 - and I just retried it and can confirm that autoinstall (with vlan) works with 20.04.1 on the same system LPAR system.

I'll proceed now using newer and newer releases and see where issues start to occur ...

Summary of autoinstall w/ vlan tests:

*** 20.04.1:
I see these messages, but the device gets finally configured:
"chzdev: Unknown device type or device ID format: c000.2653"
"Use 'chzdev --help' for more information"
"QETH device 0.0.c000:0.0.c001:0.0.c002 configured"
"IP-Config: encc000.2653 hardware address d6:21:7e:b3:49:da mtu 1500"
"IP-Config: encc000.2653 guessed broadcast address 10.245.236.255"
"IP-Config: encc000.2653 complete:"
(there seems to be a second try to get the configured after the iso was downloaded, but that doesn't seem to harm:
"chzdev: Unknown device type or device ID format: c000.2653"
"Use 'chzdev --help' for more information"
"QETH device 0.0.c000:0.0.c001:0.0.c002 already configured" )
No further error visible.

*** 20.04.2:
Similar to 20.04.1
but I get later msgs like this:
"finish: subiquity/ErrorReporter/1619073531.572634459.unknown/add_info: written"
"to /var/crash/1619073531.572634459.unknown.crash"
probably LP#1924575

*** 20.10 (and later):
The device cannot be configured anymore:
"(initramfs) [6n"
"ip: SIOCGIFFLAGS: No such device"
"ip: can't find device 'encc000'"
"ipconfig: encc000.2653: SIOCGIFINDEX: No such device"
"ipconfig: no devices to configure"
"ipconfig: encc000.2653: SIOCGIFINDEX: No such device"
"ipconfig: no devices to configure"
"ipconfig: encc000.2653: SIOCGIFINDEX: No such device"
"ipconfig: no devices to configure"
"ipconfig: encc000.2653: SIOCGIFINDEX: No such device"
"ipconfig: no devices to configure"
"ipconfig: encc000.2653: SIOCGIFINDEX: No such device"
"ipconfig: no devices to configure"
"ipconfig: encc000.2653: SIOCGIFINDEX: No such device"
"ipconfig: no devices to configure"
"ipconfig: encc000.2653: SIOCGIFINDEX: No such device"
"ipconfig: no devices to configure"
"ipconfig: encc000.2653: SIOCGIFINDEX: No such device"
"ipconfig: no devices to configure"
"ipconfig: encc000.2653: SIOCGIFINDEX: No such device"
"ipconfig: no devices to configure"
"ipconfig: encc000.2653: SIOCGIFINDEX: No such device"
"ipconfig: no devices to configure"
"no search or nameservers found in /run/net-encc000.2653.conf /run/net-*.conf /ru"
n/net6-*.conf

Well in the focal -> groovy time we find this change (http://launchpadlibrarian.net/481474731/initramfs-tools_0.137ubuntu4_0.137ubuntu5.diff.gz):

  # activate non-autoconfigured s390x devices
  for dev in $DEVICE $DEVICE6 $IP6 $VLAN_LINK; do
+ # skip $DEVICE which is processed in $VLAN_LINK
+ echo ${VLAN} | grep -q ${dev} && continue
   case ${dev} in
   enc*)
    zdev=${dev#enc}

which I think is just wrong: it's trying to avoid calling chzdev -e on the encc000.2653 device but it also avoids calling it on encc000 :(

https://code.launchpad.net/~mwhudson/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+merge/401992 might fix this

Yes, that could be it - thx for digging into this!

Can you try https://people.canonical.com/~mwh/initrd.lp1924794? This is a patched initrd from the 21.04 release ISO, so the other files should also come from that ISO.

I've uploaded the fix to impish. The bug is present in groovy and hirsute but there's probably no reason to backport the fix -- we are unlikely to make release media for them again...

I agree, no reason to backport to groovy and hirsute.
We may just need to ensure that the problem not affect 20.04.3.

So the initramfs-tools with the fix has migrated now, the next impish daily should have it (serial 20210511 or later). Would you be able to test to see if that helps?

Yes, I can confirm that autoinstall of an s390x system, where the qeth device is attached to a VLAN (in my case an LPAR system) works with the updated initramfs that is included in the impish image with timestamp 20210511.

I've attached the entire console log.

Looks good - many thx!