Bulk commissioning - failure to mount root device from trusty ephemeral image - drops to initramfs shell
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Invalid
|
Undecided
|
Unassigned |
Bug Description
I did a bulk commission of 5 nodes, one of the nodes ended up failing to mount /. It dropped to ramfs shell. I checked another node and it was also in ramfs shell. The other 3 nodes never finished commissioning either. This was with maas 1.7.1 RC1 and daily images:
ii maas 1.7.1~rc1+
This was from the boot console logs:
=======
Begin: Loading essential drivers[ 49.264063] IPv6: ADDRCONF(
... done.
Beg[ 49.351478] e1000: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
in: Running /scr[ 49.456905] IPv6: ADDRCONF(
ipts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... IP-Config: eth1 hardware address 00:22:99:e0:07:76 mtu 1500 DHCP RARP
hostname hayward-52 IP-Config: no response after 2 secs - giving up
IP-Config: eth1 [ 52.255141] Loading iSCSI transport class v2.0-870.
hardware address[ 52.332219] iscsi: registered transport (tcp)
00:22:99:e0:07:76 mtu 1500 DHCP RARP
hostname hayward-52 hostname hayward-52 IP-Config: eth1 complete (dhcp from 10.245.0.10):
address: 10.245.38.242 broadcast: 10.245.63.255 netmask: 255.255.192.0
gateway: 10.245.0.1 dns0 : 10.[ 52.648483] scsi32 : iSCSI Initiator over TCP/IP
245.0.10 dns1 : 0.0.0.0
domain : oil
rootserver: 10.245.0.10 rootpath:
filename : pxelinux.0
iscsistart: Logging into iqn.2004-
iscsistart: version 2.0-873
iscsistart: Connection1:0 to [target: iqn.2004-
[ 53.590125] scsi 32:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
[ 53.688619] scsi 32:0:0:0: Attached scsi generic sg2 type 12
[ 54.510114] scsi 32:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
[ 54.608655] sd 32:0:0:1: Attached scsi generic sg3 type 0
iscsistart: Logging into iqn.2004-
iscsistart: version 2.0-873
iscsistart: initiator reported error (15 - session exists)
done.
[ 55.343087] sd 32:0:0:1: [sdc] 2883584 512-byte logical blocks: (1.47 GB/1.37 GiB)
[ 55.434088] sd 32:0:0:1: [sdc] 4096-byte physical blocks
[ 56.049643] random: nonblocking pool is initialized
[ 56.527787] sd 32:0:0:1: [sdc] Write Protect is on
[ 57.036697] sd 32:0:0:1: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 58.445717] sdc: unknown partition table
[ 61.602703] sd 32:0:0:1: [sdc] Attached SCSI disk
Begin: Loading e[ 84.705130] hidraw: raw HID events driver (C) Jiri Kosina
ssential drivers[ 84.781226] usbcore: registered new interface driver usbhid
[ 84.862398] usbhid: USB HID core driver
... done.
Begin: Running /scripts/
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... IP-Config: eth1 hardware address 00:22:99:e0:07:76 mtu 1500 DHCP RARP
hostname hayward-52 IP-Config: no response after 2 secs - giving up
IP-Config: eth1 hardware address 00:22:99:e0:07:76 mtu 1500 DHCP RARP
hostname hayward-52 hostname hayward-52 IP-Config: eth1 complete (dhcp from 10.245.0.10):
address: 10.245.38.242 broadcast: 10.245.63.255 netmask: 255.255.192.0
gateway: 10.245.0.1 dns0 : 10.245.0.10 dns1 : 0.0.0.0
domain : oil
rootserver: 10.245.0.10 rootpath:
filename : pxelinux.0
iscsistart: Logging into iqn.2004-
iscsistart: version 2.0-873
iscsistart: Connection1:0 to [target: iqn.2004-
iscsistart: Logging into iqn.2004-
iscsistart: version 2.0-873
iscsistart: initiator reported error (15 - session exists)
done.
Gave up waiting for root device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Check root= (did the system wait for the right device?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/disk/
BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash)
Enter 'help' for a list of built-in commands.
(initramfs)
=======
description: | updated |
summary: |
- Bulk commissioning fails to mount / from trusty ephemeral image - drops - to initramfs shell + Bulk commissioning - failure to mount root device from trusty ephemeral + image - drops to initramfs shell |
tags: | added: oil |
Changed in maas: | |
milestone: | none → 1.7.2 |
Changed in maas: | |
milestone: | 1.7.2 → 1.7.3 |
So all of the nodes failed to commission?
Did you restart commissioning and they were able to complete? Or are the nodes still not able to complete commissioning?
Maybe try to restart tgt to see if that helps.