Bulk commissioning - failure to mount root device from trusty ephemeral image - drops to initramfs shell

Bug #1402124 reported by Larry Michel
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MAAS
Invalid
Undecided
Unassigned

Bug Description

I did a bulk commission of 5 nodes, one of the nodes ended up failing to mount /. It dropped to ramfs shell. I checked another node and it was also in ramfs shell. The other 3 nodes never finished commissioning either. This was with maas 1.7.1 RC1 and daily images:

ii maas 1.7.1~rc1+bzr3313-0ubuntu1~trusty1 all MAAS server all-in-one metapackage

This was from the boot console logs:
=================================================================================
Begin: Loading essential drivers[ 49.264063] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
 ... done.
Beg[ 49.351478] e1000: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
in: Running /scr[ 49.456905] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
ipts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... IP-Config: eth1 hardware address 00:22:99:e0:07:76 mtu 1500 DHCP RARP
hostname hayward-52 IP-Config: no response after 2 secs - giving up
IP-Config: eth1 [ 52.255141] Loading iSCSI transport class v2.0-870.
hardware address[ 52.332219] iscsi: registered transport (tcp)
 00:22:99:e0:07:76 mtu 1500 DHCP RARP
hostname hayward-52 hostname hayward-52 IP-Config: eth1 complete (dhcp from 10.245.0.10):
 address: 10.245.38.242 broadcast: 10.245.63.255 netmask: 255.255.192.0
 gateway: 10.245.0.1 dns0 : 10.[ 52.648483] scsi32 : iSCSI Initiator over TCP/IP
245.0.10 dns1 : 0.0.0.0
 domain : oil
 rootserver: 10.245.0.10 rootpath:
 filename : pxelinux.0
iscsistart: Logging into iqn.2004-05.com.ubuntu:maas:ephemeral-ubuntu-amd64-generic-trusty-daily 10.245.0.10:3260,1
iscsistart: version 2.0-873
iscsistart: Connection1:0 to [target: iqn.2004-05.com.ubuntu:maas:ephemeral-ubuntu-amd64-generic-trusty-daily, portal: 10.245.0.10,3260] through [iface: default] is operational now
[ 53.590125] scsi 32:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5
[ 53.688619] scsi 32:0:0:0: Attached scsi generic sg2 type 12
[ 54.510114] scsi 32:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5
[ 54.608655] sd 32:0:0:1: Attached scsi generic sg3 type 0
iscsistart: Logging into iqn.2004-05.com.ubuntu:maas:ephemeral-ubuntu-amd64-generic-trusty-daily 10.245.0.10:3260,1
iscsistart: version 2.0-873
iscsistart: initiator reported error (15 - session exists)
done.
[ 55.343087] sd 32:0:0:1: [sdc] 2883584 512-byte logical blocks: (1.47 GB/1.37 GiB)
[ 55.434088] sd 32:0:0:1: [sdc] 4096-byte physical blocks
[ 56.049643] random: nonblocking pool is initialized
[ 56.527787] sd 32:0:0:1: [sdc] Write Protect is on
[ 57.036697] sd 32:0:0:1: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 58.445717] sdc: unknown partition table
[ 61.602703] sd 32:0:0:1: [sdc] Attached SCSI disk
Begin: Loading e[ 84.705130] hidraw: raw HID events driver (C) Jiri Kosina
ssential drivers[ 84.781226] usbcore: registered new interface driver usbhid
[ 84.862398] usbhid: USB HID core driver
 ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... IP-Config: eth1 hardware address 00:22:99:e0:07:76 mtu 1500 DHCP RARP
hostname hayward-52 IP-Config: no response after 2 secs - giving up
IP-Config: eth1 hardware address 00:22:99:e0:07:76 mtu 1500 DHCP RARP
hostname hayward-52 hostname hayward-52 IP-Config: eth1 complete (dhcp from 10.245.0.10):
 address: 10.245.38.242 broadcast: 10.245.63.255 netmask: 255.255.192.0
 gateway: 10.245.0.1 dns0 : 10.245.0.10 dns1 : 0.0.0.0
 domain : oil
 rootserver: 10.245.0.10 rootpath:
 filename : pxelinux.0
iscsistart: Logging into iqn.2004-05.com.ubuntu:maas:ephemeral-ubuntu-amd64-generic-trusty-daily 10.245.0.10:3260,1
iscsistart: version 2.0-873
iscsistart: Connection1:0 to [target: iqn.2004-05.com.ubuntu:maas:ephemeral-ubuntu-amd64-generic-trusty-daily, portal: 10.245.0.10,3260] through [iface: default] is operational now
iscsistart: Logging into iqn.2004-05.com.ubuntu:maas:ephemeral-ubuntu-amd64-generic-trusty-daily 10.245.0.10:3260,1
iscsistart: version 2.0-873
iscsistart: initiator reported error (15 - session exists)
done.
Gave up waiting for root device. Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
   - Check root= (did the system wait for the right device?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/disk/by-path/ip-10.245.0.10:3260-iscsi-iqn.2004-05.com.ubuntu:maas:ephemeral-ubuntu-amd64-generic-trusty-daily-lun-1 does not exist. Dropping to a shell!

BusyBox v1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)
=================================================================================

Tags: oil
Larry Michel (lmic)
description: updated
Larry Michel (lmic)
summary: - Bulk commissioning fails to mount / from trusty ephemeral image - drops
- to initramfs shell
+ Bulk commissioning - failure to mount root device from trusty ephemeral
+ image - drops to initramfs shell
tags: added: oil
Revision history for this message
Blake Rouse (blake-rouse) wrote :

So all of the nodes failed to commission?

Did you restart commissioning and they were able to complete? Or are the nodes still not able to complete commissioning?

Maybe try to restart tgt to see if that helps.

Changed in maas:
status: New → Incomplete
Christian Reis (kiko)
Changed in maas:
milestone: none → 1.7.2
Revision history for this message
Jason Hobbs (jason-hobbs) wrote :

The problem here was our MAAS server was running on 100mbit. We can do bulk commission fine now on a 1gbit connection.

Changed in maas:
status: Incomplete → Invalid
Changed in maas:
milestone: 1.7.2 → 1.7.3
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.