zVM Cloud Connector

zvmguestconfigure on SLES12 never completes

Bug #1814321 reported by Michael Friesenegger on 2019-02-01

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	zVM Cloud Connector	Fix Committed	Medium	Michael Friesenegger

Bug Description

zvmguestconfigure is stopped which causes cloud-init to not start and no login prompt to be shown. The following is the output of the 3270 console of a SLES12SP4 guest provisioned by z/VM Cloud Connector.

zvmguestconfigureÝ821¨: modprobe: FATAL: Module vmcp not found in directory /lib
/modules/4.12.14-94.41-default
Ý OK ¨ Started wicked AutoIPv4 supplicant service.
Ý OK ¨ Started wicked DHCPv4 supplicant service.
Ý OK ¨ Started wicked DHCPv6 supplicant service.
Starting wicked network management service daemon...
Ý OK ¨ Started wicked network management service daemon.
Starting wicked network nanny service...
Ý OK ¨ Started System Logging Service.
2019-02-01T14:31:51.239676-05:00 linux-gpr9 kernel: ap.3677f7: The hardware syst
em does not support AP instructions
2019-02-01T14:31:51.239998-05:00 linux-gpr9 kernel: systemd: 13 output lines sup
pressed due to ratelimiting
Ý OK ¨ Started Rotate log files.
Ý OK ¨ Started wicked network nanny service.
zvmguestconfigureÝ821¨: Downloading record 0001: cfgdrive.iso
zvmguestconfigureÝ821¨: Downloading record 0002: iucvauth.sh
zvmguestconfigureÝ821¨: Downloading record 0003: network.doscript
zvmguestconfigureÝ821¨: QETH device 0.0.1000:0.0.1001:0.0.1002 already configure
d
zvmguestconfigureÝ821¨: Remove all network devices
zvmguestconfigureÝ821¨: Warning: this will affect network connectivity!
zvmguestconfigureÝ821¨: Successfully removed device 0.0.1000 (eth0)
zvmguestconfigureÝ821¨: Scanning for network devices...
zvmguestconfigureÝ821¨: znetconf: Error: Failed to configure layer2=1
zvmguestconfigureÝ821¨: Successfully configured device 0.0.1000 (eth1000)
Ý OK ¨ Stopped wicked managed network interfaces.

Revision history for this message

Michael Friesenegger (mfriesenegger) wrote on 2019-02-01:

https://review.gerrithub.io/c/mfcloud/python-zvm-sdk/+/443058

Changed in python-zvm-sdk:
assignee:	nobody → Michael Friesenegger (mfriesenegger)

Revision history for this message

jichenjc (jichenjc) wrote on 2019-02-11:

Mike, really appreciate your findings
can you let us know what's the version of your SLES? I believe we did some test before on 12.1 and it works , will the new tools you suggested shipped by SLES 12.x and don't need to be installed ? thanks a lot ~

Revision history for this message

jichenjc (jichenjc) wrote on 2019-02-11:

forget above this, I learned this is SLES12SP4 but still curious on why wicked is ok while systemctl doesn't ...

Revision history for this message

Michael Friesenegger (mfriesenegger) wrote on 2019-02-12:

I am able to duplicate this issue with SLES12SP1. The problem is resolved with the proposed patch.

I am not able to answer your question why using wicked works while network does not. Here is the zvmguestconfigure output from /var/log/messages for sles12 sp1 without and with the patch.

sles12 sp1 without the patch

grep zvmguestconfigure /var/log/messages
2019-02-11T23:28:16.339651-05:00 linux-pyfd zvmguestconfigureÝ1108¨: Downloading
record 0001: cfgdrive.iso
2019-02-11T23:28:16.401189-05:00 linux-pyfd zvmguestconfigureÝ1108¨: Downloading
record 0002: iucvauth.sh
2019-02-11T23:28:16.448175-05:00 linux-pyfd zvmguestconfigureÝ1108¨: Downloading
record 0003: network.doscript
2019-02-11T23:28:20.525992-05:00 linux-pyfd zvmguestconfigureÝ1108¨: Remove all
network devices
2019-02-11T23:28:20.526202-05:00 linux-pyfd zvmguestconfigureÝ1108¨: Warning: th
is will affect network connectivity!
2019-02-11T23:28:20.581358-05:00 linux-pyfd zvmguestconfigureÝ1108¨: Successfull
y removed device 0.0.1000 (eth0)
2019-02-11T23:28:24.637109-05:00 linux-pyfd zvmguestconfigureÝ1108¨: Scanning fo
r network devices...
2019-02-11T23:28:24.679149-05:00 linux-pyfd zvmguestconfigureÝ1108¨: znetconf: E
rror: Failed to configure layer2=1
2019-02-11T23:28:24.716295-05:00 linux-pyfd zvmguestconfigureÝ1108¨: Successfull
y configured device 0.0.1000 (eth1000)

sles12 sp1 with the patch

grep zvmguestconfigure /var/log/messages
2019-02-11T23:46:23.880869-05:00 linux-pyfd zvmguestconfigureÝ1092¨: Downloading
record 0001: cfgdrive.iso
2019-02-11T23:46:23.957468-05:00 linux-pyfd zvmguestconfigureÝ1092¨: Downloading
record 0002: iucvauth.sh
2019-02-11T23:46:24.005019-05:00 linux-pyfd zvmguestconfigureÝ1092¨: Downloading
record 0003: network.doscript
2019-02-11T23:46:28.137240-05:00 linux-pyfd zvmguestconfigureÝ1092¨: Remove all
network devices
2019-02-11T23:46:28.137493-05:00 linux-pyfd zvmguestconfigureÝ1092¨: Warning: th
is will affect network connectivity!
2019-02-11T23:46:28.182154-05:00 linux-pyfd zvmguestconfigureÝ1092¨: Successfull
y removed device 0.0.1000 (eth0)
2019-02-11T23:46:32.243410-05:00 linux-pyfd zvmguestconfigureÝ1092¨: Scanning fo
r network devices...
2019-02-11T23:46:32.280486-05:00 linux-pyfd zvmguestconfigureÝ1092¨: znetconf: E
rror: Failed to configure layer2=1
2019-02-11T23:46:32.332017-05:00 linux-pyfd zvmguestconfigureÝ1092¨: Successfull
y configured device 0.0.1000 (eth1000)
2019-02-11T23:46:43.481918-05:00 linux-pyfd zvmguestconfigureÝ1092¨: lo
up
2019-02-11T23:46:43.482166-05:00 linux-pyfd zvmguestconfigureÝ1092¨: eth1000
up
2019-02-11T23:46:43.484633-05:00 linux-pyfd zvmguestconfigureÝ1092¨: zvmguestcon
figure has successfully processed the reader files.

As you can see, the output from sles12sp1 with the patch allows the network devices lo and eth1000 to come up as well as the message "zvmguestconfigure has successfully processed the reader files." which is the last message before setupIso is run.

Is there additional information that I can provide as you review the patch?

I am able to duplicate this issue with SLES12SP1.  The problem is resolved with the proposed patch.

I am not able to answer your question why using wicked works while network does not. Here is the zvmguestconfigure output from /var/log/messages for sles12 sp1 without and with the patch.

sles12 sp1 without the patch

grep zvmguestconfigure /var/log/messages  
2019-02-11T23:28:16.339651-05:00 linux-pyfd zvmguestconfigureÝ1108¨: Downloading
 record 0001: cfgdrive.iso  
2019-02-11T23:28:16.401189-05:00 linux-pyfd zvmguestconfigureÝ1108¨: Downloading
 record 0002: iucvauth.sh  
2019-02-11T23:28:16.448175-05:00 linux-pyfd zvmguestconfigureÝ1108¨: Downloading
 record 0003: network.doscript  
2019-02-11T23:28:20.525992-05:00 linux-pyfd zvmguestconfigureÝ1108¨: Remove all 
network devices  
2019-02-11T23:28:20.526202-05:00 linux-pyfd zvmguestconfigureÝ1108¨: Warning: th
is will affect network connectivity!  
2019-02-11T23:28:20.581358-05:00 linux-pyfd zvmguestconfigureÝ1108¨: Successfull
y removed device 0.0.1000 (eth0)  
2019-02-11T23:28:24.637109-05:00 linux-pyfd zvmguestconfigureÝ1108¨: Scanning fo
r network devices...  
2019-02-11T23:28:24.679149-05:00 linux-pyfd zvmguestconfigureÝ1108¨: znetconf: E
rror: Failed to configure layer2=1  
2019-02-11T23:28:24.716295-05:00 linux-pyfd zvmguestconfigureÝ1108¨: Successfull
y configured device 0.0.1000 (eth1000)

sles12 sp1 with the patch

grep zvmguestconfigure /var/log/messages  
2019-02-11T23:46:23.880869-05:00 linux-pyfd zvmguestconfigureÝ1092¨: Downloading
 record 0001: cfgdrive.iso  
2019-02-11T23:46:23.957468-05:00 linux-pyfd zvmguestconfigureÝ1092¨: Downloading
 record 0002: iucvauth.sh  
2019-02-11T23:46:24.005019-05:00 linux-pyfd zvmguestconfigureÝ1092¨: Downloading
 record 0003: network.doscript  
2019-02-11T23:46:28.137240-05:00 linux-pyfd zvmguestconfigureÝ1092¨: Remove all 
network devices  
2019-02-11T23:46:28.137493-05:00 linux-pyfd zvmguestconfigureÝ1092¨: Warning: th
is will affect network connectivity!  
2019-02-11T23:46:28.182154-05:00 linux-pyfd zvmguestconfigureÝ1092¨: Successfull
y removed device 0.0.1000 (eth0)  
2019-02-11T23:46:32.243410-05:00 linux-pyfd zvmguestconfigureÝ1092¨: Scanning fo
r network devices...  
2019-02-11T23:46:32.280486-05:00 linux-pyfd zvmguestconfigureÝ1092¨: znetconf: E
rror: Failed to configure layer2=1  
2019-02-11T23:46:32.332017-05:00 linux-pyfd zvmguestconfigureÝ1092¨: Successfull
y configured device 0.0.1000 (eth1000)  
2019-02-11T23:46:43.481918-05:00 linux-pyfd zvmguestconfigureÝ1092¨: lo         
     up  
2019-02-11T23:46:43.482166-05:00 linux-pyfd zvmguestconfigureÝ1092¨: eth1000    
     up  
2019-02-11T23:46:43.484633-05:00 linux-pyfd zvmguestconfigureÝ1092¨: zvmguestcon
figure has successfully processed the reader files.

Is there additional information that I can provide as you review the patch?

Revision history for this message

jichenjc (jichenjc) wrote on 2019-02-12:

Thanks for the detailed information, let me merge the code first and see why we havn't have the issue later , in the mean time need dig more on the command used.. thanks for the patch

jichenjc (jichenjc) on 2019-03-15

Changed in python-zvm-sdk:
status:	New → Fix Committed
importance:	Undecided → Medium

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.