Test Procedure with KVM guests + iPXE ===================================== - 2 guests: iSCSI target/server and iSCSI initiator/client. - 1 bridge for iSCSI traffic (virbr-iscsi, new), static ip. - 1 bridge for internet access (virbr0, exists), dhcp ip. Host: ---- Configure the iSCSI bridge and QEMU access in the host: $ sudo ip link add dev virbr-iscsi type bridge $ sudo ip link set dev virbr-iscsi up $ echo 'allow virbr-iscsi' | sudo tee -a /etc/qemu/bridge.conf $ sudo chmod +s /usr/lib/qemu/qemu-bridge-helper iSCSI target: ------------ This guest serves an iSCSI target with one LUN in iSCSI NIC with IP 10.0.0.1 for IP 10.0.0.2. Install/boot this guest: $ qemu-img create -f qcow2 guest-iscsi-target.qcow2 16g $ qemu-system-x86_64 \ -nodefaults \ -enable-kvm \ -smp 2 -m 4096 \ -serial stdio \ -vga virtio \ -display vnc=0.0.0.0:2 \ -netdev bridge,id=bridge-world,br=virbr0 \ -netdev bridge,id=bridge-iscsi,br=virbr-iscsi \ -device virtio-net-pci,netdev=bridge-world,id=nic-world,mac=52:54:00:00:00:11 \ -device virtio-net-pci,netdev=bridge-iscsi,id=nic-iscsi,mac=52:54:00:00:00:22 \ -drive file=guest-iscsi-target.qcow2,if=virtio \ -drive file=$RELEASE-server-amd64.iso,media=cdrom,read-only,if=scsi \ -boot once=d Configure iSCSI NIC: $ cat < link/ether 52:54:00:00:00:22 brd ff:ff:ff:ff:ff:ff inet 10.0.0.1/24 brd 10.0.0.255 scope global ens4 ... Configure iSCSI target/lun: # apt-get install -y tgt # mkdir /var/lib/iscsi # dd if=/dev/zero of=/var/lib/iscsi/disk bs=1 count=0 seek=8G # tgtadm --lld iscsi --op new --mode target --tid 1 -T iqn.2019-03.com.example:target1 # tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /var/lib/iscsi/disk # tgtadm --lld iscsi --op bind --mode target --tid 1 -I 10.0.0.2 # tgt-admin --dump >/etc/tgt/conf.d/target1.conf iSCSI initiator: --------------- This guest first boots iPXE to configure iBFT, and then boots/chainloads to debian-installer. The netboot initrd does not contain all patched udebs, so download and install disk-detect and partman-iscsi from the PPA during the install. $ wget http://boot.ipxe.org/ipxe.lkrn $ wget http://ppa.launchpad.net/mfo/lp1817321v3/ubuntu/dists/$RELEASE/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/{linux,initrd.gz} $ python3 -m http.server & Serving HTTP on 0.0.0.0 port 8000 ... $ qemu-system-x86_64 \ -nodefaults \ -enable-kvm \ -smp 2 -m 4096 \ -serial stdio \ -vga virtio \ -display vnc=0.0.0.0:1 \ -netdev bridge,id=bridge-world,br=virbr0 \ -netdev bridge,id=bridge-iscsi,br=virbr-iscsi \ -device virtio-net-pci,netdev=bridge-world,id=nic-world,mac=52:54:00:00:00:01 \ -device virtio-net-pci,netdev=bridge-iscsi,id=nic-iscsi,mac=52:54:00:00:00:02 \ -kernel ipxe.lkrn Connect to VNC for iPXE shell: $ vncviewer
:1 iPXE <...> Press Ctrl-B for iPXE command line. ^B iPXE> Configure iSCSI NIC: iPXE> ifopen net1 iPXE> set net1/ip 10.0.0.2 iPXE> set net1/netmask 255.255.255.0 Configure iBFT: (iSCSI portal 10.0.0.1, LUN 1 on target iqn.<...>:target1) iPXE> sanhook iscsi:10.0.0.1:::1:iqn.2019-03.com.example:target1 Registered SAN device 0x80 Boot the installer (add option 'disk-detect/ibft/enable=true' for installer to detect iBFT iSCSI disks and option 'partman-iscsi/iscsi_auto=true' to set the system to boot from iBFT): iPXE> ifopen net0 iPXE> kernel http://192.168.122.1:8000/linux initrd=initrd.gz disk-detect/ibft/enable=true partman-iscsi/iscsi_auto=true --- console=ttyS0 iPXE> initrd http://192.168.122.1:8000/initrd.gz iPXE> boot Back to serial console. Proceed with the installer. In 'Users and passwords' dialog, select 'Go back', and 'Execute a shell', and 'Continue'. Check kernel version and iscsi_ibft.ko module is present. ~ # uname -rv 5.0.0-8-generic #9-Ubuntu SMP Tue Mar 12 21:58:11 UTC 2019 ~ # depmod -a ~ # modinfo --filename iscsi_ibft /lib/modules/5.0.0-8-generic/kernel/drivers/firmware/iscsi_ibft.ko ~ # wget http://ppa.launchpad.net/mfo/lp1817321v3/ubuntu/pool/main/h/hw-detect/disk-detect_1.117ubuntu7.$VERSION_amd64.udeb ~ # wget http://ppa.launchpad.net/mfo/lp1817321v3/ubuntu/pool/main/p/partman-iscsi/partman-iscsi_40ubuntu4.$VERSION_all.udeb ~ # udpkg --unpack *.udeb ~ # debconf-get disk-detect/ibft/enable true ~ # debconf-get partman-iscsi/iscsi_auto true (Use this if you need it; e.g., forgot kernel cmdline options) ~ # debconf-set disk-detect/ibft/enable true Start another installer menu with the new debconf templates/question: ~ # debconf -o d-i /usr/bin/main-menu Proceed with the installer. In the 'Partition disks' dialog, the iSCSI LUN should be present: SCSI3 (0,0,1) (sda) - 8.6 GB IET VIRTUAL-DISK Proceed with the installer. In the 'Finish the Installation' dialog, 'Go Back', 'Execute a shell', 'Continue'. Check ISCSI_AUTO is set to true in the installation target's /etc/iscsi/iscsi.initramfs. ~ # cat /target/etc/iscsi/iscsi.initramfs ISCSI_AUTO=true Resume the installation with 'exit' and finish it. System reboots. Back to VNC console, go again to iPXE shell: $ vncviewer
:1 iPXE <...> Press Ctrl-B for iPXE command line. ^B iPXE> Configure iSCSI NIC: iPXE> ifopen net1 iPXE> set net1/ip 10.0.0.2 iPXE> set net1/netmask 255.255.255.0 Boot from iSCSI (iSCSI portal 10.0.0.1, LUN 1 on target iqn.<...>:target1) iPXE> sanboot iscsi:10.0.0.1:::1:iqn.2019-03.com.example:target1 Registered SAN device 0x80 Booting from SAN device 0x80 Back to serial console. Watch the system boot: ... Setting up software interface ens4 iscsistart: Logging into iqn.2019-03.com.example:target1 10.0.0.1:3260,1 iscsistart: version 2.0-874 iscsistart: Connection1:0 to [target: iqn.2019-03.com.example:target1, portal: 10.0.0.1,3260] through [iface: default] is operational now /scripts/local-top/iscsi: line 397: : Permission denied /dev/sda1: recovering journal /dev/sda1: clean, 60800/524288 files, 519693/2096640 blocks ... ubuntu login: ubuntu Password: ubuntu@ubuntu:~$ mount | grep -w / /dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro) # plus 'data=ordered' in bionic ubuntu@ubuntu:~$ ls -ld /sys/block/sda lrwxrwxrwx 1 root root 0 Apr 3 21:38 /sys/block/sda -> ../devices/platform/host2/session1/target2:0:0/2:0:0:1/block/sda ubuntu@ubuntu:~$ cat /sys/devices/platform/host2/session1/iscsi_session/session1/targetname iqn.2019-03.com.example:target1 syslog example ============== $ sed -n '/iBFT found/p ; /iBFT detected/,/iBFT disk detection finished/p' syslog Apr 3 20:03:53 kernel: [ 0.006887] iBFT found at 0x9e520. Apr 3 20:12:59 kernel: [ 546.682048] iBFT detected. Apr 3 20:12:59 disk-detect: insmod /lib/modules/5.0.0-8-generic/kernel/drivers/scsi/iscsi_boot_sysfs.ko Apr 3 20:12:59 disk-detect: insmod /lib/modules/5.0.0-8-generic/kernel/drivers/firmware/iscsi_ibft.ko Apr 3 20:12:59 disk-detect: # BEGIN RECORD 2.0-874 Apr 3 20:12:59 disk-detect: iface.initiatorname = iqn.2010-04.org.ipxe:00000000-0000-0000-0000-000000000000 Apr 3 20:12:59 disk-detect: iface.hwaddress = 52:54:00:00:00:02 Apr 3 20:12:59 disk-detect: iface.bootproto = STATIC Apr 3 20:12:59 disk-detect: iface.ipaddress = 10.0.0.2 Apr 3 20:12:59 disk-detect: iface.subnet_mask = 255.255.255.0 Apr 3 20:12:59 disk-detect: iface.primary_dns = 192.168.122.1 Apr 3 20:12:59 disk-detect: iface.vlan_id = 0 Apr 3 20:12:59 disk-detect: iface.net_ifacename = ens4 Apr 3 20:12:59 disk-detect: node.name = iqn.2019-03.com.example:target1 Apr 3 20:12:59 disk-detect: node.conn[0].address = 10.0.0.1 Apr 3 20:12:59 disk-detect: node.conn[0].port = 3260 Apr 3 20:12:59 disk-detect: node.boot_lun = 01000000 Apr 3 20:12:59 disk-detect: # END RECORD Apr 3 20:12:59 disk-detect: Setting up software interface ens4 Apr 3 20:12:59 disk-detect: iscsistart: Apr 3 20:12:59 disk-detect: can not connect to iSCSI daemon (111)! Apr 3 20:12:59 disk-detect: Apr 3 20:12:59 disk-detect: iscsistart: version 2.0-874 Apr 3 20:12:59 kernel: [ 546.699057] Loading iSCSI transport class v2.0-870. Apr 3 20:13:00 kernel: [ 547.707438] iscsi: registered transport (tcp) Apr 3 20:13:00 kernel: [ 547.709836] scsi host2: iSCSI Initiator over TCP/IP Apr 3 20:13:00 disk-detect: iscsistart: Connection1:0 to [target: iqn.2019-03.com.example:target1, portal: 10.0.0.1,3260] through [iface: default] is operational now Apr 3 20:13:00 kernel: [ 547.716901] scsi 2:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5 Apr 3 20:13:00 kernel: [ 547.719266] scsi 2:0:0:0: Attached scsi generic sg0 type 12 Apr 3 20:13:00 kernel: [ 547.722572] scsi 2:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5 Apr 3 20:13:00 kernel: [ 547.724360] sd 2:0:0:1: Attached scsi generic sg1 type 0 Apr 3 20:13:00 kernel: [ 547.724668] sd 2:0:0:1: Power-on or device reset occurred Apr 3 20:13:00 disk-detect: iscsistart: Logging into iqn.2019-03.com.example:target1 10.0.0.1:3260,1 Apr 3 20:13:00 kernel: [ 547.727114] sd 2:0:0:1: [sda] 16777216 512-byte logical blocks: (8.59 GB/8.00 GiB) Apr 3 20:13:00 kernel: [ 547.727117] sd 2:0:0:1: [sda] 4096-byte physical blocks Apr 3 20:13:00 kernel: [ 547.727463] sd 2:0:0:1: [sda] Write Protect is off Apr 3 20:13:00 kernel: [ 547.727466] sd 2:0:0:1: [sda] Mode Sense: 69 00 10 08 Apr 3 20:13:00 kernel: [ 547.728047] sd 2:0:0:1: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA Apr 3 20:13:00 kernel: [ 547.742488] sd 2:0:0:1: [sda] Attached SCSI disk Apr 3 20:13:00 disk-detect: iBFT disk detection finished.