Bug #1865120 “arm64 vm boot failed when set num_pcie_ports to 28...” : Bugs : OpenStack Compute (nova)

Revision history for this message

norman shen (jshen28) wrote on 2020-02-28:

#1

Download full text (12.5 KiB)

xml for the problematic vm is shown below

virsh # dumpxml 52
<domain type='kvm' id='52'>
  <name>instance-000002f6</name>
  <uuid>809a8d51-8ad0-4472-9479-8d0554a66265</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="18.2.3"/>
      <nova:name>sw-test2</nova:name>
      <nova:creationTime>2020-02-17 05:56:08</nova:creationTime>
      <nova:flavor name="ecs_1C2G40G">
        <nova:memory>2048</nova:memory>
        <nova:disk>40</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>1</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="b4653e7431264ebf99809dc8efe9e49b">admin</nova:user>
        <nova:project uuid="5d51ef16f6644af2971560618c93f071">admin</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="d3fe4150-af9f-47af-b3a8-93a31bfc2ebe"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <shares>1024</shares>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='aarch64' machine='virt-2.11'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/AAVMF/AAVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/instance-000002f6_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <gic version='2'/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-aarch64</emulator>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
      <auth username='cinder'>
        <secret type='ceph' uuid='457eb676-33da-42ec-9a8c-9293d545c337'/>
      </auth>
      <source protocol='rbd' name='nova.vms/809a8d51-8ad0-4472-9479-8d0554a66265_disk'>
        <host name='172.16.22.67' port='6789'/>
        <host name='172.16.22.68' port='6789'/>
        <host name='172.16.22.69' port='6789'/>
      </source>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <...

xml for the problematic vm is shown below

virsh # dumpxml 52
<domain type='kvm' id='52'>
  <name>instance-000002f6</name>
  <uuid>809a8d51-8ad0-4472-9479-8d0554a66265</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="18.2.3"/>
      <nova:name>sw-test2</nova:name>
      <nova:creationTime>2020-02-17 05:56:08</nova:creationTime>
      <nova:flavor name="ecs_1C2G40G">
        <nova:memory>2048</nova:memory>
        <nova:disk>40</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>1</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="b4653e7431264ebf99809dc8efe9e49b">admin</nova:user>
        <nova:project uuid="5d51ef16f6644af2971560618c93f071">admin</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="d3fe4150-af9f-47af-b3a8-93a31bfc2ebe"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <shares>1024</shares>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='aarch64' machine='virt-2.11'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/AAVMF/AAVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/instance-000002f6_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <gic version='2'/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-aarch64</emulator>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
      <auth username='cinder'>
        <secret type='ceph' uuid='457eb676-33da-42ec-9a8c-9293d545c337'/>
      </auth>
      <source protocol='rbd' name='nova.vms/809a8d51-8ad0-4472-9479-8d0554a66265_disk'>
        <host name='172.16.22.67' port='6789'/>
        <host name='172.16.22.68' port='6789'/>
        <host name='172.16.22.69' port='6789'/>
      </source>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xe'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0xf'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x10'/>
      <alias name='pci.9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x11'/>
      <alias name='pci.10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='11' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='11' port='0x12'/>
      <alias name='pci.11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='12' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='12' port='0x13'/>
      <alias name='pci.12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='13' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='13' port='0x14'/>
      <alias name='pci.13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='14' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='14' port='0x15'/>
      <alias name='pci.14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='15' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='15' port='0x16'/>
      <alias name='pci.15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='16' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='16' port='0x17'/>
      <alias name='pci.16'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
    </controller>
    <controller type='pci' index='17' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='17' port='0x18'/>
      <alias name='pci.17'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='18' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='18' port='0x19'/>
      <alias name='pci.18'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    </controller>
    <controller type='pci' index='19' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='19' port='0x1a'/>
      <alias name='pci.19'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
    </controller>
    <controller type='pci' index='20' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='20' port='0x1b'/>
      <alias name='pci.20'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
    </controller>
    <controller type='pci' index='21' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='21' port='0x1c'/>
      <alias name='pci.21'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/>
    </controller>
    <controller type='pci' index='22' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='22' port='0x1d'/>
      <alias name='pci.22'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/>
    </controller>
    <controller type='pci' index='23' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='23' port='0x1e'/>
      <alias name='pci.23'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x6'/>
    </controller>
    <controller type='pci' index='24' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='24' port='0x1f'/>
      <alias name='pci.24'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x7'/>
    </controller>
    <controller type='pci' index='25' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='25' port='0x20'/>
      <alias name='pci.25'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='26' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='26' port='0x21'/>
      <alias name='pci.26'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/>
    </controller>
    <controller type='pci' index='27' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='27' port='0x22'/>
      <alias name='pci.27'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/>
    </controller>
    <controller type='pci' index='28' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='28' port='0x23'/>
      <alias name='pci.28'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x3'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='fa:16:3e:df:97:1a'/>
      <source bridge='qbr08ae6926-ed'/>
      <target dev='tap08ae6926-ed'/>
      <model type='virtio'/>
      <mtu size='1550'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/19'/>
      <log file='/var/lib/nova/instances/809a8d51-8ad0-4472-9479-8d0554a66265/console.log' append='off'/>
      <target type='system-serial' port='0'>
        <model name='pl011'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/19'>
      <source path='/dev/pts/19'/>
      <log file='/var/lib/nova/instances/809a8d51-8ad0-4472-9479-8d0554a66265/console.log' append='off'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/org.qemu.guest_agent.0.instance-000002f6.sock'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='keyboard' bus='usb'>
      <alias name='input1'/>
      <address type='usb' bus='0' port='2'/>
    </input>
    <graphics type='vnc' port='5909' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='virtio' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <stats period='10'/>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+42424:+107</label>
    <imagelabel>+42424:+107</imagelabel>
  </seclabel>
</domain>

Kevin Zhao (kevin-zhao) on 2020-03-11

Changed in nova:
assignee:	nobody → Kevin Zhao (kevin-zhao)
status:	New → Confirmed

Revision history for this message

Kevin Zhao (kevin-zhao) wrote on 2020-03-11:

#2

Hi, pls check here as a reference:
https://opendev.org/openstack/nova/src/branch/master/nova/conf/libvirt.py#L815

Revision history for this message

norman shen (jshen28) wrote on 2020-03-11:

#3

Hi thank you for reply. I am definitely missing something here, but I am wondering why guest failed to boot if I change this value to something other than 0? thanks.

Revision history for this message

Jeffrey Zhang (jeffrey4l) wrote on 2021-01-05:

#4

aarch-15.xml Edit (9.3 KiB, application/xml)

I have a similar issue. seems aarch64 qemu only support when num_pcie_ports=15. guest failed to boot when use num_pcie_ports=16. Following is what i get

when using num_pcie_ports=16, guest OS is stuck at UEFI page with following message

```
UEFI Interactive Shell v2.2
EDK II
UEFI v2.70 (EDK II, 0x00010000)
Mapping table
     BLK1: Alias(s):
          VenHw(F9B94AE2-8BA6-409B-9D56-B9B417F53CB3)
     BLK0: Alias(s):
          VenHw(8047DB4B-7E9C-4C0C-8EBC-DFBBAACACE8F)
Press ESC in 5 seconds to skip startup.nsh or any other key to continue.
```

when using num_pcie_ports=15, the guest OS can be boot successfully, i check the uefi manage, found following

```
UEFI Interactive Shell v2.2
EDK II
UEFI v2.70 (EDK II, 0x00010000)
Mapping table
      FS0: Alias(s):HD1b:;BLK2:
          PciRoot(0x0)/Pci(0x1,0x3)/Pci(0x0,0x0)/HD(1,GPT,8AD1EB5A-1F33-44F8-8A74-EEF37D4802F5,0x800,0x113000)
     BLK6: Alias(s):
          VenHw(F9B94AE2-8BA6-409B-9D56-B9B417F53CB3)
     BLK5: Alias(s):
          VenHw(8047DB4B-7E9C-4C0C-8EBC-DFBBAACACE8F)
     BLK1: Alias(s):
          PciRoot(0x0)/Pci(0x1,0x3)/Pci(0x0,0x0)
     BLK3: Alias(s):
          PciRoot(0x0)/Pci(0x1,0x3)/Pci(0x0,0x0)/HD(2,GPT,1F802CD3-DD0B-4DA7-BF90-B928E443E0F0,0x113800,0x4000)
     BLK4: Alias(s):
          PciRoot(0x0)/Pci(0x1,0x3)/Pci(0x0,0x0)/HD(3,GPT,18DB5C2F-CE88-43D6-8A0F-011E4A1FEE29,0x117800,0x3EE800)
     BLK0: Alias(s):
          PciRoot(0x0)/Pci(0x1,0x1)/Pci(0x0,0x0)/Scsi(0x0,0x0)

Press ESC in 1 seconds to skip startup.nsh or any other key to continue.
Shell>
```

seems when using num_pcie_ports=16, the kvm can not find disks.

the libvirt xml file is attached

Revision history for this message

Jeffrey Zhang (jeffrey4l) wrote on 2021-01-05:

#5

aarch-16.xml Edit (9.5 KiB, application/xml)

aarch-16.xml

Revision history for this message

Jeffrey Zhang (jeffrey4l) wrote on 2021-01-05:

#6

btw, i environment is

* centos 7 4.18.0-193.28.1.el7.aarch64
* libvirt-daemon-4.5.0-36.el7_9.3.aarch64
* qemu-kvm-ev-2.12.0-44.1.el7_8.1.aarch64
* CPU: FT/2000

OpenStack Compute (nova)

arm64 vm boot failed when set num_pcie_ports to 28

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches