[libvirt] ARMv7 guest PCI support broken in 3.0.0 onwards

Thu Feb 16 15:58:52 UTC 2017

On Wed, 2017-02-15 at 09:24 +0000, Daniel P. Berrange wrote:
[...]
> $ virsh start f22-arm32
> error: Failed to start domain f22-arm32
> error: internal error: qemu unexpectedly closed the monitor: 2017-02-15T09:24:03.967648Z qemu-system-arm: -device
ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1: MSI is not supported by interrupt controller
> 2017-02-15T09:24:03.968154Z qemu-system-arm: -device ioh3420,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1: Device initialization failed
> 
> I've not bisected it other than to find it works in 2.5.0 and is
> broken in 3.0.0

I'd like you to share a few additional details:

  * What does the guest XML look like after libvirt has
    had a change to augment it?

  * What QEMU command line does libvirt generate based
    on said guest XML?

  * What QEMU version are you running?

I'm asking because I've just spent some time trying to run
ARM guests[1] on my laptop[2] and I can't reproduce the
failure you're describing, so I need more information to
try and narrow it down.

After the skip, a bunch of unstructured rambling about
stuff I've discovered in the process, interleaved with
more questions about your issue. Have fun! :)

---

I would expect such an error to pop up simply by running

  $ qemu-system-arm -nodefaults -M virt \
    -device ioh3420,port=0x8,...

Does that happen on your system?

---

I didn't realize the mach-virt machine type was not an
aarch64-only thing. Indeed, it's available through
qemu-system-arm too, and the hardware seems to be the
same, eg. running

  $ echo -e 'info qtree\nquit\n' | \
    qemu-system-$arch -M virt -monitor stdio

yields the same output.

The only difference AFAICT is that qemu-system-arm limits
the selection of CPUs to 32-bit models only, eg. cortex-a15
is available on both but only qemu-system-aarch64 lets you
use cortex-a57.

---

I know 32-bit UEFI is a thing, because it's used on a bunch
of budget x86 tablet and causes grief and pain to anyone
trying to run Linux on them. However, Fedora only ships
64-bit binaries (edk2-ovmf and edk2-aarch64 packages) so I
can't really try whether an armv7l guest can boot using UEFI.

---

Speaking of booting the guest, how would that work with the
guest XML you're feeding libvirt, exactly? Since you don't
have any sort of firmware, the only way I can see it working
is to to have <kernel>, <initrd> and <cmdline> elements
inside <domain><os>, and libvirt can't possibly figure out
their values for you...

---

Does it help at all to use

  <address type='virtio-mmio'/>

in order to force the the SCSI controller and network
interface to use virtio-mmio instead of virtio-pci?

---

As a reference, here's the XML for my Fedora 25/armv7l
guest:

<domain type='qemu'>
  <name>f25-armv7l</name>
  <uuid>029956e6-5e98-4ca6-8b2b-72322f73f3ea</uuid>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='armv7l' machine='virt-2.9'>hvm</type>
    <kernel>/home/abologna/.local/share/libvirt/images/f25-armv7l.vmlinuz</kernel>
    <initrd>/home/abologna/.local/share/libvirt/images/f25-armv7l.initramfs</initrd>
    <cmdline>console=ttyAMA0 rw root=LABEL=_/ rootwait</cmdline>
    <boot dev='hd'/>
  </os>
  <features>
    <gic version='2'/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>cortex-a15</model>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/bin/qemu-system-arm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/abologna/.local/share/libvirt/images/f25-armv7l.qcow2'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <interface type='user'>
      <mac address='52:54:00:9a:47:50'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
  </devices>
</domain>

---

And here's that for my Fedora 25/aarch64 guest:

<domain type='qemu'>
  <name>f25-aarch64</name>
  <uuid>e3e954ba-5280-4b6b-9bfc-7f944f2c67b2</uuid>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='aarch64' machine='virt-2.9'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader>
    <nvram>/home/abologna/.config/libvirt/qemu/nvram/f25-aarch64_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <gic version='2'/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>cortex-a57</model>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/bin/qemu-system-aarch64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/abologna/.local/share/libvirt/images/f25-aarch64.qcow2'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <interface type='user'>
      <mac address='52:54:00:c3:33:4a'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
  </devices>
</domain>

[1] Fedora 25/armv7l, plus Fedora 25/aarch64 for comparison
[2] Fedora 24/x86_64 with libvirt 3.0.0; I've tried, in turn,
    QEMU master, QEMU 2.6 (as shipped with Fedora) and
    upstream QEMU 2.5
-- 
Andrea Bolognani / Red Hat / Virtualization