[libvirt] Host device assignment driver name vfio/ kvm

Sun Apr 3 16:14:14 UTC 2016

(please configure your email client to properly quote earlier messages, 
and reply inline rather as a top-post; it makes it easier for others to 
determine the context of replies.)

On 04/03/2016 05:00 AM, Moshe Levi wrote:
>
> Hi,
>
> Ok, so here are the errors I get when using vfio/( kvm is working fine)
>
> This is the Error [1] I am getting on resume when using vfio. And [2] 
> is the ls output on the /dev/vfio/
>
> Please note that I am trying to attach pci which is VF
>
> My setup is:
>
> Fedora 21
>
> Libvirt 1.3.0
>
> Qemu 2.5
>
> Kernel  3.17.4-301.fc21.x86_64
>
> OpenStack – master.
>
> Let me know if you need more information.
>
> [1] –
>
> root at r-dcs78:/var/log/libvirt/qemu# cat ./instance-0000000e.log
>
> 2016-03-31 15:31:09.895+0000: starting up libvirt version: 1.3.0, 
> package: 1.fc21 (Unknown, 2016-02-25-10:56:09, 
> r-ufm231.mtr.labs.mlnx), qemu version: 2.5.0 (qemu-2.5.0-11.fc21)
>
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin 
> QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name instance-0000000e -S 
> -machine pc-i440fx-2.5,accel=kvm,usb=off -m 2048 -realtime mlock=off 
> -smp 1,sockets=1,cores=1,threads=1 -uuid 
> a809a3b5-ba75-41e6-889f-ffeebecfe54e -smbios 
> 'type=1,manufacturer=OpenStack Foundation,product=OpenStack 
> Nova,version=13.0.0,serial=533a28c3-7519-4e6d-8cdb-4b1f72649e71,uuid=a809a3b5-ba75-41e6-889f-ffeebecfe54e,family=Virtual 
> Machine' -no-user-config -nodefaults -chardev 
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-instance-0000000e/monitor.sock,server,nowait 
> -mon chardev=charmonitor,id=monitor,mode=control -rtc 
> base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard 
> -no-hpet -no-shutdown -boot strict=on -device 
> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 
> file=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none 
> -device 
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 
> -drive 
> file=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/disk.config,if=none,id=drive-ide0-1-1,readonly=on,format=raw,cache=none 
> -device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 
> -chardev 
> file,id=charserial0,path=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/console.log 
> -device isa-serial,chardev=charserial0,id=serial0 -chardev 
> pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 
> -vnc 127.0.0.1:0 -k en-us -device 
> cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device 
> vfio-pci,host=04:00.7,id=hostdev0,bus=pci.0,addr=0x4 -device 
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
>
> char device redirected to /dev/pts/29 (label charserial1)
>
> 2016-03-31T15:31:46.775166Z qemu-system-x86_64: terminating on signal 
> 15 from pid 6866
>
> 2016-03-31 15:32:07.547+0000: starting up libvirt version: 1.3.0, 
> package: 1.fc21 (Unknown, 2016-02-25-10:56:09, 
> r-ufm231.mtr.labs.mlnx), qemu version: 2.5.0 (qemu-2.5.0-11.fc21)
>
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin 
> QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name instance-0000000e -S 
> -machine pc-i440fx-2.5,accel=kvm,usb=off -m 2048 -realtime mlock=off 
> -smp 1,sockets=1,cores=1,threads=1 -uuid 
> a809a3b5-ba75-41e6-889f-ffeebecfe54e -smbios 
> 'type=1,manufacturer=OpenStack Foundation,product=OpenStack 
> Nova,version=13.0.0,serial=533a28c3-7519-4e6d-8cdb-4b1f72649e71,uuid=a809a3b5-ba75-41e6-889f-ffeebecfe54e,family=Virtual 
> Machine' -no-user-config -nodefaults -chardev 
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-instance-0000000e/monitor.sock,server,nowait 
> -mon chardev=charmonitor,id=monitor,mode=control -rtc 
> base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard 
> -no-hpet -no-shutdown -boot strict=on -device 
> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 
> file=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none 
> -device 
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 
> -drive 
> file=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/disk.config,if=none,id=drive-ide0-1-1,readonly=on,format=raw,cache=none 
> -device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 
> -chardev 
> file,id=charserial0,path=/opt/stack/data/nova/instances/a809a3b5-ba75-41e6-889f-ffeebecfe54e/console.log 
> -device isa-serial,chardev=charserial0,id=serial0 -chardev 
> pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 
> -vnc 127.0.0.1:0 -k en-us -device 
> cirrus-vga,id=video0,bus=pci.0,addr=0x2 -incoming defer -device 
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
>
> char device redirected to /dev/pts/29 (label charserial1)
>
> vfio: error opening /dev/vfio/57: Permission denied
>
> vfio: failed to get group 57
>

It's very strange that this error follows a commandline that doesn't 
have any "-device vfio-pci" in it, and the previous commandline (which 
*did* have a "-device vfio-pci") had no error message. Is the device 
being hotplugged? If not, why isn't it in the commandline? Can you 
include the XML for the device in your next reply? (Since you're 
assigning a network device, I'm hoping that you're using <interface 
type='hostdev'> rather than <hostdev> - the former allows you to set the 
VF's MAC address / vlan tag after detaching it from the host driver and 
before assigning it to the guest)

Alex: what could be the reasons for permission-denied when opening the 
node for the iommu group? Could this be an issue with other devices 
being in the same iommu group? (I would hope this isn't the case, since 
the device in question is apparently an SRIOV VF, so it should have its 
own iommu group).

I'll be offline for about 6 days starting Monday AM, so may not be able 
to participate in this discussion much more. Good luck!

> 2016-03-31T16:05:54.975312Z qemu-system-x86_64: terminating on signal 
> 15 from pid 6866
>
> [2] root at r-dcs78:/var/log/libvirt/qemu# ls -l /dev/vfio/
>
> total 0
>
> crw-rw-rw- 1 root root 10, 196 Mar 29 11:11 vfio
>
> *From:*sendmail [mailto:justsendmailnothingelse at gmail.com] *On Behalf 
> Of *Laine Stump
> *Sent:* Thursday, March 31, 2016 7:07 PM
> *To:* Libvirt <libvir-list at redhat.com>
> *Cc:* Moshe Levi <moshele at mellanox.com>
> *Subject:* Re: [libvirt] Host device assignment driver name vfio/ kvm
>
> On 03/31/2016 11:58 AM, Moshe Levi wrote:
>
>     Thanks Laine,
>
>     Adding  the driver_name o the config did the trick,  thanks J
>
>     Regarding the vifo error it seem that openstack does roleback if
>     operation failed so that why you see the virDomainAttachDeviceFlags
>
>     Anyhow I found that the in qemu 2.1 suspend is not working (I got
>     error [1] )  so I upgrade to qemu 2.5 and then suspend work but it
>     failed on resume.
>
>     So Just to clarify vfio is not supporting “attach device” right?
>      is qemu going to support it?
>
>
> Not at all. attach device works just fine with vfio (that's how 
> hotplug is done). I've just been using it myself in some testing. 
> (note that vfio has been supported since kernel 3.6 and libvirt 1.0.5, 
> i.e. a "very long time")
>
>
>
>     Now I wonder if I need to hardcode in the hostdev config to be
>     with driver_name=kvm…
>
>
>
> I certainly hope not. You would be viciously attacked by the keeper of 
> vfio for your transgression! :-)
>
> Seriously, there should be no reason whatsoever to force driver 
> name='kvm'. This was made configurable *only* as a fallback in case 
> errors were encountered with vfio during its early days. As a matter 
> of fact, I'm pretty sure that RHEL7 has legacy kvm device assignment 
> completely disabled. If you are needing to force legacy kvm device 
> assignment to make your setup work, then there is a bug somewhere that 
> needs to be investigated and squashed.
>
>
>
>
>     [1] -ERROR:qom/object.c:725:object_unref: assertion failed:
>     (obj->ref > 0)
>
>     *From:*sendmail [mailto:justsendmailnothingelse at gmail.com] *On
>     Behalf Of *Laine Stump
>     *Sent:* Wednesday, March 30, 2016 9:25 PM
>     *To:* Libvirt <libvir-list at redhat.com> <mailto:libvir-list at redhat.com>
>     *Cc:* Moshe Levi <moshele at mellanox.com> <mailto:moshele at mellanox.com>
>     *Subject:* Re: [libvirt] Host device assignment driver name vfio/ kvm
>
>     On 03/29/2016 07:45 AM, Moshe Levi wrote:
>
>         Hi,
>
>         I was  testing Host device assignment  in OpenStack
>         environment where the driver name is vfio or kvm.
>
>         My setup is as follow:
>
>         1.Fedora 21
>
>         2.Libvirt 1.3.0 which I compiled
>
>         3.OpenStack master
>
>         I have also other setups with older Libvirt version and the
>         same OpenStack environment.
>
>         I notice that on my fedora environment the driver name is vfio
>         were in my old environment the driver name is kvm.
>
>         According to Libvirt documentation default is "vfio" on
>         systems where the VFIO driver is available and loaded, see [1]
>
>         I remove the vfio modules by removing vfio, vfio_iommu_type1,
>         vfio_pci but when I boot a vm the drive name is vfio
>
>         How can change the driver name to be kvm?
>
>
>     libvirt tries very hard to use vfio rather than legacy kvm,
>     because legacy kvm is old, deprecated, and "declared bad" :-). But
>     it won't changed it to vfio if you've explicitly said that you
>     want to use kvm. If you really want to use legacy kvm device
>     assignment, manually set that in the config. When you do that, if
>     the system you're running on doesn't support it, it will error out
>     rather than switching.
>
>         Another thing that I encounter is an error when suspending VM
>         (in OpenStack environment)  when the driver name is  vfio.
>
>         In such case I am getting  the following error from Libvirt:
>
>         2016-03-28 11:42:59.527 1966 ERROR
>         oslo_messaging.rpc.dispatcher   File
>         "/usr/lib64/python2.7/site-packages/libvirt.py", line 560, in
>         attachDeviceFlags
>
>         2016-03-28 11:42:59.527 1966 ERROR
>         oslo_messaging.rpc.dispatcher     if ret == -1: raise
>         libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
>
>         2016-03-28 11:42:59.527 1966 ERROR
>         oslo_messaging.rpc.dispatcher libvirtError: internal error:
>         unable to execute QEMU command 'device_add': Device
>         initialization failed.
>
>         I would appreciate for some pointers on what can cause this issue.
>
>
>     Assuming that openstack uses libvirt's virDomainSave API I would
>     expect suspending a guest to fail if it had an assigned device
>     (since libvirt implements this by "migrating to disk", and qemu
>     doesn't allow migration of a guest with an assigned device. But
>     your problem is that it's trying to *attach* a device, which I
>     wouldn't consider to be a part of a save or suspend or whatever
>     operation. Is it possible to get more information about what leads
>     up to this?
>
>
>
>
>         [1] https://libvirt.org/formatdomain.html#elementsHostDev
>
>
>
>
>
>         --
>
>         libvir-list mailing list
>
>         libvir-list at redhat.com <mailto:libvir-list at redhat.com>
>
>         https://www.redhat.com/mailman/listinfo/libvir-list
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20160403/d4334e20/attachment-0001.htm>