[vfio-users] permission denied on /dev/vfio/1 gpu passthrough kvm
Manuel Ullmann
labre at posteo.de
Thu Jan 4 23:08:47 UTC 2018
Hm,
I happen to use a qemu command line currently. I recall, that guest/host
sound sharing was close to impossible otherwise (using Alsa).
I incorporated your drives and vfio devices, but since I don’t have
discrete graphics, I have no idea how to get console output with
OVMF. Found something here and added the serial ports and stdio monitor
as suggested. https://wiki.ubuntu.com/UEFI/OVMF
You can attach to them with screen.
If you don’t get any OVMF errors anymore, but still no output, you could
try to chmod the vfio device to 0666 temporarily, although this should
not be necessary, since you don’t get a permission denied error anymore.
I use the following to initialize my tap device. You probably don’t use
shorewall, so I commented the restart. Also you should replace the
device placeholders and those from ip and gateway. Use ip link and ip
addr to get yours.
#!/bin/sh
date >> /var/log/virsh.log
sysctl net.ipv4.ip_forward=1 >> /var/log/virsh.log
ip link add name br0 type bridge >>/var/log/virsh.log 2>&1
ip link set dev br0 up >>/var/log/virsh.log 2>&1
ip link set dev <your_dev> up >>/var/log/virsh.log 2>&1
ip link set dev <your_dev> master br0 >>/var/log/virsh.log 2>&1
ip tuntap add dev tap0 mode tap user qemu group qemu >>/var/log/virsh.log 2>&1
ip link set dev tap0 up >>/var/log/virsh.log 2>&1
ip link set dev tap0 master br0 >>/var/log/virsh.log 2>&1
dhclient -4 br0 >>/var/log/virsh.log 2>&1
ip route change <your_ip>/24 via <gateway_ip> >>/var/log/virsh.log 2>&1
#rc-service shorewall restart >>/var/log/virsh.log 2>&1
#rc-service shorewall6 restart >>/var/log/virsh.log 2>&1
iptables --table nat --append POSTROUTING --out-interface br0 -j MASQUERADE >>/var/log/virsh.log 2>&1
iptables --insert FORWARD --in-interface tap0 -j ACCEPT >>/var/log/virsh.log 2>&1
echo "finished net setup" >>/var/log/virsh.log
That VM won’t have input devices momentarily. Once qemu indicates a
successful start and you get output, kill the VM and replace the
commented -device usb-host entries with appropriate ids for your
keyboard and mouse device. You get these with lsusb. Bus from lsusb
output is hostbus and Device is hostaddr. Should the VM get stuck, you
can reattach the keyboard to regain control on the host.
You were correct. The openrc runlevel is optional. If you would have one
and can write openrc-run scripts, you could put the preparation and
cleaning up stuff into one, so that you later just have to do something
like init qemuvm and init default to switch into a VM friendly mode
(less stuff running) and back.
https://wiki.gentoo.org/wiki/Handbook:AMD64/Working/Initscripts#Writing_initscripts
Let me know, what you get.
LC_ALL=C /usr/bin/qemu-system-x86_64 \
-machine pc-i440fx-2.7,accel=kvm,usb=off,vmport=off,dump-guest-core=off,kernel_irqchip=on \
-cpu host,migratable=no,+invtsc,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff \
-m 5000 \
-realtime mlock=off \
-smp 3,sockets=1,cores=3,threads=1 \
-object iothread,id=iothread1 \
-display none \
-no-user-config \
-nodefaults \
-rtc base=localtime,clock=vm \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-reboot \
-serial pty \
-serial pty \
-monitor stdio \
-nographic \
-global PIIX4_PM.disable_s3=1 \
-global PIIX4_PM.disable_s4=1 \
-boot menu=off,strict=on \
-device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 \
-device ahci,id=sata0,bus=pci.0,addr=0x8 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 \
-drive if=none,media=cdrom,id=drive-sata0-0-1,readonly=on \
-device ide-cd,bus=sata0.1,drive=drive-sata0-0-1,id=sata0-0-1 \
-device ide-cd,bus=sata0.4,drive=drive-sata0-0-4,id=sata0-0-4 \
-drive file=/usr/share/edk2-ovmf/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on \
-drive file=/var/lib/libvirt/qemu/nvram/win10_VARS.fd,if=pflash,format=raw,unit=1 \
-device virtio-blk-pci,iothread=iothread1,scsi=off,bus=pci.0,addr=0xa,drive=drive-virtio-disk1,id=virtio-disk1 \
-device virtio-blk-pci,iothread=iothread1,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk4,id=virtio-disk4 \
-device virtio-net-pci,netdev=hostnet0,addr=0xc \
-netdev tap,ifname=tap0,script=no,downscript=no,vhost=on,id=hostnet0 \
-device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x2 \
-device vfio-pci,host=01:00.1,id=hostdev1,bus=pci.0,addr=0x9 \
# -device usb-host,hostbus=3,hostaddr=7,id=hostdev1,bus=usb.0,port=1 \
# -device usb-host,hostbus=3,hostaddr=6,id=hostdev3,bus=usb.0,port=2 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 \
-device usb-audio,id=sound0 \
-msg timestamp=on 2>&1
> Additionally, I experimented removing references to hostnet0, and
> while I get no error messages, I also fail to get any image on the
> display port in the monitor.
>
> /ts
>
> On Thu, Jan 4, 2018 at 7:28 PM, Tiago Seco <tiago.seco at gmail.com> wrote:
>> Hello Manuel
>>
>> With regards to permissions, I will try getting qemu to run with a
>> lower privilege user, but since my setup is not working right now, I
>> want to impose as little restrictions as possible. Thank you for the
>> input, as well as the tip regarding hugepages.
>>
>> As far as your suggestion concerning openrc I must confess I do not
>> fully follow what you're suggesting, but you made it sound like it was
>> an improvement rather than a requirement?
>>
>> Finally, for virsh start win10-clone
>> /var/log/libvirt/qemu/win10-clone.log reads:
>>
>> 2018-01-04 20:10:41.397+0000: starting up libvirt version: 3.8.0, qemu
>> version: 2.10.1, hostname: rathalos
>> LC_ALL=C
>> PATH=/bin:/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/x86_64-pc-linux-gnu/gcc-bin/6.4.0:/usr/lib/llvm/4/bin:/opt/bin
>> HOME=/root USER=root QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64
>> -name guest=win10-clone,debug-threads=on -S -object
>> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-8-win10-clone/master-key.aes
>> -machine pc-i440fx-2.10,accel=kvm,usb=off,vmport=off,dump-guest-core=off
>> -cpu
>> Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,xsaves=on,pdpe1gb=on,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff
>> -drive file=/usr/share/edk2-ovmf/OVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on
>> -drive file=/var/lib/libvirt/qemu/nvram/win10_VARS.fd,if=pflash,format=raw,unit=1
>> -m 4096 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
>> a673c9c8-a1f5-4c3d-97a7-16096bcf1b01 -display none -no-user-config
>> -nodefaults -chardev
>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-8-win10-clone/monitor.sock,server,nowait
>> -mon chardev=charmonitor,id=monitor,mode=control -rtc
>> base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=delay
>> -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
>> PIIX4_PM.disable_s4=1 -boot strict=on -device
>> ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device
>> ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5
>> -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1
>> -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2
>> -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x6 -device
>> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive
>> file=/home/tse/kvm_vm/test1-clone.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0
>> -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1
>> -drive if=none,id=drive-ide0-0-1,readonly=on -device
>> ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -drive
>> if=none,id=drive-ide0-1-0,readonly=on -device
>> ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev
>> tap,fd=21,id=hostnet0 -device
>> rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:45:6f:82,bus=pci.0,addr=0x3
>> -chardev pty,id=charserial0 -device
>> isa-serial,chardev=charserial0,id=serial0 -device
>> AC97,id=sound0,bus=pci.0,addr=0x4 -chardev
>> spicevmc,id=charredir0,name=usbredir -device
>> usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev
>> spicevmc,id=charredir1,name=usbredir -device
>> usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device
>> vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x2 -device
>> vfio-pci,host=01:00.1,id=hostdev1,bus=pci.0,addr=0x9 -device
>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
>> 2018-01-04T20:10:41.406580Z qemu-system-x86_64: -chardev
>> pty,id=charserial0: char device redirected to /dev/pts/3 (label
>> charserial0)
>> 2018-01-04T20:10:41.432099Z qemu-system-x86_64: -device
>> vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x2: vfio error:
>> 0000:01:00.0: failed to open /dev/vfio/1: Permission denied
>> 2018-01-04 20:10:41.442+0000: shutting down, reason=failed
>>
>> Interestingly enough, if I try to run that same command (as root) I get:
>>
>> 2018-01-04T20:14:51.989313Z qemu-system-x86_64: -object
>> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-8-win10-clone/master-key.aes:
>> Unable to read /var/lib/libvirt/qemu/domain-8-win10-clone/master-key.aes:
>> Failed to open file
>> '/var/lib/libvirt/qemu/domain-8-win10-clone/master-key.aes': No such
>> file or directory
>>
>> Removing '-object
>> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-8-win10-clone/master-key.aes'
>> I get:
>>
>> 2018-01-04T20:15:57.204616Z qemu-system-x86_64: -chardev
>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-8-win10-clone/monitor.sock,server,nowait:
>> Failed to bind socket to
>> /var/lib/libvirt/qemu/domain-8-win10-clone/monitor.sock: No such file
>> or directory
>>
>> I'm unsure wether or not this file should exist, if I create it, and
>> run the same command I get:
>>
>> 2018-01-04T20:17:07.490052Z qemu-system-x86_64: -chardev
>> pty,id=charserial0: char device redirected to /dev/pts/3 (label
>> charserial0)
>> 2018-01-04T20:17:07.490500Z qemu-system-x86_64: -netdev
>> tap,fd=21,id=hostnet0: TUNGETIFF ioctl() failed: Bad file descriptor
>> TUNSETOFFLOAD ioctl() failed: Bad file descriptor
>>
>> (the logs show nothing)
>>
>> If I do not create the file, and run the command without any
>> references to the monitor (so without '-chardev
>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-8-win10-clone/monitor.sock,server,nowait
>> -mon chardev=charmonitor,id=monitor,mode=control'), I get:
>>
>> 2018-01-04T20:22:04.201180Z qemu-system-x86_64: -chardev
>> pty,id=charserial0: char device redirected to /dev/pts/3 (label
>> charserial0)
>> 2018-01-04T20:22:04.201597Z qemu-system-x86_64: -netdev
>> tap,fd=21,id=hostnet0: TUNGETIFF ioctl() failed: Bad file descriptor
>> TUNSETOFFLOAD ioctl() failed: Bad file descriptor
>>
>> (Which leaves me wondering wether or not the -chardev and -mon
>> switches are really necessary).
>>
>> I have very little experience with qemu, and could not spot anything
>> blatantly wrong in that command line.
>>
>> Once again, thank you very much for your help.
>>
>> /ts
>>
>> On Thu, Jan 4, 2018 at 5:32 PM, Manuel Ullmann <labre at posteo.de> wrote:
>>>> Hello,
>>>>
>>>> I had initially posted this on the gentoo forums, but since the thread
>>>> is getting little traction, perhaps here is a better place to ask the
>>>> question:
>>>>
>>>> I'm trying to achieve PCI-express GPU passthrough to a Window 10 VM,
>>>> however, when trying to start the virtual machine (as root) with the
>>>> hardware attached, this is what I get:
>>>>
>>>> virsh # start win10-clone
>>>> error: Failed to start domain win10-clone
>>>> error: internal error: process exited while connecting to monitor:
>>>> 2018-01-03T16:58:47.761436Z qemu-system-x86_64: -chardev
>>>> pty,id=charserial0: char device redirected to /dev/pts/4 (label
>>>> charserial0)
>>>> 2018-01-03T16:58:47.786124Z qemu-system-x86_64: -device
>>>> vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x2: vfio error:
>>>> 0000:01:00.0: failed to open /dev/vfio/1: Permission denied
>>>>
>>>> The file however, exists:
>>>>
>>>> # file /dev/vfio/1
>>>> /dev/vfio/1: character special (250/0)
>>>> # ls -l /dev/vfio/1
>>>> crw------- 1 root root 250, 0 Jan 3 16:51 /dev/vfio/1
>>>>
>>>>
>>>> I followed the steps detailed in
>>>> https://wiki.installgentoo.com/index.php/PCI_passthrough for host
>>>> preparation, and in
>>>> https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Setting_up_the_guest_OS
>>>> for setting up the guest.
>>>>
>>>> Before running virsh, I:
>>>>
>>>> /etc/init.d/libvirtd stop
>>>> /etc/init.d/libvirt-guests stop
>>>> ./vfio-bind 0000:01:00.0 0000:01:00.1
>>>> /etc/init.d/libvirtd start
>>>> /etc/init.d/libvirt-guests start
>>>>
>>>>
>>>> I've tried playing around with /etc/libvirt/qemu.conf, namely with the
>>>> "group", "user", and "dynamic_ownership" settings - even though I'm
>>>> running everything as root - and with "cgroup_device_acl" (by adding
>>>> "/dev/vfio/1" to the list).
>>>> I cannot figure out what the problem in my setup is or what to try
>>>> next. Is there anyone that can shine some light on this?
>>>>
>>>>
>>>> System information, scripts used, and qemu USE flags can be found
>>>> here: https://pastebin.com/XuGtDmSm
>>>>
>>>> Thank you!
>>> Hello Tiago,
>>>
>>> I can vaguely recall having the same problem initially. Can you verify,
>>> that qemu really runs as root? If you strip the libvirt stuff from the
>>> qemu command line indicated in libvirtd logs and run it as root, does
>>> that work?
>>>
>>> Although I never would question the freedom, that Gentoo gives to its
>>> users, I still would suggest setting user and group to qemu though. root
>>> is quite insecure and we have a longstanding Intel CPU Bug disclosed in
>>> a few weeks tops, leaking page table content to userspace, which affects
>>> VM sandboxing gravely. Add kvm and audio groups to qemu user. If you use
>>> hugepages, those also have to belong to qemu.
>>>
>>> vfio nodes (or at least /dev/vfio/vfio) have not much for udev to match
>>> against, so it could be easier to modify permission in your startup
>>> scripts. I have this conveniently tackled by using a second bootentry
>>> with another default boot runlevel, which my /etc/local.d/*.start
>>> scripts will recognize and prepare everything for the VM (with openrc of
>>> course). This makes less sense with discrete graphics passthrough
>>> though, so I still would create a dedicated runlevel, but add a
>>> openrc-run script, which prepares it, adding a rc_need entry in
>>> conf.d/libvirtd.
>>>
>>> Best regards,
>>> Manuel
More information about the vfio-users
mailing list