[libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev framework to libvirt

Mon Mar 20 07:27:19 UTC 2017

tested v3, on the mdev-next branch:

none-root:
======
1. hacker the privilege operations
sudo sh -c "ulimit -l 3074424832 && exec su $LOGNAME"
sudo chown ubuntu:ubuntu /dev/vfio/0

  RFC: i don't know the correct way to do such thing while build it from 
sources. updated me, thanks.

2. myvirsh -c qemu:///session
#define ./libvirt/vgpu-win10.xml
Domain vgpu-win10 defined from ./libvirt/vgpu-win10.xml

  #start vgpu-win10
Domain vgpu-win10 started

3.ps aux | grep qemu

ubuntu    3262  141 12.3 2929432 2017172 ?     SLl  23:58   0:51 
/usr/bin/qemu-system-x86_64 -name guest=vgpu-win10,debug-threads=on -S 
-object 
secret,id=masterKey0,format=raw,file=/home/ubuntu/.config/libvirt/qemu/lib/domain-1-vgpu-win10/master-key.aes 
-machine pc-i440fx-2.3,accel=kvm,usb=off,dump-guest-core=off -m 1908 
-realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 
916c5c36-0437-11e7-a23d-830ed1295d00 -no-user-config -nodefaults 
-chardev 
socket,id=charmonitor,path=/home/ubuntu/.config/libvirt/qemu/lib/domain-1-vgpu-win10/monitor.sock,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc 
-no-shutdown -boot strict=on -device 
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 
file=/home/ubuntu/vgpu-meta/libvirt-stage/win10-64.qcow2,format=qcow2,if=none,id=drive-ide0-0-0,cache=none,aio=native 
-device 
ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 
-chardev pty,id=charserial0 -device 
isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device 
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device 
vfio-pci,sysfsdev=/sys/bus/mdev/devices/894f3983-1a36-42b3-b52c-1024aca216be,bus=pci.0,addr=0x4 
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -msg timestamp=on
ubuntu    3276  0.0  0.0  10468  2216 pts/0    S+   23:59   0:00 grep 
--color=auto qemu

rooted mode
========

start libvirt-d trace:
--------------------------

this trace shows occasionally while starting the libvirt-d, not every time.

2017-03-19 19:22:45.559+0000: 13104: info : libvirt version: 3.2.0
2017-03-19 19:22:45.559+0000: 13104: info : hostname: z-nuc-11.maas
2017-03-19 19:22:45.559+0000: 13104: error : qemuMonitorOpenUnix:367 : 
failed to connect to monitor socket: No such process
2017-03-19 19:22:45.562+0000: 13000: info : libvirt version: 3.2.0
2017-03-19 19:22:45.562+0000: 13000: info : hostname: z-nuc-11.maas
2017-03-19 19:22:45.562+0000: 13000: error : virNetSocketReadWire:1800 : 
End of file while reading data: Input/output error

start domian trace:
--------------------------
some time there is trace while starting the domain.

2017-03-19 19:22:50.912+0000: 13034: warning : qemuDomainObjTaint:4113 : 
Domain id=3 name='vgpu-win10' uuid=916c5c36-0437-11e7-a23d-830ed1295d00 
is tainted: high-privileges
2017-03-19 19:22:51.859+0000: 13000: error : virNetSocketReadWire:1800 : 
End of file while reading data: Input/output error
2017-03-19 19:22:51.859+0000: 13034: warning : virDomainAuditHostdev:456 
: Unexpected hostdev type while encoding audit message: 4
Domain vgpu-win10 started

NOTES:
    there is no traces under none root mode, though i don't think the 
trace is related to user privilege. fix me.

Regards
Yongli He

On 2017年03月16日 22:41, Erik Skultety wrote:
>> [2] 2005
>> ubuntu at z-nuc-11:~/vgpu-meta/libvirt-stage$
>> ***************************************************
>> start libvirt-d
>> 2017-03-09 19:04:57.211+0000: 2059: info : libvirt version: 3.1.0
>> 2017-03-09 19:04:57.211+0000: 2059: info : hostname: z-nuc-11.maas
>> 2017-03-09 19:04:57.211+0000: 2059: error : qemuMonitorOpenUnix:367 :
>> failed to connect to monitor socket: No such process
>> 2017-03-09 19:04:57.213+0000: 2059: error :
>> virMediatedDeviceGetIOMMUGroupDev:153 : internal error: IOMMU group file /sys/bus/mdev/devices/894f3983-1a36-42b3-b52c-1024aca216be/iommu_group
>> is not a symlink
> When I saw this error message for the first time in the original thread, I got
> confused, since this just checks whether the symlink exists, if it
> doesn't, the vfio device probably also doesn't exist (but take this with a
> grain of salt, I haven't investigated that deep) and libvirt needs it to pass
> it onto qemu command line. I hit this issue once by accident in the past and at
> that time I didn't understand what caused it, but after a reboot it was gone.
> So seeing it here it caught my eye and I investigated it last week. What I
> found out was that it's caused by the vfio-mdev module not being loaded
> automatically as a dependency. I solved it by autoloading the module on system
> boot. So this is not a libvirt issue, but just for a reference, there is a BZ
> on this [1].
>
>> 2017-03-09 19:04:57.213+0000: 2003: info : libvirt version: 3.1.0
>> 2017-03-09 19:04:57.213+0000: 2003: info : hostname: z-nuc-11.maas
>> 2017-03-09 19:04:57.213+0000: 2003: error : virNetSocketReadWire:1800 :
>> End of file while reading data: Input/output error
> I suppose this corresponds to the problem above, do you hit this error if you
> work around the vfio-mdev module problem described above?
>
>> the screen call trace while start the VM (same for Ubuntu, Win10 etc) ======================================================
>>
>> ubuntu at z-nuc-11:~/vgpu-meta/libvirt-stage$ myvirsh start vgpu-ubuntu
>> 2017-03-09 19:06:50.483+0000: 2232: info : libvirt version: 3.1.0
>> 2017-03-09 19:06:50.483+0000: 2232: info : hostname: z-nuc-11.maas
>> 2017-03-09 19:06:50.483+0000: 2232: warning : qemuDomainObjTaint:4056 :
>> Domain id=1 name='vgpu-ubuntu' uuid=972b5e38-0437-11e7-8f97-d36dba74552d
>> is tainted: high-privileges
>> 2017-03-09 19:06:50.819+0000: 2204: info : libvirt version: 3.1.0
>> 2017-03-09 19:06:50.819+0000: 2232: warning : virDomainAuditHostdev:456
>> : Unexpected hostdev type while encoding audit message: 4
> This one's interesting, again, are you able to hit the error when you work
> around the missing vfio-mdev module? I'll have a look at this if you actually
> can hit the error, even if the XML is correct.
>
> I posted v3 of the series and also created a new branch 'mdev-next' on my
> github [2]. I dropped the attribute 'type' from the source address element, so
> follow the example in the updated docs.
>
> Thanks for giving it a try.
> Erik
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1420572
> [2] https://github.com/eskultety/libvirt/commits/mdev-next