[libvirt] [PATCH v4 00/14] Introduce vGPU mdev framework to libvirt

[Laine Stump]

On 03/28/2017 10:05 PM, yonglihe wrote:
> On 2017年03月27日 15:42, yonglihe wrote:
>>
>> Verify Summary:
>> * the none rooted mode starting a high-privileges VM actually.
>>
>> The configurations is source generated default value except tls disabled.
>>
>>
>> 1. rooted
>>
>> virsh define ./libvirt/vgpu-win10.xml
>> Domain vgpu-win10 defined from ./libvirt/vgpu-win10.xml
>>
>> ubuntu at z-nuc-11:~/vgpu-meta/libvirt-stage$ virsh start vgpu-win10
>> 2017-03-26 23:28:57.385+0000: 2886: info : libvirt version: 3.2.0
>> 2017-03-26 23:28:57.385+0000: 2886: info : hostname: z-nuc-11.maas
>> 2017-03-26 23:28:57.385+0000: 2886: warning : qemuDomainObjTaint:4155
>> : Domain id=1 name='vgpu-win10'
>> uuid=916c5c36-0437-11e7-a23d-830ed1295d00 is tainted: high-privileges
>> 2017-03-26 23:28:58.010+0000: 2886: warning :
>> virDomainAuditHostdev:456 : Unexpected hostdev type while encoding
>> audit message: 4
>> Domain vgpu-win10 started
>>
>>
>> 2. None rooted
>> virsh -c qemu:///session
>> Welcome to lt-virsh, the virtualization interactive terminal.


The above line makes me think that you're mixing up "unprivileged
libvirtd" with "unprivileged qemu".

When you connect to virsh with "virsh -c qemu:///session" you are using
an unprivileged copy of libvirtd started for your specific uid, and that
libvirtd will:

1) not do any of the uid/permission/selinux/apparmor changes to the
files/devices that will be used by the qemu process. (because it can't)

2) try to modify the locked memory limit for the qemu process, but
likely fail because it needs more than the user's default limit. (I see
below that you ran into this).

3) run qemu as the same unprivileged user.


When you connect to virsh with the default URL (qemu:///system) you will
connect to the system instance of libvirtd, which is running as root. It
will:

1) modify uid/permissions/selinux/apparmor settings of any files/devices
according to the "user" setting in /etc/libvirt/qemu.conf.

and after forking the qemu process:

2) modify the locked memory limit to accommodate the needs of any
assigned devices and

3) change the uid of the qemu process to the "user" setting from
qemu.conf and drop all privileges

(in the case that the "user" in qemu.conf is set to root, then step 3
doesn't happen).

It sounds like you are using an "unprivileged libvirtd" in your tests,
which will create the need to chown the various device files and
manually change the ulimit for the login session that is running "virsh
-c qemu:///session" (and thus starting up the unprivileged libvirtd
which gets started on demand).

The more common scenario is to use virsh -c qemu:///system (or simply
run virsh as root and not add the URL so that the default is used), and
to leave the qemu user set to "qemu" (or in some distros I think it is
set to "kvm" by default).

>>
>> virsh # define ./libvirt/vgpu-win10.xml
>> Domain vgpu-win10 defined from ./libvirt/vgpu-win10.xml
>>
>> virsh # start vgpu-win10
>> 2017-03-26 23:38:11.220+0000: 2882: warning : qemuDomainObjTaint:4155
>> : Domain id=4 name='vgpu-win10'
>> uuid=916c5c36-0437-11e7-a23d-830ed1295d00 is tainted: high-privileges
>> 2017-03-26 23:38:12.356+0000: 2882: warning :
>> virDomainAuditHostdev:456 : Unexpected hostdev type while encoding
>> audit message: 4
>> Domain vgpu-win10 started
> Please ignore above none rooted testing result, my fault. the proper
> test given following result:
> 
> to successfully starting a non rooted vm, the following operation needed:
> 1.change the ownership/access right of the mdev corresponding vfio
>    sudo chown ubuntu:ubuntu /dev/vfio/0
> 
> 2. set a correct ulimit -l  for the vm
> sudo sh -c "ulimit -l 3074424832 && exec su $LOGNAME"
> 
> otherwise, it running into the following error:
> virsh # start vgpu-win10
>  internal error: Process exited prior to exec: libvirt:  error : cannot
> limit locked memory to 3074424832: Operation not permitted

This is to be expected - both of these extra steps are also needed if
you try to assign a standard PCI device using VFIO using unprivileged
libvirtd. This is the best that can be expected without any component
having root privileges.

If you run the same test using qemu:///system, both of these should be
taken care of automatically.

> 
> my testing bed is Ubuntu 14.04, there is a similar bug ever reported:
> https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1276719

The original report (filed 2.5 years ago, and resolved soon after) was
due to apparmor not doing the right stuff to the necessary files in
sysfs. Many later comments and error messages were from people who were
using the tools incorrectly (e.g. using <qemu:commandline> to manually
add "-device vfio-pci" args to the qemu process, making it impossible
for libvirt to recognize that it must perform steps 2 & 3 listed above.

> 
> I could not make sure if there is special requirements  run virsh
> directly from the source tree using the ./run scripts. fix me.

I'm fairly certain the reason you're needing to perform those two extra
steps are because you're using qemu:///session instead of qemu:///system.