libvirt-shim: libvirt to run qemu in a different container

[Itamar Holder]

Hey all,

I'm Itamar Holder, a Kubevirt developer.
Lately we came across a problem w.r.t. properly supporting VMs with
dedicated CPUs on Kubernetes. The full details can be seen in this PR
<https://github.com/kubevirt/kubevirt/pull/8869> [1], but to make a very
long story short, we would like to use two different containers in the
virt-launcher pod that is responsible to run a VM:

   - "Managerial container": would be allocated with a shared cpuset. Would
   run all of the virtualization infrastructure, such as libvirtd and its
   dependencies.
   - "Emulator container": would be allocated with a dedicated cpuset.
   Would run the qemu process.

There are many reasons for choosing this design, but in short, the main
reasons are that it's impossible to allocate both shared and dedicated cpus
to a single container, and that it would allow finer-grained control and
isolation for the different containers.

Since there is no way to start the qemu process in a different container, I
tried to start qemu in the "managerial" container, then move it into the
"emulator" container. This fails however, since libvirt uses
sched_setaffinity to pin the vcpus into the dedicated cpuset, which is not
allocated to the managerial container, resulting in an EINVAL error.

Therefore, I thought about discussing a new approach - introducing a small
shim that could communicate with libvirtd in order to start and control the
qemu process that would run on a different container.

As I see it, the main workflow could be described as follows:

   - The emulator container would start with the shim.
   - libvirtd, running in the managerial container, would ask for some
   information from the target, e.g. cpuset.
   - libvirtd would create the domain xml and would transfer to the shim
   everything needed in order to launch the guest.
   - The shim, running in the emulator container, would run the
   qemu-process.


What do you think? Feedback is much appreciated.

Best Regards,
Itamar Holder.

[1] https://github.com/kubevirt/kubevirt/pull/8869
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20230326/4f5446be/attachment.htm>