[libvirt] [PATCH 0/3] several cgroups/cpuset fixes

[John Ferlan]

>>
>> I'm leaning towards something in the test. I'll check if reverting
>> these changes alters the results. I don't imagine it will.
> 
> The real question is which thread it fails on and at what point in
> time. My patches only changed the order of operations where threads
> enter the cpuset cgroups at a slightly different time. And the qemu main
> thread never enters the parent group, it becomes an emulator-thread.
> Maybe you can point to exactly the assertion that fails. Including a
> link to the test code. And yes if you can confirm that the patches are
> to blame that would be a good first step ;).
> 
> Thanks,
> Henning
> 

Not quite sure how to answer your question about which thread - I'm
still at the point of figuring out the symptoms.

At startup, there's a priv->cgroup created. Then when vcpu, emulator,
iothread calls are made - each seems to create it's own cgroup thread
via virCgroupNewThread using the priv->cgroup, then make the adjustment,
and free the cgroup.

As for which test - it's part of the 'virt-test' suite. It's run on a
Red Hat internal system every night in order to help determine what/if
any changes made during the work day have caused a regression. You can
look up 'virt-test' on github, but it's being replaced by something
known as avacado.

A test was run with all the patches reverted and the test passed, so
something in the way things were moved. What's "interesting" (to me at
least) is that if I start the vm on that system used for the test, the
/proc/$pid/cgroup file is as follows:

10:hugetlb:/
9:perf_event:/machine.slice/machine-qemu\x2dvirt\x2dtests\x2dvm1.scope
8:blkio:/machine.slice/machine-qemu\x2dvirt\x2dtests\x2dvm1.scope
7:net_cls,net_prio:/machine.slice/machine-qemu\x2dvirt\x2dtests\x2dvm1.scope
6:freezer:/machine.slice/machine-qemu\x2dvirt\x2dtests\x2dvm1.scope
5:devices:/machine.slice/machine-qemu\x2dvirt\x2dtests\x2dvm1.scope
4:memory:/machine.slice/machine-qemu\x2dvirt\x2dtests\x2dvm1.scope
3:cpu,cpuacct:/machine.slice/machine-qemu\x2dvirt\x2dtests\x2dvm1.scope/emulator
2:cpuset:/machine.slice/machine-qemu\x2dvirt\x2dtests\x2dvm1.scope/emulator
1:name=systemd:/machine.slice/machine-qemu\x2dvirt\x2dtests\x2dvm1.scope

If I then "unrevert" the patches one by one (e.g. poor man's git
bisect), I find that patch 2/3 results in the following adjustment to
the /proc/$pid/cgroup file:

10:hugetlb:/
9:perf_event:/
8:blkio:/machine.slice
7:net_cls,net_prio:/
6:freezer:/
5:devices:/machine.slice
4:memory:/machine.slice
3:cpu,cpuacct:/machine.slice/machine-qemu\x2dvirt\x2dtests\x2dvm1.scope/emulator
2:cpuset:/machine.slice/machine-qemu\x2dvirt\x2dtests\x2dvm1.scope/emulator
1:name=systemd:/machine.slice/machine-qemu\x2dvirt\x2dtests\x2dvm1.scope

So, I have a candidate... It seems that by changing the AddTask from
using 'priv->cgroup' to a copy of the cgroup as created by
virCgroupNewThread in qemuSetupCgroupForEmulator, the /proc/$pid/cgroup
file only modifies the 'cpuset and cpu,cpuacct'. Thus changing the other
entries back to /machine.slice.  I'm not clear why that happens (yet).


BTW:
What's interesting with the file changes is that they differ from my f23
system in which the same revert processing would have the following
/proc/$pid/cgroup file when patch 2 is re-applied:

10:devices:/machine.slice/machine-qemu\x2dvirt\x2dtests\x2dvm1.scope
9:memory:/machine.slice/machine-qemu\x2dvirt\x2dtests\x2dvm1.scope
8:freezer:/
7:cpuset:/machine.slice/machine-qemu\x2dvirt\x2dtests\x2dvm1.scope/emulator
6:net_cls,net_prio:/
5:cpu,cpuacct:/machine.slice/machine-qemu\x2dvirt\x2dtests\x2dvm1.scope/emulator
4:blkio:/machine.slice/machine-qemu\x2dvirt\x2dtests\x2dvm1.scope
3:hugetlb:/
2:perf_event:/
1:name=systemd:/machine.slice/machine-qemu\x2dvirt\x2dtests\x2dvm1.scope


This does seem similar in a way to something I found while doing a
search to https://bugzilla.redhat.com/show_bug.cgi?id=1139223. It's not
completely the same, but the symptom of systemd overwriting non changing
controller entries feels similar.

John