[libvirt] [RFC] cgroup settings and systemd daemon-reload conflict

[Nikolay Shirokovskiy]

On 14.02.2018 13:34, Daniel P. Berrangé wrote:
> On Tue, Jan 30, 2018 at 10:34:14AM +0300, Nikolay Shirokovskiy wrote:
>> Hi, all.
>>
>> It turns out that systemd daemon-reload reset settings that are managable
>> thru 'systemctl set-property' interface.
>>
>>> virsh schedinfo tst3  | grep global_quota
>> global_quota   : -1
>>> virsh schedinfo tst3 --set global_quota=50000 | grep global_quota
>> global_quota   : 50000
>>> systemctl daemon-reload
>>> virsh schedinfo tst3  | grep global_quota
>> global_quota   : -1
>>
>> This behaviour does not limited to cpu controller, same for blkio for example.
>> I checked different versions of systemd (219 - Feb 15, and quite recent 236 - Dec 17)
>> to make sure it is not kind of bug of old version. So systemd does not play well
>> with direct writes to cgroup parameters that managable thru systemd. Looks like
>> libvirtd needs to use systemd's dbus interface to change all such parameters.
>>
>> I only wonder how this can be unnoticed for such long time (creating cgroup for domain
>> thru systemd - Jul 2013) as daemon-reload is called upon libvirtd package update. May
>> be I miss something?
> 
> I guess the reasons its unnoticed is that using global_* constants is
> relatively uncommon. Traditionally we had the per-vCPU tunables for
> quota and the global stuff was a newer addition.  Also for a long
> time I didn't think systemd even supported the quota+period settings
> in unit files at all.
> 
> It is incredibly frustrating that systemd would just reset th cgroups
> settings on daemon reload - something which ostensibly is supposed to
> be used to reload config files on disk. Since this behaviour has existed
> a long time though, I guess we have little choice but to cope with it
> now.
> 
> We kind of need to use the systemd dbus API more in order to support
> cgroups v2 properly too.
> 
> None of this will be pleasant changes to make though as this area of
> code is fairly complex. Also I don't think there's any nice way to
> introspect what properties a given version of systemd supports, which
> makes it hard to know when to set direct vs when to set via dbus. I
> think we would have to create the machine via dbus, then read back the
> auto-generated unit file for the machine to see what properties are
> listed as existing, then we can see which we now need to set via dbus.
> 

Worse then that. systemd hardcodes cpu.cfs_period_us to 100ms so even
if we set cpu.cfs_quota_us via systemd we will still have cpu.cfs_period_us reseted
on daemon-reload.

The best solution I see is to teach systemd not to store these
runtime settings in unit files but rather consult cgroup itself.

By the way some versions of systemd behave a bit differently. For
example for recent censos7 version one need to start another domain
thru libvirt after daemon-reload to get the described reset.

Nikolay