removing VMs logs

[Daniel P. Berrangé]

On Mon, Feb 07, 2022 at 02:04:14PM +0300, Nikolay Shirokovskiy wrote:
> пн, 7 февр. 2022 г. в 13:29, Michal Prívozník <mprivozn at redhat.com>:
> 
> > On 2/7/22 09:22, Nikolay Shirokovskiy wrote:
> > > Hi, all.
> > >
> > > Libvirt QEMU driver writes QEMU process log to
> > > /var/log/libvirt/qemu/<VM_NAME>.log file. This file is owned by libvirt
> > > that is no API client knows about it and no client will remove it. Thus
> > > this file stays forever. I guess it does not cause harddisk space waste
> > > in some deploys as the number of VMs thru node lifecycle is not very
> > large.
> > >
> > > In Virtuozzo we have a service which checks VMs disks every 5 min using
> > > guestfs. This generates about 100k files and about 400MiB disk usage per
> > > VM per year. This is a lot. I guess we could refine the service to avoid
> > > this issue yet I think the issue is general and need to be addressed.
> > >
> > > If this is agreed then in terms of architecture should we have a
> > > distinct timer/service to cleanup log files or we'd better clean up in
> > > scope of qemu driver itself?
> >
> > I believe that this was one the issues that virtlogd tried to solve. By
> > doing log rotation and keeping the last three files (by default). This
> > can be fine tuned in virtlogd.conf.
> >
> 
> Hi, Michal.
> 
> Yes virtlogd/logrotate can limit the size of logs for a single VM. But if
> we keep logs forever for every VM that existed at some point in time then
> the total size of logs can increase without any limit.
> 
> I should mention that when we check VMs disks using guestfs it creates a
> new transient VM every time. That is why we have so many logs left. But
> most of these logs are not of interest - a VM was deleted a long time ago.
> 
> So the suggestion is to keep logs for deleted VMs only for some period of
> time, say a month.

We have a logrotate config, that was pre-dating the existance of
virtlogd, so its rules currently should be a no-op because
virtlogd will have already rotated stuff before the logrotrate
rules trigger. 

I wonder, however, if there is a way to set the logrotate rules
to delete files, without interfering/clashing with virtlogd
rollover.


A completely different, perhaps complementary, approach would be
to offer a flag to virDomainDestroy VIR_DOMAIN_DESTROY_REMOVE_LOGS
that libguestfs could pass, on the basis that having these logs
persist is essentially a waste of time for transient VMs like
these. It could omit this VIR_DOMAIN_DESTROY_REMOVE_LOGS flag if
asked to run in a debugging mode perhaps.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|