removing VMs logs

Nikolay Shirokovskiy nshirokovskiy at virtuozzo.com
Fri Feb 11 07:07:27 UTC 2022


ср, 9 февр. 2022 г. в 12:30, Daniel P. Berrangé <berrange at redhat.com>:

> On Mon, Feb 07, 2022 at 02:04:14PM +0300, Nikolay Shirokovskiy wrote:
> > пн, 7 февр. 2022 г. в 13:29, Michal Prívozník <mprivozn at redhat.com>:
> >
> > > On 2/7/22 09:22, Nikolay Shirokovskiy wrote:
> > > > Hi, all.
> > > >
> > > > Libvirt QEMU driver writes QEMU process log to
> > > > /var/log/libvirt/qemu/<VM_NAME>.log file. This file is owned by
> libvirt
> > > > that is no API client knows about it and no client will remove it.
> Thus
> > > > this file stays forever. I guess it does not cause harddisk space
> waste
> > > > in some deploys as the number of VMs thru node lifecycle is not very
> > > large.
> > > >
> > > > In Virtuozzo we have a service which checks VMs disks every 5 min
> using
> > > > guestfs. This generates about 100k files and about 400MiB disk usage
> per
> > > > VM per year. This is a lot. I guess we could refine the service to
> avoid
> > > > this issue yet I think the issue is general and need to be addressed.
> > > >
> > > > If this is agreed then in terms of architecture should we have a
> > > > distinct timer/service to cleanup log files or we'd better clean up
> in
> > > > scope of qemu driver itself?
> > >
> > > I believe that this was one the issues that virtlogd tried to solve. By
> > > doing log rotation and keeping the last three files (by default). This
> > > can be fine tuned in virtlogd.conf.
> > >
> >
> > Hi, Michal.
> >
> > Yes virtlogd/logrotate can limit the size of logs for a single VM. But if
> > we keep logs forever for every VM that existed at some point in time then
> > the total size of logs can increase without any limit.
> >
> > I should mention that when we check VMs disks using guestfs it creates a
> > new transient VM every time. That is why we have so many logs left. But
> > most of these logs are not of interest - a VM was deleted a long time
> ago.
> >
> > So the suggestion is to keep logs for deleted VMs only for some period of
> > time, say a month.
>
> We have a logrotate config, that was pre-dating the existance of
> virtlogd, so its rules currently should be a no-op because
> virtlogd will have already rotated stuff before the logrotrate
> rules trigger.
>
> I wonder, however, if there is a way to set the logrotate rules
> to delete files, without interfering/clashing with virtlogd
> rollover.
>
>
I investigated this approach a bit. This piece of configuration looks
most promising:

    weekly
    maxsize 2097153
    # copytruncate is off
    nocreate
    maxage 30
    sharedscripts
    postrotate
        kill -HUP virtlogd
    endscript

We also need to support reopening log files on SIGHUP in virtlogd of course.

'maxage' allows you to remove outdated rotated files. 'nocreate' is also
required
as otherwise logrotate will create foo.log again with a fresh timestamp and
eventually foo.log will never be deleted. We need 'maxsize' instead of
'size'
also. The thing is 'maxage' is only applied on rotation which will never
happened
with 'size' if log is abandoned.

Unfortunately 'maxsize' allows weekly rotations which will clash with
virtlogd
rotations. I also think of introducing 'forceage' option instead of
weekly/maxsize
pair to ask logrotate to apply 'maxage' even without rotation. Still I think
in this way we can clash with virtlogd on managing log files.

I guess we can either make delete/rename in virtlogd/logrotate in a very
specific
order to handle races or introduce file locks to cooperate nicely. But to me
it does not look the right way. It will complicate logrotate because it will
need to cooperate with another log files manager.

I would add a new timer/service to cleanup log files. The service can be a
very
simple shell script. It will need to cooperate with virtlogd too and we
can use file locks on log files for this purpose.



>
> A completely different, perhaps complementary, approach would be
> to offer a flag to virDomainDestroy VIR_DOMAIN_DESTROY_REMOVE_LOGS
> that libguestfs could pass, on the basis that having these logs
> persist is essentially a waste of time for transient VMs like
> these. It could omit this VIR_DOMAIN_DESTROY_REMOVE_LOGS flag if
> asked to run in a debugging mode perhaps.
>
>
Yes this will make the log directory much cleaner. Even if we cleanup log
files on time basis we will still have a lot of not very meaningful logs
from guestfs in the log directory without this part.

Nikolay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20220211/eeb9c306/attachment-0001.htm>


More information about the libvir-list mailing list