[Ovirt-devel] Limiting node writes...

[Chris Lalancette]

Darryl L. Pierce wrote:
> Objective:
> Reduce the amount of writing done to the managed node's filesystem. 
> Since the managed node is run on an in-memory file system, excessive 
> writes to such things as log files will eventually cause an 
> out-of-memory condition. Our goal is to limit the amount of writes.
> 
> Discussion:
> Initially, rotating logs every hour and deleting all old logs is the 
> likely first pass.

Yes, this is the minimum way to ensure that we don't cause OOM on the managed
nodes.  It's what we should implement for now (and it should be easy enough just
to modify some cron scripts in the %post section of the ovirt-host-creator).

> 
> A usable feature would be for the WUI to push down (during the identify 
> phase, maybe?) a threshold for when logs should be aggressively deleted. 
> Then the admin can decide to hold onto logs until disk space hits some 
> threshold and then start throwing away old logs. That way some history 
> can be viewed if needed.

Actually, I'm not sure this is really useful.  Administrators really shouldn't
need to be aware of the threshold, and ideally they wouldn't have to log into
the individual nodes at all.

> 
> In future, a more preferable solution would be to have the oVirt server 
> collect logging information for all managed nodes centrally. However, 
> this would need to use an encrypted pipe to send the data.

This is the right way to do things.  When the managed node comes up, it could
pull some configuration data down from the WUI, and then it would know where to
send the log files, and how to do so (encrypted, not-encrypted, etc. etc.).
This way the administrator just has to go to one place (of his choosing) to look
at the logs from all of his machines.

That being said, I think the minimal "delete logs every hour" should prevent the
worst disasters for now, and we can visit this again in the future.

Chris Lalancette