[Ovirt-devel] Limiting node writes...

[Jeff Schroeder]

On Wed, Jun 25, 2008 at 7:40 AM, Alan Pevec <apevec at redhat.com> wrote:
> Chris Lalancette wrote:
>>
>> Darryl L. Pierce wrote:
>
> ...
>>
>> That being said, I think the minimal "delete logs every hour" should
>> prevent the
>> worst disasters for now, and we can visit this again in the future.
>
> We also must move frequently written folders to tmpfs - livecd overlay is
> implemented using 512MB dm-snapshot which could fill up quickly.
>
> For example, here's what livecd upstream has in its base kickstart:
> http://git.fedorahosted.org/git/?p=livecd;a=commitdiff;h=c0f49393a954084e7a3bb9a6564096054aa6e49d
>
> So 1st step is to let node run for a day, taking filesystem state (ls -lR or
> something more fancy?) periodically and analyze what and where is written.

For testing, prevent the frequently written folders from being on a
tmpfs. Then you can enable
block dumping and see exactly how much these devices are being written to.
Fedora should have CONFIG_BLK_DEV_IO_TRACE=y enabled by default.

echo 1 > /proc/sys/vm/block_dump
... wait a minute or so ...
echo 0 > /proc/sys/vm/block_dump

"grep WRITE kern.log" or "dmesg | grep WRITE"

To do this you would be much better off streaming syslog to another host.
If you keep block dump enabled for a full day the logs will be pretty big.

Here is a quick & dirty script thrown together to help troubleshoot some
production issues. It uses the same block dumping mechanism:
http://www.digitalprognosis.com/opensource/scripts/top-disk-users

-- 
Jeff Schroeder

Don't drink and derive, alcohol and analysis don't mix.
http://www.digitalprognosis.com