File caching?

[Michael McGlothlin]

This isn't for a web server although we might apply the same approach to
that if this speeds things up a lot.

I was going to store the cached copy on a RAM disk as many of the files are
larger than the 1MB limit of memlockd and I don't feel like coming up with
my own solution if I can avoid it.

Is there a way to know how much RAM is being used for file cache or to tell
it to use more? If the server has 128GB of RAM and typically uses half of
that for it's actual work will it use the rest as file cache? Likewise is
there a way to track/test if file stats are being pushed out of cache a lot?

We've been considering switching to SSD or RAM drives but it seems they'd
always be slower than system RAM and we haven't found a product that can
affordably store sufficient data. I couldn't find a product that just sits
between the disk and controller, or a controller that does this itself, and
adds a large RAM-based file cache either.

Thanks,
Michael McGlothlin


On Tue, Mar 23, 2010 at 12:52 PM, David Schwartz <davids at webmaster.com>wrote:

>
> Michael McGlothlin wrote:
>
> > I've been asked to cache some high traffic files on one of our server.
> > Is there an easy way to get ext3/ext4 filesystems to cache several GB
> > of files in memory at once? I'd like writes to happen normally but reads
> > to happen from RAM. (We have plenty of RAM so that isn't an issue.)
>
> > If that isn't possible I can cache the files myself. Does the filesystem
> > keep a cache in memory of the file attributes such as modification time?
> > So if I check for a change will the disk actually have to physically move
> > to check the mod time?
>
> I would first investigate whether your web server has some specific way to
> do this. Failing that, I strongly recommend just letting the disk cache do
> its job. If they really are frequently-accessed, they will stay in cache if
> sufficient RAM is available anyway. I would only suggest going further if
> you have specific latency requirements.
>
> If you do, I'd recommend simply using a separate program to map the files
> and then lock the pages in memory. The 'memlockd' program can do this. I'm
> not sure how well it handles file changes, but it shouldn't be difficult to
> modify it to restart if any file changes.
>
> The other possibility is to put the files on a ramdisk. You can use a
> scheduled script to update them from an on-disk copy if needed.
>
> Linux has good stat caching, so the need to move the disk to check the
> modification time will only occur if that information was pushed out of
> cache.
>
> DS
>
>
>
> _______________________________________________
> Ext3-users mailing list
> Ext3-users at redhat.com
> https://www.redhat.com/mailman/listinfo/ext3-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/ext3-users/attachments/20100323/6cf98ac6/attachment.htm>