[Linux-cluster] Synchronized filesystem

[Bill Rugolsky Jr.]

On Thu, Dec 29, 2005 at 03:05:17PM +0100, Jean-Eric wrote:
> Yes, I want local caching but also transparent writes. If I'm in the UK 
> I want to write FileServerInTheUk (ans that goes auto to 
> FileServerInTheUS which holds the storage) and if I'm in the US, I want 
> to write to US server.
> Is this case handled by AFS?

Disclaimer: I'm no expert on AFS, it just seemed more plausible than GFS
for your application.  Ask on their mail lists if you have questions.

What you do depends on whether the files that you are caching are different
(and well separated) from the files that you are writing, and whether
there is a single writer or multiple writers.  As a practical
matter, the size of the files can't be ignored either; 2Mbps isn't much
bandwidth if many users are writing bloated MS Office documents ...

AFS volume replication is designed for mostly static data, e.g., /usr
partitions.  They can be updated, but administratively, not in real-time.

There is also the small matter of (Windows) programs that open files
for update even though they may not write to them.  Depending on
the distributed filesystem in use, opening for write may immediately
invalidate or bypass caches.  AFS, IIRC, has seperate traversal paths for
read-only access and read/write access, due to the replication support.

> The last case is that I need to mount the partition with NFS *and* Samba 
> (we are a mixed Windows/UNIX shop) on Linux and Windows hosts (access 
> through AFS only is out of the question...) so I'm not sure that AFS 
> will do it... Or am I wrong?

It sounds like you want the file server on the UK side configured as
an AFS/NFSv4/whatever client and re-exporting the mount via NFS/Samba.
In that case, AFS server volume replication is not the right thing;
you instead want persistent client-side write-through caching for AFS
or NFSv4 (or CIFS).  I believe that the infrastructure for that is in
David Howells's and Steve Dickson's (as yet unmerged) fscache patches,
but I have no idea whether it is production ready (and available in an
Enterprise Linux distro) or how well it works with caches in the range of
tens or hundreds of gigabytes.  [Perhaps not well, but it couldn't be much
worse than trying to pull the file over the WAN.]  In order to populate
the cache on the UK side you'd probably have to set up a fscache client
and use tar or similar to populate the fscache partition, then overnight
it to the UK office.  That works for the initial cache load, but ongoing
maintenance would be a hassle and might require specialized tools.

There are commercial products in this niche, but I know nothing about
them.

Regards,

	Bill Rugolsky