[Linux-cluster] samba on top of GFS

[Christopher R. Hertel]

On Mon, Nov 01, 2004 at 12:30:47PM -0800, Alan Wood wrote:
> I am running a cluster with GFS-formatted file systems mounted on multiple 
> nodes.  What I was hoping to do was to set up one node running httpd to be 
> my webserver and another node running samba to share the same data 
> internally.
> What I am getting when running that is instability.

Yeah.  This is a known problem.  The reason is that Samba must maintain a 
great deal of metadata internally.  This works well enough with multiple 
Samba processes running on a single machine dealing (more or less) 
directly with the filesystem.

The problem is that Samba must keep track of translations between Posix 
and Windows metadata, locking semantics, file sharing mode semantics, etc.

I had assumed that this would only be a problem if Samba was running on 
multiple machines all GFS-sharing the same back-end block storage.  Your 
report suggests that there's more to the interaction between Samba and GFS 
than I had anticipated.  Interesting...

> The samba serving node 
> keeps crashing.  I have heartbeat set up so that failover happens to the 
> webserver node, at which point the system apparently behaves well.

Which kind of failover?  Do you start Samba on the webserver node?  It 
would be interesting to know if the two run well together on the same 
node, but fail on separate nodes.

> After reading a few articles on the list it seemed to me that the problem 
> might be samba using oplocks or some other caching mechanism that breaks 
> synchronization.

Yeah... that was my next question...

> I tried turning oplocks=off in my smb.conf file, but that 
> made the system unusably slow (over 3 minutes to right-click on a two-meg 
> file).

Curious.

...but did it fix the other problems?

I'd really love to work with someone to figure all this out.  (Hint hint.)
:)

> I am also not sure that is the extent of the problem, as I seem to be able 
> to re-create the crash simply by accessing the same file on multiple 
> clients just via samba (which locking should be able to handle).

Should be...

> If the 
> problem were merely that the remote node and the samba node were both
> accessing an oplocked file I could understand, but that doesn't always seem 
> to be the case.

There's more here than I can figure out just from the description.  It'd 
take some digging along-side someone who knows GFS.

> has anyone had any success running the same type of setup?  I am also 
> serving nfs on the samba server, though with very little load there.

Is there any overlap in the files they're serving?

> below is the syslog output of a crash.  I'm running 2.6.8-1.521smp with a 
> GFS CVS dump from mid-september.
> -alan

Wish I could be more help...

Chris -)-----

-- 
"Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X
Samba Team -- http://www.samba.org/     -)-----   Christopher R. Hertel
jCIFS Team -- http://jcifs.samba.org/   -)-----   ubiqx development, uninq.
ubiqx Team -- http://www.ubiqx.org/     -)-----   crh at ubiqx.mn.org
OnLineBook -- http://ubiqx.org/cifs/    -)-----   crh at ubiqx.org