[Linux-cluster] GFS2 - F_SETLK fails with "ENOSYS" after umount + mount

[Steven Whitehouse]

Hi,

On Wed, 2013-01-30 at 12:31 +0100, Kristian Grønfeldt Sørensen wrote:
> Hi,
> 
> I'm setting up a two-node cluster sharing a single GFS2 filesystem
> backed by a dual-primary DRBD-device (DRBD on top of LVM, so no CLVM
> involved).
> 
> I am experiencing more or less the same as the OP in this thread:
> http://www.redhat.com/archives/linux-cluster/2010-July/msg00136.html
> 

Well I'm not so sure about that. We never found out what the issue was
in that case, but in your case it seems that you are doing something
which should work. Also, in the msg00136 case it seems that the lock
request didn't work at all, whereas in your case it appears that it does
work until a umount/mount of one node - at least if I've understood it
correctly.

Which kernel and userspace are you using?

It would be a good plan to report this as a bug (or via support if you
are a supported customer and are using RHEL) as it should work
correctly,

Steve.


> I have an activemq-5.6.0 instance on each server that tries to lock a
> file on the GFS2-filesystem (using ).  
> 
> When i start the cluster, everything works as expected. The first
> activemq instance that starts up acquires the lock, the lock is released
> when the activemq exits, and the second instance takes the lock. 
> 
> The problem shows when I unmount and subsequently mount the GFS2
> filesystem  again on one of the nodes, or reboot one of the nodes (after
> having started at least one activemq instance.) 
> The I start seeing statements like this in the activemq log files:
> 
> Database /srv/activemq/queue#3a#2f#2fstat.#3e/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: Function not implemented | org.apache.activemq.store.kahadb.MessageDatabase
> 
> strace -f while that message is logged gives the following:
> 
> [pid  3549] stat("/srv/activemq/queue#3a#2f#2fstat.#3e", {st_mode=S_IFDIR|0755, st_size=3864, ...}) = 0
> [pid  3549] stat("/srv/activemq/queue#3a#2f#2fstat.#3e", {st_mode=S_IFDIR|0755, st_size=3864, ...}) = 0
> [pid  3549] open("/srv/activemq/queue#3a#2f#2fstat.#3e/lock", O_RDWR|O_CREAT, 0666) = 133
> [pid  3549] fstat(133, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
> [pid  3549] fcntl(133, F_GETFD)         = 0
> [pid  3549] fcntl(133, F_SETFD, FD_CLOEXEC) = 0
> [pid  3549] fstat(133, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
> [pid  3549] fstat(133, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
> [pid  3549] fcntl(133, F_SETLK, {type=F_WRLCK, whence=SEEK_SET, start=0, len=1}) = -1 ENOSYS (Function not implemented)
> [pid  3549] dup2(138, 133)              = 133
> [pid  3549] close(133)
> 
> As you can see, the "Function not implemented" originates from the
> F_SETLK fnctl that the JVM does. 
> The only way to recover from this state seems to be by unmounting the
> GFS2-filesystem on both nodes, then mounting it again again on both
> nodes. 
> 
> I've tried to isolate this by using a simpler testcase than starting two
> activemq instances. I ended up using the java sample from
> http://www.javabeat.net/2007/10/locking-files-using-java/ . 
> 
> I haven't managed to get the system in to a state where F_SETLK returns
> "Function no implemented" by only using the above FileLockTest class, (I
> need activemq in order to trigger the situation) but when the system is
> in that state, I can run FileLockTest, and it will print out the
> following stacktrace.
> 
> Exception in thread "main" java.io.IOException: Function not implemented
>         at sun.nio.ch.FileChannelImpl.lock0(Native Method)
>         at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:871)
>         at java.nio.channels.FileChannel.tryLock(FileChannel.java:962)
>         at FileLockTest.main(FileLockTest.java:15)
> 
> 
> If I run this on the other server (where the GFS2 fs was not unmounted
> and mounted again), it works correctly. 
> 
> Any ideas to what happens, and why?
> 
> BR
> Kristian Sørensen
> 
> -- 
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster