From bagcib at itu.edu.tr Tue Jan 5 18:37:10 2016 From: bagcib at itu.edu.tr (B.Baransel =?utf-8?b?QkHEnkNJ?=) Date: Tue, 05 Jan 2016 20:37:10 +0200 Subject: [Linux-cluster] GFS2 mount hangs for some disks Message-ID: <20160105203710.Horde.PCltOrIy02t3hXcegg9sBA2@webmail.itu.edu.tr> Hi list, I have some problems with GFS2 with failed nodes. After one of the cluster nodes fenced and rebooted, it cannot mount some of the gfs2 file systems but hangs on the mount operation. No output. I've waited nearly 10 minutes to mount single disk but it didn't respond. Only solution is to shutdown all nodes and clean start of the cluster. I'm suspecting journal size or file system quotas. I have 8-node rhel-6 cluster with GFS2 formatted disks which are all mounted by all nodes. There are two types of disk: Type A : ~50 GB disk capacity 8 journal with size 512MB block-size: 1024 very small files (Avg: 50 byte - sym.links) ~500.000 file (inode) Usage: 10% Nearly no write IO (under 1000 file per day) No user quota (quota=off) Mount options: async,quota=off,nodiratime,noatime Tybe B : ~1 TB disk capacity 8 journal with size 512MB block-size: 4096 relatively small files (Avg: 20 KB) ~5.000.000 file (inode) Usage: 20% write IO ~50.000 file per day user quota is on (some of the users exceeded quota) Mount options: async,quota=on,nodiratime,noatime To improve performance, I set journal size to 512 MB instead of 128 MB default. All disk are connected with fiber from SAN Storage. All disk on cluster LVM. All nodes connected to each other with private Gb-switch. For example, after "node5" failed and fenced, it can re-enter the cluster. When i try "service gfs2 start", it can mount "Type A" disks, but hangs on the first "Tybe B" disk. Logs hangs on the "Trying to join cluster lock_dlm" message: ... Jan 05 00:01:52 node5 lvm[4090]: Found volume group "VG_of_TYPE_A" Jan 05 00:01:52 node5 lvm[4119]: Activated 2 logical volumes in volume group VG_of_TYPE_A Jan 05 00:01:52 node5 lvm[4119]: 2 logical volume(s) in volume group "VG_of_TYPE_A" now active Jan 05 00:01:52 node5 lvm[4119]: Wiping internal VG cache Jan 05 00:02:26 node5 kernel: Slow work thread pool: Starting up Jan 05 00:02:26 node5 kernel: Slow work thread pool: Ready Jan 05 00:02:26 node5 kernel: GFS2 (built Dec 12 2014 16:06:57) installed Jan 05 00:02:26 node5 kernel: GFS2: fsid=: Trying to join cluster "lock_dlm", "TESTCLS:typeA1" Jan 05 00:02:26 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: Joined cluster. Now mounting FS... Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: jid=5, already locked for use Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: jid=5: Looking at journal... Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: jid=5: Done Jan 05 00:02:27 node5 kernel: GFS2: fsid=: Trying to join cluster "lock_dlm", "TESTCLS:typeA2" Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: Joined cluster. Now mounting FS... Jan 05 00:02:28 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: jid=5, already locked for use Jan 05 00:02:28 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: jid=5: Looking at journal... Jan 05 00:02:28 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: jid=5: Done Jan 05 00:02:28 node5 kernel: GFS2: fsid=: Trying to join cluster "lock_dlm", "TESTCLS:typeB1" I've waited nearly 10 minutes in this state without respond or log. In this state, I cannot do `ls` in another nodes for this file system. Any idea of the cause of the problem? How is the cluster affected by journal size or count? -- B.Baransel BA?CI From lists at alteeve.ca Tue Jan 5 19:04:22 2016 From: lists at alteeve.ca (Digimer) Date: Tue, 5 Jan 2016 14:04:22 -0500 Subject: [Linux-cluster] GFS2 mount hangs for some disks In-Reply-To: <20160105203710.Horde.PCltOrIy02t3hXcegg9sBA2@webmail.itu.edu.tr> References: <20160105203710.Horde.PCltOrIy02t3hXcegg9sBA2@webmail.itu.edu.tr> Message-ID: <568C13B6.3020104@alteeve.ca> Can you re-ask this on the clusterlabs user's list? This list is being phased out. http://clusterlabs.org/mailman/listinfo/users digimer On 05/01/16 01:37 PM, B.Baransel BA?CI wrote: > Hi list, > > I have some problems with GFS2 with failed nodes. After one of the > cluster nodes fenced and rebooted, it cannot mount some of the gfs2 file > systems but hangs on the mount operation. No output. I've waited nearly > 10 minutes to mount single disk but it didn't respond. Only solution is > to shutdown all nodes and clean start of the cluster. I'm suspecting > journal size or file system quotas. > > I have 8-node rhel-6 cluster with GFS2 formatted disks which are all > mounted by all nodes. > There are two types of disk: > Type A : > ~50 GB disk capacity > 8 journal with size 512MB > block-size: 1024 > very small files (Avg: 50 byte - sym.links) > ~500.000 file (inode) > Usage: 10% > Nearly no write IO (under 1000 file per day) > No user quota (quota=off) > Mount options: async,quota=off,nodiratime,noatime > > Tybe B : > ~1 TB disk capacity > 8 journal with size 512MB > block-size: 4096 > relatively small files (Avg: 20 KB) > ~5.000.000 file (inode) > Usage: 20% > write IO ~50.000 file per day > user quota is on (some of the users exceeded quota) > Mount options: async,quota=on,nodiratime,noatime > > To improve performance, I set journal size to 512 MB instead of 128 MB > default. All disk are connected with fiber from SAN Storage. All disk on > cluster LVM. All nodes connected to each other with private Gb-switch. > > For example, after "node5" failed and fenced, it can re-enter the > cluster. When i try "service gfs2 start", it can mount "Type A" disks, > but hangs on the first "Tybe B" disk. Logs hangs on the "Trying to join > cluster lock_dlm" message: > > ... > Jan 05 00:01:52 node5 lvm[4090]: Found volume group "VG_of_TYPE_A" > Jan 05 00:01:52 node5 lvm[4119]: Activated 2 logical volumes in > volume group VG_of_TYPE_A > Jan 05 00:01:52 node5 lvm[4119]: 2 logical volume(s) in volume group > "VG_of_TYPE_A" now active > Jan 05 00:01:52 node5 lvm[4119]: Wiping internal VG cache > Jan 05 00:02:26 node5 kernel: Slow work thread pool: Starting up > Jan 05 00:02:26 node5 kernel: Slow work thread pool: Ready > Jan 05 00:02:26 node5 kernel: GFS2 (built Dec 12 2014 16:06:57) > installed > Jan 05 00:02:26 node5 kernel: GFS2: fsid=: Trying to join cluster > "lock_dlm", "TESTCLS:typeA1" > Jan 05 00:02:26 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: Joined > cluster. Now mounting FS... > Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: jid=5, > already locked for use > Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: jid=5: > Looking at journal... > Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: jid=5: Done > Jan 05 00:02:27 node5 kernel: GFS2: fsid=: Trying to join cluster > "lock_dlm", "TESTCLS:typeA2" > Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: Joined > cluster. Now mounting FS... > Jan 05 00:02:28 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: jid=5, > already locked for use > Jan 05 00:02:28 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: jid=5: > Looking at journal... > Jan 05 00:02:28 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: jid=5: Done > Jan 05 00:02:28 node5 kernel: GFS2: fsid=: Trying to join cluster > "lock_dlm", "TESTCLS:typeB1" > > > I've waited nearly 10 minutes in this state without respond or log. In > this state, I cannot do `ls` in another nodes for this file system. Any > idea of the cause of the problem? How is the cluster affected by journal > size or count? -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? From emi2fast at gmail.com Tue Jan 5 18:51:21 2016 From: emi2fast at gmail.com (emmanuel segura) Date: Tue, 5 Jan 2016 19:51:21 +0100 Subject: [Linux-cluster] GFS2 mount hangs for some disks In-Reply-To: <20160105203710.Horde.PCltOrIy02t3hXcegg9sBA2@webmail.itu.edu.tr> References: <20160105203710.Horde.PCltOrIy02t3hXcegg9sBA2@webmail.itu.edu.tr> Message-ID: 1: share your config, nobody has the crystal ball!!! 2: you need to be share that your fencing is working. 2016-01-05 19:37 GMT+01:00 B.Baransel BA?CI : > Hi list, > > I have some problems with GFS2 with failed nodes. After one of the cluster > nodes fenced and rebooted, it cannot mount some of the gfs2 file systems but > hangs on the mount operation. No output. I've waited nearly 10 minutes to > mount single disk but it didn't respond. Only solution is to shutdown all > nodes and clean start of the cluster. I'm suspecting journal size or file > system quotas. > > I have 8-node rhel-6 cluster with GFS2 formatted disks which are all mounted > by all nodes. > There are two types of disk: > Type A : > ~50 GB disk capacity > 8 journal with size 512MB > block-size: 1024 > very small files (Avg: 50 byte - sym.links) > ~500.000 file (inode) > Usage: 10% > Nearly no write IO (under 1000 file per day) > No user quota (quota=off) > Mount options: async,quota=off,nodiratime,noatime > > Tybe B : > ~1 TB disk capacity > 8 journal with size 512MB > block-size: 4096 > relatively small files (Avg: 20 KB) > ~5.000.000 file (inode) > Usage: 20% > write IO ~50.000 file per day > user quota is on (some of the users exceeded quota) > Mount options: async,quota=on,nodiratime,noatime > > To improve performance, I set journal size to 512 MB instead of 128 MB > default. All disk are connected with fiber from SAN Storage. All disk on > cluster LVM. All nodes connected to each other with private Gb-switch. > > For example, after "node5" failed and fenced, it can re-enter the cluster. > When i try "service gfs2 start", it can mount "Type A" disks, but hangs on > the first "Tybe B" disk. Logs hangs on the "Trying to join cluster lock_dlm" > message: > > ... > Jan 05 00:01:52 node5 lvm[4090]: Found volume group "VG_of_TYPE_A" > Jan 05 00:01:52 node5 lvm[4119]: Activated 2 logical volumes in volume > group VG_of_TYPE_A > Jan 05 00:01:52 node5 lvm[4119]: 2 logical volume(s) in volume group > "VG_of_TYPE_A" now active > Jan 05 00:01:52 node5 lvm[4119]: Wiping internal VG cache > Jan 05 00:02:26 node5 kernel: Slow work thread pool: Starting up > Jan 05 00:02:26 node5 kernel: Slow work thread pool: Ready > Jan 05 00:02:26 node5 kernel: GFS2 (built Dec 12 2014 16:06:57) > installed > Jan 05 00:02:26 node5 kernel: GFS2: fsid=: Trying to join cluster > "lock_dlm", "TESTCLS:typeA1" > Jan 05 00:02:26 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: Joined > cluster. Now mounting FS... > Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: jid=5, > already locked for use > Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: jid=5: > Looking at journal... > Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: jid=5: Done > Jan 05 00:02:27 node5 kernel: GFS2: fsid=: Trying to join cluster > "lock_dlm", "TESTCLS:typeA2" > Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: Joined > cluster. Now mounting FS... > Jan 05 00:02:28 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: jid=5, > already locked for use > Jan 05 00:02:28 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: jid=5: > Looking at journal... > Jan 05 00:02:28 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: jid=5: Done > Jan 05 00:02:28 node5 kernel: GFS2: fsid=: Trying to join cluster > "lock_dlm", "TESTCLS:typeB1" > > > I've waited nearly 10 minutes in this state without respond or log. In this > state, I cannot do `ls` in another nodes for this file system. Any idea of > the cause of the problem? How is the cluster affected by journal size or > count? > -- > B.Baransel BA?CI > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- .~. /V\ // \\ /( )\ ^`~'^ From rpeterso at redhat.com Tue Jan 5 20:49:35 2016 From: rpeterso at redhat.com (Bob Peterson) Date: Tue, 5 Jan 2016 15:49:35 -0500 (EST) Subject: [Linux-cluster] GFS2 mount hangs for some disks In-Reply-To: <20160105203710.Horde.PCltOrIy02t3hXcegg9sBA2@webmail.itu.edu.tr> References: <20160105203710.Horde.PCltOrIy02t3hXcegg9sBA2@webmail.itu.edu.tr> Message-ID: <656087379.4943593.1452026975772.JavaMail.zimbra@redhat.com> ----- Original Message ----- > Hi list, > > I have some problems with GFS2 with failed nodes. After one of the > cluster nodes fenced and rebooted, it cannot mount some of the gfs2 > file systems but hangs on the mount operation. No output. I've waited > nearly 10 minutes to mount single disk but it didn't respond. Only > solution is to shutdown all nodes and clean start of the cluster. I'm > suspecting journal size or file system quotas. > > I have 8-node rhel-6 cluster with GFS2 formatted disks which are all > mounted by all nodes. > There are two types of disk: > Type A : > ~50 GB disk capacity > 8 journal with size 512MB > block-size: 1024 > very small files (Avg: 50 byte - sym.links) > ~500.000 file (inode) > Usage: 10% > Nearly no write IO (under 1000 file per day) > No user quota (quota=off) > Mount options: async,quota=off,nodiratime,noatime > > Tybe B : > ~1 TB disk capacity > 8 journal with size 512MB > block-size: 4096 > relatively small files (Avg: 20 KB) > ~5.000.000 file (inode) > Usage: 20% > write IO ~50.000 file per day > user quota is on (some of the users exceeded quota) > Mount options: async,quota=on,nodiratime,noatime > > To improve performance, I set journal size to 512 MB instead of 128 MB > default. All disk are connected with fiber from SAN Storage. All disk > on cluster LVM. All nodes connected to each other with private > Gb-switch. > > For example, after "node5" failed and fenced, it can re-enter the > cluster. When i try "service gfs2 start", it can mount "Type A" disks, > but hangs on the first "Tybe B" disk. Logs hangs on the "Trying to > join cluster lock_dlm" message: > > ... > Jan 05 00:01:52 node5 lvm[4090]: Found volume group "VG_of_TYPE_A" > Jan 05 00:01:52 node5 lvm[4119]: Activated 2 logical volumes in > volume group VG_of_TYPE_A > Jan 05 00:01:52 node5 lvm[4119]: 2 logical volume(s) in volume > group "VG_of_TYPE_A" now active > Jan 05 00:01:52 node5 lvm[4119]: Wiping internal VG cache > Jan 05 00:02:26 node5 kernel: Slow work thread pool: Starting up > Jan 05 00:02:26 node5 kernel: Slow work thread pool: Ready > Jan 05 00:02:26 node5 kernel: GFS2 (built Dec 12 2014 16:06:57) > installed > Jan 05 00:02:26 node5 kernel: GFS2: fsid=: Trying to join cluster > "lock_dlm", "TESTCLS:typeA1" > Jan 05 00:02:26 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: Joined > cluster. Now mounting FS... > Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: jid=5, > already locked for use > Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: jid=5: > Looking at journal... > Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: jid=5: Done > Jan 05 00:02:27 node5 kernel: GFS2: fsid=: Trying to join cluster > "lock_dlm", "TESTCLS:typeA2" > Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: Joined > cluster. Now mounting FS... > Jan 05 00:02:28 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: jid=5, > already locked for use > Jan 05 00:02:28 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: jid=5: > Looking at journal... > Jan 05 00:02:28 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: jid=5: Done > Jan 05 00:02:28 node5 kernel: GFS2: fsid=: Trying to join cluster > "lock_dlm", "TESTCLS:typeB1" > > > I've waited nearly 10 minutes in this state without respond or log. In > this state, I cannot do `ls` in another nodes for this file system. > Any idea of the cause of the problem? How is the cluster affected by > journal size or count? > -- > B.Baransel BA?CI > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster Hi, If mount hangs, it's hard to say what it's doing. It could be waiting for a dlm lock, which is waiting on a pending fencing option. There have been occasional hangs discovered in journal replay, but not for a long time. It's less likely. What kernel version is this? December 12, 2014 is more than a year old, so it might be something we've already found and fixed. If this is RHEL6 or Centos6 or similar, you could try catting the /proc//stack file of the mount helper process, aka mount.gfs2 and see what it's doing. Normally, dlm recovery and gfs2 recovery take only a few seconds time. The size of journals and number of journals will likely have no effect. If I was a betting man, I'd bet that GFS2 is waiting for DLM, and DLM is waiting for a fence operation to be completed successfully before continuing. If this is rhel6 or earlier, you could do "group_tool dump" to find out if the cluster membership is sane or if it's waiting for something like this. Regards, Bob Peterson Red Hat File Systems From bagcib at itu.edu.tr Wed Jan 6 10:08:06 2016 From: bagcib at itu.edu.tr (B.Baransel =?utf-8?b?QkHEnkNJ?=) Date: Wed, 06 Jan 2016 12:08:06 +0200 Subject: [Linux-cluster] GFS2 mount hangs for some disks In-Reply-To: <656087379.4943593.1452026975772.JavaMail.zimbra@redhat.com> References: <20160105203710.Horde.PCltOrIy02t3hXcegg9sBA2@webmail.itu.edu.tr> <656087379.4943593.1452026975772.JavaMail.zimbra@redhat.com> Message-ID: <20160106120806.Horde.dTbrPvAn0nk1zF64H60NwQ1@webmail.itu.edu.tr> Alinti Bob Peterson > ----- Original Message ----- >> Hi list, >> >> I have some problems with GFS2 with failed nodes. After one of the >> cluster nodes fenced and rebooted, it cannot mount some of the gfs2 >> file systems but hangs on the mount operation. No output. I've waited >> nearly 10 minutes to mount single disk but it didn't respond. Only >> solution is to shutdown all nodes and clean start of the cluster. I'm >> suspecting journal size or file system quotas. >> >> I have 8-node rhel-6 cluster with GFS2 formatted disks which are all >> mounted by all nodes. >> There are two types of disk: >> Type A : >> ~50 GB disk capacity >> 8 journal with size 512MB >> block-size: 1024 >> very small files (Avg: 50 byte - sym.links) >> ~500.000 file (inode) >> Usage: 10% >> Nearly no write IO (under 1000 file per day) >> No user quota (quota=off) >> Mount options: async,quota=off,nodiratime,noatime >> >> Tybe B : >> ~1 TB disk capacity >> 8 journal with size 512MB >> block-size: 4096 >> relatively small files (Avg: 20 KB) >> ~5.000.000 file (inode) >> Usage: 20% >> write IO ~50.000 file per day >> user quota is on (some of the users exceeded quota) >> Mount options: async,quota=on,nodiratime,noatime >> >> To improve performance, I set journal size to 512 MB instead of 128 MB >> default. All disk are connected with fiber from SAN Storage. All disk >> on cluster LVM. All nodes connected to each other with private >> Gb-switch. >> >> For example, after "node5" failed and fenced, it can re-enter the >> cluster. When i try "service gfs2 start", it can mount "Type A" disks, >> but hangs on the first "Tybe B" disk. Logs hangs on the "Trying to >> join cluster lock_dlm" message: >> >> ... >> Jan 05 00:01:52 node5 lvm[4090]: Found volume group "VG_of_TYPE_A" >> Jan 05 00:01:52 node5 lvm[4119]: Activated 2 logical volumes in >> volume group VG_of_TYPE_A >> Jan 05 00:01:52 node5 lvm[4119]: 2 logical volume(s) in volume >> group "VG_of_TYPE_A" now active >> Jan 05 00:01:52 node5 lvm[4119]: Wiping internal VG cache >> Jan 05 00:02:26 node5 kernel: Slow work thread pool: Starting up >> Jan 05 00:02:26 node5 kernel: Slow work thread pool: Ready >> Jan 05 00:02:26 node5 kernel: GFS2 (built Dec 12 2014 16:06:57) >> installed >> Jan 05 00:02:26 node5 kernel: GFS2: fsid=: Trying to join cluster >> "lock_dlm", "TESTCLS:typeA1" >> Jan 05 00:02:26 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: Joined >> cluster. Now mounting FS... >> Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: jid=5, >> already locked for use >> Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: jid=5: >> Looking at journal... >> Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: jid=5: Done >> Jan 05 00:02:27 node5 kernel: GFS2: fsid=: Trying to join cluster >> "lock_dlm", "TESTCLS:typeA2" >> Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: Joined >> cluster. Now mounting FS... >> Jan 05 00:02:28 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: jid=5, >> already locked for use >> Jan 05 00:02:28 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: jid=5: >> Looking at journal... >> Jan 05 00:02:28 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: jid=5: Done >> Jan 05 00:02:28 node5 kernel: GFS2: fsid=: Trying to join cluster >> "lock_dlm", "TESTCLS:typeB1" >> >> >> I've waited nearly 10 minutes in this state without respond or log. In >> this state, I cannot do `ls` in another nodes for this file system. >> Any idea of the cause of the problem? How is the cluster affected by >> journal size or count? >> -- >> B.Baransel BA?CI >> >> -- >> Linux-cluster mailing list >> Linux-cluster at redhat.com >> https://www.redhat.com/mailman/listinfo/linux-cluster > > Hi, > > If mount hangs, it's hard to say what it's doing. It could be waiting > for a dlm lock, which is waiting on a pending fencing option. > > There have been occasional hangs discovered in journal replay, but not > for a long time. It's less likely. What kernel version is this? > December 12, 2014 is more than a year old, so it might be something > we've already found and fixed. If this is RHEL6 or Centos6 or similar, > you could try catting the /proc//stack file of the mount helper > process, aka mount.gfs2 and see what it's doing. > OS is Rhel-6 and kernel is 2.6.32-504.3.3.el6.x86_64. Actually this problem began with the increasing of the data. At beginning with very low data and IO, this problem didn't exist. This cluster system is isolated and don't get updates. So, no kernel or no package change for a year. Also, I execute fsck on every disk after each crash. In next crash, i will look to stack file, but I cannot make system crash right now, because it's used by other services. > Normally, dlm recovery and gfs2 recovery take only a few seconds time. > The size of journals and number of journals will likely have no effect. > If I was a betting man, I'd bet that GFS2 is waiting for DLM, and > DLM is waiting for a fence operation to be completed successfully > before continuing. If this is rhel6 or earlier, you could do > "group_tool dump" to find out if the cluster membership is sane or > if it's waiting for something like this. Fence devices are working correctly. Cluster master calls fence device (ipmi) and failed node restarts. Then I can start "cman" and "clvmd" succesfully. Until this point there no problem. Also all other nodes are working correctly. Thus, I don't think there is a waiting fence operation. And this problem started when the data on the disk increased. My cluster conf like this: Note: Fence agent is a script and works correctly. It calls ipmi or ilo fence agents and returns 0 after success. thanks -- B.Baransel BA?CI From emi2fast at gmail.com Wed Jan 6 10:42:17 2016 From: emi2fast at gmail.com (emmanuel segura) Date: Wed, 6 Jan 2016 11:42:17 +0100 Subject: [Linux-cluster] GFS2 mount hangs for some disks In-Reply-To: <20160106120806.Horde.dTbrPvAn0nk1zF64H60NwQ1@webmail.itu.edu.tr> References: <20160105203710.Horde.PCltOrIy02t3hXcegg9sBA2@webmail.itu.edu.tr> <656087379.4943593.1452026975772.JavaMail.zimbra@redhat.com> <20160106120806.Horde.dTbrPvAn0nk1zF64H60NwQ1@webmail.itu.edu.tr> Message-ID: Maybe you don't think you have a problem with your fencing, but anyway, you can try to use group_tool dump when you have the problem, In this way we can exclude the fencing and dlm waiting for fencing. 2016-01-06 11:08 GMT+01:00 B.Baransel BA?CI : > > Alinti Bob Peterson > >> ----- Original Message ----- >>> >>> Hi list, >>> >>> I have some problems with GFS2 with failed nodes. After one of the >>> cluster nodes fenced and rebooted, it cannot mount some of the gfs2 >>> file systems but hangs on the mount operation. No output. I've waited >>> nearly 10 minutes to mount single disk but it didn't respond. Only >>> solution is to shutdown all nodes and clean start of the cluster. I'm >>> suspecting journal size or file system quotas. >>> >>> I have 8-node rhel-6 cluster with GFS2 formatted disks which are all >>> mounted by all nodes. >>> There are two types of disk: >>> Type A : >>> ~50 GB disk capacity >>> 8 journal with size 512MB >>> block-size: 1024 >>> very small files (Avg: 50 byte - sym.links) >>> ~500.000 file (inode) >>> Usage: 10% >>> Nearly no write IO (under 1000 file per day) >>> No user quota (quota=off) >>> Mount options: async,quota=off,nodiratime,noatime >>> >>> Tybe B : >>> ~1 TB disk capacity >>> 8 journal with size 512MB >>> block-size: 4096 >>> relatively small files (Avg: 20 KB) >>> ~5.000.000 file (inode) >>> Usage: 20% >>> write IO ~50.000 file per day >>> user quota is on (some of the users exceeded quota) >>> Mount options: async,quota=on,nodiratime,noatime >>> >>> To improve performance, I set journal size to 512 MB instead of 128 MB >>> default. All disk are connected with fiber from SAN Storage. All disk >>> on cluster LVM. All nodes connected to each other with private >>> Gb-switch. >>> >>> For example, after "node5" failed and fenced, it can re-enter the >>> cluster. When i try "service gfs2 start", it can mount "Type A" disks, >>> but hangs on the first "Tybe B" disk. Logs hangs on the "Trying to >>> join cluster lock_dlm" message: >>> >>> ... >>> Jan 05 00:01:52 node5 lvm[4090]: Found volume group "VG_of_TYPE_A" >>> Jan 05 00:01:52 node5 lvm[4119]: Activated 2 logical volumes in >>> volume group VG_of_TYPE_A >>> Jan 05 00:01:52 node5 lvm[4119]: 2 logical volume(s) in volume >>> group "VG_of_TYPE_A" now active >>> Jan 05 00:01:52 node5 lvm[4119]: Wiping internal VG cache >>> Jan 05 00:02:26 node5 kernel: Slow work thread pool: Starting up >>> Jan 05 00:02:26 node5 kernel: Slow work thread pool: Ready >>> Jan 05 00:02:26 node5 kernel: GFS2 (built Dec 12 2014 16:06:57) >>> installed >>> Jan 05 00:02:26 node5 kernel: GFS2: fsid=: Trying to join cluster >>> "lock_dlm", "TESTCLS:typeA1" >>> Jan 05 00:02:26 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: Joined >>> cluster. Now mounting FS... >>> Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: jid=5, >>> already locked for use >>> Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: jid=5: >>> Looking at journal... >>> Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA1.5: jid=5: >>> Done >>> Jan 05 00:02:27 node5 kernel: GFS2: fsid=: Trying to join cluster >>> "lock_dlm", "TESTCLS:typeA2" >>> Jan 05 00:02:27 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: Joined >>> cluster. Now mounting FS... >>> Jan 05 00:02:28 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: jid=5, >>> already locked for use >>> Jan 05 00:02:28 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: jid=5: >>> Looking at journal... >>> Jan 05 00:02:28 node5 kernel: GFS2: fsid=TESTCLS:typeA2.5: jid=5: >>> Done >>> Jan 05 00:02:28 node5 kernel: GFS2: fsid=: Trying to join cluster >>> "lock_dlm", "TESTCLS:typeB1" >>> >>> >>> I've waited nearly 10 minutes in this state without respond or log. In >>> this state, I cannot do `ls` in another nodes for this file system. >>> Any idea of the cause of the problem? How is the cluster affected by >>> journal size or count? >>> -- >>> B.Baransel BA?CI >>> >>> -- >>> Linux-cluster mailing list >>> Linux-cluster at redhat.com >>> https://www.redhat.com/mailman/listinfo/linux-cluster >> >> >> Hi, >> >> If mount hangs, it's hard to say what it's doing. It could be waiting >> for a dlm lock, which is waiting on a pending fencing option. >> >> There have been occasional hangs discovered in journal replay, but not >> for a long time. It's less likely. What kernel version is this? >> December 12, 2014 is more than a year old, so it might be something >> we've already found and fixed. If this is RHEL6 or Centos6 or similar, >> you could try catting the /proc//stack file of the mount helper >> process, aka mount.gfs2 and see what it's doing. >> > > OS is Rhel-6 and kernel is 2.6.32-504.3.3.el6.x86_64. Actually this problem > began with the increasing of the data. At beginning with very low data and > IO, this problem didn't exist. This cluster system is isolated and don't get > updates. So, no kernel or no package change for a year. Also, I execute fsck > on every disk after each crash. > In next crash, i will look to stack file, but I cannot make system crash > right now, because it's used by other services. > >> Normally, dlm recovery and gfs2 recovery take only a few seconds time. >> The size of journals and number of journals will likely have no effect. >> If I was a betting man, I'd bet that GFS2 is waiting for DLM, and >> DLM is waiting for a fence operation to be completed successfully >> before continuing. If this is rhel6 or earlier, you could do >> "group_tool dump" to find out if the cluster membership is sane or >> if it's waiting for something like this. > > > Fence devices are working correctly. Cluster master calls fence device > (ipmi) and failed node restarts. Then I can start "cman" and "clvmd" > succesfully. Until this point there no problem. Also all other nodes are > working correctly. Thus, I don't think there is a waiting fence operation. > And this problem started when the data on the disk increased. > > My cluster conf like this: > > > > > > > > > nodeid="5"> > > > > > > > > > > nodeid="11"> > > > > > > > > > > nodeid="12"> > > > > > > > > > > nodeid="21"> > > > > > > > > > > nodeid="22"> > > > > > > > > > > nodeid="23"> > > > > > > > > > > nodeid="31"> > > > > > > > > > > nodeid="32"> > > > > > > > > > > > post_join_delay="10"/> > > name="custom_fence"/> > > > > name="NFS" ordered="1" restricted="1"> > > > > > > > > address="192.168.1.34/24" sleeptime="10"/> > name="NFS_service_resource"/> > > exclusive="1" name="NFS_service" recovery="relocate"> > ref="192.168.1.34/24"> > > > > > > > > > > Note: Fence agent is a script and works correctly. It calls ipmi or ilo > fence agents and returns 0 after success. > > > thanks > -- > B.Baransel BA?CI > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster -- .~. /V\ // \\ /( )\ ^`~'^