From bergman at merctech.com Mon Aug 8 22:40:02 2016 From: bergman at merctech.com (bergman at merctech.com) Date: Mon, 8 Aug 2016 18:40:02 -0400 Subject: [Linux-cluster] unable to start mysql as a clustered service, OK stand-alone Message-ID: <20160808184002.6d23fa73@.> I've got a 3-node CentOS6 cluster and I'm trying to add mysql 5.1 as a new service. Other cluster services (IP addresses, Postgresql, applications) work fine. The mysql config file and data files are located on shared, cluster-wide storage (GPFS). On each node, I can successfully start mysql via: service mysqld start and via: rg_test test /etc/cluster/cluster.conf start service mysql (in each case, the corresponding command with the 'stop' option will also successfully shut down mysql). However, attempting to start the mysql service with clusvcadm results in the service failing over from one node to the next, and being marked as "stopped" after the last node. Each failover happens very quickly, in about 5 seconds. I suspect that rgmanager isn't waiting long enough for mysql to start before checking if it is running and I have added startup delays in cluster.conf, but they don't seem to be honored. Nothing is written into the mysql log file at this time -- no startup or failure messages. The only log entries (/var/log/messages, /var/log/cluster/*, etc) reference rgmanager, not the mysql process itself. Any suggestions? RHCS components: cman-3.0.12.1-78.el6.x86_64 luci-0.26.0-78.el6.centos.x86_64 rgmanager-3.0.12.1-26.el6_8.3.x86_64 ricci-0.16.2-86.el6.x86_64 --------------------- /etc/cluster/cluster.conf (edited) ----------------- -------------------------------------------------------------------------- --------------------- /var/log/cluster/rgmanager.log from attempt to start mysql with clusvcadm ----------------------- Aug 08 11:58:16 rgmanager Recovering failed service service:mysql Aug 08 11:58:16 rgmanager [ip] Link for eth2: Detected Aug 08 11:58:16 rgmanager [ip] Adding IPv4 address 192.168.169.173/24 to eth2 Aug 08 11:58:16 rgmanager [ip] Pinging addr 192.168.169.173 from dev eth2 Aug 08 11:58:18 rgmanager [ip] Sending gratuitous ARP: 192.168.169.173 c8:1f:66:e8:bb:34 brd ff:ff:ff:ff:ff:ff Aug 08 11:58:19 rgmanager [mysql] Verifying Configuration Of mysql:mysql Aug 08 11:58:19 rgmanager [mysql] Verifying Configuration Of mysql:mysql > Succeed Aug 08 11:58:19 rgmanager [mysql] Monitoring Service mysql:mysql Aug 08 11:58:19 rgmanager [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed Aug 08 11:58:19 rgmanager [mysql] Monitoring Service mysql:mysql > Service Is Not Running Aug 08 11:58:19 rgmanager [mysql] Starting Service mysql:mysql Aug 08 11:58:19 rgmanager [mysql] Looking For IP Address > Succeed - IP Address Found Aug 08 11:58:20 rgmanager [mysql] Starting Service mysql:mysql > Succeed Aug 08 11:58:21 rgmanager [mysql] Monitoring Service mysql:mysql Aug 08 11:58:21 rgmanager 1 events processed Aug 08 11:58:21 rgmanager [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed Aug 08 11:58:21 rgmanager [mysql] Monitoring Service mysql:mysql > Service Is Not Running Aug 08 11:58:21 rgmanager start on mysql "mysql" returned 7 (unspecified) Aug 08 11:58:21 rgmanager #68: Failed to start service:mysql; return value: 1 Aug 08 11:58:21 rgmanager Stopping service service:mysql Aug 08 11:58:21 rgmanager [mysql] Verifying Configuration Of mysql:mysql Aug 08 11:58:21 rgmanager [mysql] Verifying Configuration Of mysql:mysql > Succeed Aug 08 11:58:21 rgmanager [mysql] Stopping Service mysql:mysql Aug 08 11:58:21 rgmanager [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed - File Doesn't Exist Aug 08 11:58:21 rgmanager [mysql] Stopping Service mysql:mysql > Succeed -------------------------------------------------------------------------------- ------------------------ rg_test output ----------------------------------------- Running in test mode. Link for eth2: Detected [ip] Link for eth2: Detected Adding IPv4 address 192.168.169.173/24 to eth2 [ip] Adding IPv4 address 192.168.169.173/24 to eth2 Pinging addr 192.168.169.173 from dev eth2 [ip] Pinging addr 192.168.169.173 from dev eth2 Sending gratuitous ARP: 192.168.169.173 c8:1f:66:e8:bb:70 brd ff:ff:ff:ff:ff:ff [ip] Sending gratuitous ARP: 192.168.169.173 c8:1f:66:e8:bb:70 brd ff:ff:ff:ff:ff:ff Verifying Configuration Of mysql:mysql [mysql] Verifying Configuration Of mysql:mysql Verifying Configuration Of mysql:mysql > Succeed [mysql] Verifying Configuration Of mysql:mysql > Succeed Monitoring Service mysql:mysql [mysql] Monitoring Service mysql:mysql Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed Monitoring Service mysql:mysql > Service Is Not Running [mysql] Monitoring Service mysql:mysql > Service Is Not Running Starting Service mysql:mysql [mysql] Starting Service mysql:mysql Looking For IP Address > Succeed - IP Address Found [mysql] Looking For IP Address > Succeed - IP Address Found Starting Service mysql:mysql > Succeed [mysql] Starting Service mysql:mysql > Succeed Monitoring Service mysql:mysql [mysql] Monitoring Service mysql:mysql Monitoring Service mysql:mysql > Service Is Running [mysql] Monitoring Service mysql:mysql > Service Is Running Loading resource rule from /usr/share/cluster/checkquorum Loading resource rule from /usr/share/cluster/mysql.sh Loading resource rule from /usr/share/cluster/ip.sh Loading resource rule from /usr/share/cluster/SAPDatabase Loading resource rule from /usr/share/cluster/fs.sh Loading resource rule from /usr/share/cluster/fence_scsi_check.pl Loading resource rule from /usr/share/cluster/script.sh Loading resource rule from /usr/share/cluster/fence_scsi_check_hardreboot.pl Loading resource rule from /usr/share/cluster/samba.sh Loading resource rule from /usr/share/cluster/oradg.sh Loading resource rule from /usr/share/cluster/nfsexport.sh Loading resource rule from /usr/share/cluster/orainstance.sh Loading resource rule from /usr/share/cluster/service.sh Loading resource rule from /usr/share/cluster/postgres-8.sh Loading resource rule from /usr/share/cluster/named.sh Loading resource rule from /usr/share/cluster/bind-mount.sh Loading resource rule from /usr/share/cluster/svclib_nfslock Loading resource rule from /usr/share/cluster/lvm_by_lv.sh Loading resource rule from /usr/share/cluster/ASEHAagent.sh Loading resource rule from /usr/share/cluster/netfs.sh Loading resource rule from /usr/share/cluster/tomcat-6.sh Loading resource rule from /usr/share/cluster/clusterfs.sh Loading resource rule from /usr/share/cluster/lvm.sh Loading resource rule from /usr/share/cluster/openldap.sh Loading resource rule from /usr/share/cluster/vm.sh Loading resource rule from /usr/share/cluster/ocf-shellfuncs Loading resource rule from /usr/share/cluster/lvm_by_vg.sh Loading resource rule from /usr/share/cluster/SAPInstance Loading resource rule from /usr/share/cluster/nfsserver.sh Loading resource rule from /usr/share/cluster/nfsclient.sh Loading resource rule from /usr/share/cluster/apache.sh Loading resource rule from /usr/share/cluster/oracledb.sh Loading resource rule from /usr/share/cluster/oralistener.sh Starting mysql... Start of mysql complete --------------------------------------------------------------------------------- -- Mark Bergman From lists at alteeve.ca Tue Aug 9 03:02:21 2016 From: lists at alteeve.ca (Digimer) Date: Mon, 8 Aug 2016 23:02:21 -0400 Subject: [Linux-cluster] unable to start mysql as a clustered service, OK stand-alone In-Reply-To: <20160808184002.6d23fa73@.> References: <20160808184002.6d23fa73@.> Message-ID: <79633422-3585-ada7-3c19-53b0eaa13e66@alteeve.ca> Please ask again on the Clusterlabs - Users list. This list is (quite) deprecated now. http://clusterlabs.org/mailman/listinfo/users digimer On 08/08/16 06:40 PM, bergman at merctech.com wrote: > I've got a 3-node CentOS6 cluster and I'm trying to add mysql 5.1 as a new service. Other cluster > services (IP addresses, Postgresql, applications) work fine. > > The mysql config file and data files are located on shared, cluster-wide storage (GPFS). > > On each node, I can successfully start mysql via: > service mysqld start > and via: > rg_test test /etc/cluster/cluster.conf start service mysql > > (in each case, the corresponding command with the 'stop' option will also successfully shut down mysql). > > However, attempting to start the mysql service with clusvcadm results in the service failing over > from one node to the next, and being marked as "stopped" after the last node. > > Each failover happens very quickly, in about 5 seconds. I suspect that rgmanager isn't waiting long > enough for mysql to start before checking if it is running and I have added startup delays in > cluster.conf, but they don't seem to be honored. Nothing is written into the mysql log file at this > time -- no startup or failure messages. The only log entries (/var/log/messages, /var/log/cluster/*, > etc) reference rgmanager, not the mysql process itself. > > > Any suggestions? > > > RHCS components: > cman-3.0.12.1-78.el6.x86_64 > luci-0.26.0-78.el6.centos.x86_64 > rgmanager-3.0.12.1-26.el6_8.3.x86_64 > ricci-0.16.2-86.el6.x86_64 > > > --------------------- /etc/cluster/cluster.conf (edited) ----------------- > > > > > > > > > > > > > > > -------------------------------------------------------------------------- > > > --------------------- /var/log/cluster/rgmanager.log from attempt to start mysql with clusvcadm ----------------------- > Aug 08 11:58:16 rgmanager Recovering failed service service:mysql > Aug 08 11:58:16 rgmanager [ip] Link for eth2: Detected > Aug 08 11:58:16 rgmanager [ip] Adding IPv4 address 192.168.169.173/24 to eth2 > Aug 08 11:58:16 rgmanager [ip] Pinging addr 192.168.169.173 from dev eth2 > Aug 08 11:58:18 rgmanager [ip] Sending gratuitous ARP: 192.168.169.173 c8:1f:66:e8:bb:34 brd ff:ff:ff:ff:ff:ff > Aug 08 11:58:19 rgmanager [mysql] Verifying Configuration Of mysql:mysql > Aug 08 11:58:19 rgmanager [mysql] Verifying Configuration Of mysql:mysql > Succeed > Aug 08 11:58:19 rgmanager [mysql] Monitoring Service mysql:mysql > Aug 08 11:58:19 rgmanager [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed > Aug 08 11:58:19 rgmanager [mysql] Monitoring Service mysql:mysql > Service Is Not Running > Aug 08 11:58:19 rgmanager [mysql] Starting Service mysql:mysql > Aug 08 11:58:19 rgmanager [mysql] Looking For IP Address > Succeed - IP Address Found > Aug 08 11:58:20 rgmanager [mysql] Starting Service mysql:mysql > Succeed > Aug 08 11:58:21 rgmanager [mysql] Monitoring Service mysql:mysql > Aug 08 11:58:21 rgmanager 1 events processed > Aug 08 11:58:21 rgmanager [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed > Aug 08 11:58:21 rgmanager [mysql] Monitoring Service mysql:mysql > Service Is Not Running > Aug 08 11:58:21 rgmanager start on mysql "mysql" returned 7 (unspecified) > Aug 08 11:58:21 rgmanager #68: Failed to start service:mysql; return value: 1 > Aug 08 11:58:21 rgmanager Stopping service service:mysql > Aug 08 11:58:21 rgmanager [mysql] Verifying Configuration Of mysql:mysql > Aug 08 11:58:21 rgmanager [mysql] Verifying Configuration Of mysql:mysql > Succeed > Aug 08 11:58:21 rgmanager [mysql] Stopping Service mysql:mysql > Aug 08 11:58:21 rgmanager [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed - File Doesn't Exist > Aug 08 11:58:21 rgmanager [mysql] Stopping Service mysql:mysql > Succeed > -------------------------------------------------------------------------------- > > > ------------------------ rg_test output ----------------------------------------- > Running in test mode. > Link for eth2: Detected > [ip] Link for eth2: Detected > Adding IPv4 address 192.168.169.173/24 to eth2 > [ip] Adding IPv4 address 192.168.169.173/24 to eth2 > Pinging addr 192.168.169.173 from dev eth2 > [ip] Pinging addr 192.168.169.173 from dev eth2 > Sending gratuitous ARP: 192.168.169.173 c8:1f:66:e8:bb:70 brd ff:ff:ff:ff:ff:ff > [ip] Sending gratuitous ARP: 192.168.169.173 c8:1f:66:e8:bb:70 brd ff:ff:ff:ff:ff:ff > Verifying Configuration Of mysql:mysql > [mysql] Verifying Configuration Of mysql:mysql > Verifying Configuration Of mysql:mysql > Succeed > [mysql] Verifying Configuration Of mysql:mysql > Succeed > Monitoring Service mysql:mysql > [mysql] Monitoring Service mysql:mysql > Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed > [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed > Monitoring Service mysql:mysql > Service Is Not Running > [mysql] Monitoring Service mysql:mysql > Service Is Not Running > Starting Service mysql:mysql > [mysql] Starting Service mysql:mysql > Looking For IP Address > Succeed - IP Address Found > [mysql] Looking For IP Address > Succeed - IP Address Found > Starting Service mysql:mysql > Succeed > [mysql] Starting Service mysql:mysql > Succeed > Monitoring Service mysql:mysql > [mysql] Monitoring Service mysql:mysql > Monitoring Service mysql:mysql > Service Is Running > [mysql] Monitoring Service mysql:mysql > Service Is Running > Loading resource rule from /usr/share/cluster/checkquorum > Loading resource rule from /usr/share/cluster/mysql.sh > Loading resource rule from /usr/share/cluster/ip.sh > Loading resource rule from /usr/share/cluster/SAPDatabase > Loading resource rule from /usr/share/cluster/fs.sh > Loading resource rule from /usr/share/cluster/fence_scsi_check.pl > Loading resource rule from /usr/share/cluster/script.sh > Loading resource rule from /usr/share/cluster/fence_scsi_check_hardreboot.pl > Loading resource rule from /usr/share/cluster/samba.sh > Loading resource rule from /usr/share/cluster/oradg.sh > Loading resource rule from /usr/share/cluster/nfsexport.sh > Loading resource rule from /usr/share/cluster/orainstance.sh > Loading resource rule from /usr/share/cluster/service.sh > Loading resource rule from /usr/share/cluster/postgres-8.sh > Loading resource rule from /usr/share/cluster/named.sh > Loading resource rule from /usr/share/cluster/bind-mount.sh > Loading resource rule from /usr/share/cluster/svclib_nfslock > Loading resource rule from /usr/share/cluster/lvm_by_lv.sh > Loading resource rule from /usr/share/cluster/ASEHAagent.sh > Loading resource rule from /usr/share/cluster/netfs.sh > Loading resource rule from /usr/share/cluster/tomcat-6.sh > Loading resource rule from /usr/share/cluster/clusterfs.sh > Loading resource rule from /usr/share/cluster/lvm.sh > Loading resource rule from /usr/share/cluster/openldap.sh > Loading resource rule from /usr/share/cluster/vm.sh > Loading resource rule from /usr/share/cluster/ocf-shellfuncs > Loading resource rule from /usr/share/cluster/lvm_by_vg.sh > Loading resource rule from /usr/share/cluster/SAPInstance > Loading resource rule from /usr/share/cluster/nfsserver.sh > Loading resource rule from /usr/share/cluster/nfsclient.sh > Loading resource rule from /usr/share/cluster/apache.sh > Loading resource rule from /usr/share/cluster/oracledb.sh > Loading resource rule from /usr/share/cluster/oralistener.sh > Starting mysql... > Start of mysql complete > --------------------------------------------------------------------------------- > -- Digimer Papers and Projects: https://alteeve.ca/w/ What if the cure for cancer is trapped in the mind of a person without access to education? From jpokorny at redhat.com Thu Aug 11 09:10:49 2016 From: jpokorny at redhat.com (Jan =?utf-8?Q?Pokorn=C3=BD?=) Date: Thu, 11 Aug 2016 11:10:49 +0200 Subject: [Linux-cluster] [Cluster-devel] git repo of fence-virt In-Reply-To: <20150323224838.GC22309@redhat.com> References: <20150209.171912.399832125393796182.yamato@redhat.com> <20150209131424.GA12489@redhat.com> <28403ACB-026C-404A-9CCA-8B9CE03A705C@beekhof.net> <20150223142934.GA766919@redhat.com> <41143705-C6AB-4A65-9EA2-DD84532D4D10@beekhof.net> <20150323224838.GC22309@redhat.com> Message-ID: <20160811091049.GD8666@redhat.com> On 23/03/15 23:48 +0100, Jan Pokorn? wrote: > On 24/02/15 07:13 +1100, Andrew Beekhof wrote: >>> On 24 Feb 2015, at 1:29 am, Ryan McCabe wrote: >>> On Mon, Feb 23, 2015 at 07:15:42AM +1100, Andrew Beekhof wrote: >>>> Ryan, since we're trying to consolidate everything else in the >>>> clusterlabs org area of github, perhaps consider moving it there. >>>> You'll still have complete control over it. >>>> >>> >>> Sure, works for me. Could you or somebody else who has the access add me >>> to the clusterlabs org on github? >> >> done :) > > Maybe worth updating https://fedorahosted.org/cluster/wiki/FenceVirt > once again, then. Would save a "301 Moved Permanently" redirection > to https://github.com/ClusterLabs/fence-virt ;) There, fixed: https://fedorahosted.org/cluster/wiki/FenceVirt?action=diff&version=7 -- Jan (Poki) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From amjadcsu at gmail.com Mon Aug 29 13:23:33 2016 From: amjadcsu at gmail.com (Amjad Syed) Date: Mon, 29 Aug 2016 16:23:33 +0300 Subject: [Linux-cluster] Two node failover with shared storage and Luns in active/passive Message-ID: Hello, We have implemented a two node failover cluster using RHEL pcs in active passive mode We have shared storage with one LUN, Our concern is what if this LUN fails or there is hardware problem accessing the data in it. What is the best way to synchronize data between two LUNS ? any suggestion will help. Thanks, Amjad -------------- next part -------------- An HTML attachment was scrubbed... URL: From ricks at alldigital.com Mon Aug 29 17:17:18 2016 From: ricks at alldigital.com (Rick Stevens) Date: Mon, 29 Aug 2016 10:17:18 -0700 Subject: [Linux-cluster] Two node failover with shared storage and Luns in active/passive In-Reply-To: References: Message-ID: <08e0c0da-2538-c17b-3196-760e844483a9@alldigital.com> On 08/29/2016 06:23 AM, Amjad Syed wrote: > Hello, > We have implemented a two node failover cluster using RHEL pcs in > active passive mode > We have shared storage with one LUN, > Our concern is what if this LUN fails or there is hardware problem > accessing the data in it. > What is the best way to synchronize data between two LUNS ? any > suggestion will help. If you have two identical LUNs, you could run something like drbd to mirror one LUN to a backup: https://www.drbd.org/en/comp/drbd-linux-driver Note this is done at the block level. It doesn't look at filesystem- specific stuff, it just watches block-level disk writes to the primary device and replicates those to the backup over the network. There are other things you can do at a filesystem level, such as rsyncing often, corosync, etc. ---------------------------------------------------------------------- - Rick Stevens, Systems Engineer, AllDigital ricks at alldigital.com - - AIM/Skype: therps2 ICQ: 226437340 Yahoo: origrps2 - - - - Squawk! Pieces of Seven! Pieces of Seven! Parity Error! - ---------------------------------------------------------------------- From debjyoti.mail at gmail.com Mon Aug 29 18:53:32 2016 From: debjyoti.mail at gmail.com (Debjyoti Banerjee) Date: Tue, 30 Aug 2016 00:23:32 +0530 Subject: [Linux-cluster] Two node failover with shared storage and Luns in active/passive In-Reply-To: References: Message-ID: Hi Amjad, You should try glusterfs. Regards, DJ On Aug 29, 2016 6:59 PM, "Amjad Syed" wrote: > Hello, > We have implemented a two node failover cluster using RHEL pcs in active > passive mode > We have shared storage with one LUN, > Our concern is what if this LUN fails or there is hardware problem > accessing the data in it. > What is the best way to synchronize data between two LUNS ? any > suggestion will help. > > Thanks, > Amjad > > -- > Linux-cluster mailing list > Linux-cluster at redhat.com > https://www.redhat.com/mailman/listinfo/linux-cluster > -------------- next part -------------- An HTML attachment was scrubbed... URL: