From bergman at merctech.com Mon Aug 8 22:40:02 2016
From: bergman at merctech.com (bergman at merctech.com)
Date: Mon, 8 Aug 2016 18:40:02 -0400
Subject: [Linux-cluster] unable to start mysql as a clustered service,
OK stand-alone
Message-ID: <20160808184002.6d23fa73@.>
I've got a 3-node CentOS6 cluster and I'm trying to add mysql 5.1 as a new service. Other cluster
services (IP addresses, Postgresql, applications) work fine.
The mysql config file and data files are located on shared, cluster-wide storage (GPFS).
On each node, I can successfully start mysql via:
service mysqld start
and via:
rg_test test /etc/cluster/cluster.conf start service mysql
(in each case, the corresponding command with the 'stop' option will also successfully shut down mysql).
However, attempting to start the mysql service with clusvcadm results in the service failing over
from one node to the next, and being marked as "stopped" after the last node.
Each failover happens very quickly, in about 5 seconds. I suspect that rgmanager isn't waiting long
enough for mysql to start before checking if it is running and I have added startup delays in
cluster.conf, but they don't seem to be honored. Nothing is written into the mysql log file at this
time -- no startup or failure messages. The only log entries (/var/log/messages, /var/log/cluster/*,
etc) reference rgmanager, not the mysql process itself.
Any suggestions?
RHCS components:
cman-3.0.12.1-78.el6.x86_64
luci-0.26.0-78.el6.centos.x86_64
rgmanager-3.0.12.1-26.el6_8.3.x86_64
ricci-0.16.2-86.el6.x86_64
--------------------- /etc/cluster/cluster.conf (edited) -----------------
--------------------------------------------------------------------------
--------------------- /var/log/cluster/rgmanager.log from attempt to start mysql with clusvcadm -----------------------
Aug 08 11:58:16 rgmanager Recovering failed service service:mysql
Aug 08 11:58:16 rgmanager [ip] Link for eth2: Detected
Aug 08 11:58:16 rgmanager [ip] Adding IPv4 address 192.168.169.173/24 to eth2
Aug 08 11:58:16 rgmanager [ip] Pinging addr 192.168.169.173 from dev eth2
Aug 08 11:58:18 rgmanager [ip] Sending gratuitous ARP: 192.168.169.173 c8:1f:66:e8:bb:34 brd ff:ff:ff:ff:ff:ff
Aug 08 11:58:19 rgmanager [mysql] Verifying Configuration Of mysql:mysql
Aug 08 11:58:19 rgmanager [mysql] Verifying Configuration Of mysql:mysql > Succeed
Aug 08 11:58:19 rgmanager [mysql] Monitoring Service mysql:mysql
Aug 08 11:58:19 rgmanager [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed
Aug 08 11:58:19 rgmanager [mysql] Monitoring Service mysql:mysql > Service Is Not Running
Aug 08 11:58:19 rgmanager [mysql] Starting Service mysql:mysql
Aug 08 11:58:19 rgmanager [mysql] Looking For IP Address > Succeed - IP Address Found
Aug 08 11:58:20 rgmanager [mysql] Starting Service mysql:mysql > Succeed
Aug 08 11:58:21 rgmanager [mysql] Monitoring Service mysql:mysql
Aug 08 11:58:21 rgmanager 1 events processed
Aug 08 11:58:21 rgmanager [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed
Aug 08 11:58:21 rgmanager [mysql] Monitoring Service mysql:mysql > Service Is Not Running
Aug 08 11:58:21 rgmanager start on mysql "mysql" returned 7 (unspecified)
Aug 08 11:58:21 rgmanager #68: Failed to start service:mysql; return value: 1
Aug 08 11:58:21 rgmanager Stopping service service:mysql
Aug 08 11:58:21 rgmanager [mysql] Verifying Configuration Of mysql:mysql
Aug 08 11:58:21 rgmanager [mysql] Verifying Configuration Of mysql:mysql > Succeed
Aug 08 11:58:21 rgmanager [mysql] Stopping Service mysql:mysql
Aug 08 11:58:21 rgmanager [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed - File Doesn't Exist
Aug 08 11:58:21 rgmanager [mysql] Stopping Service mysql:mysql > Succeed
--------------------------------------------------------------------------------
------------------------ rg_test output -----------------------------------------
Running in test mode.
Link for eth2: Detected
[ip] Link for eth2: Detected
Adding IPv4 address 192.168.169.173/24 to eth2
[ip] Adding IPv4 address 192.168.169.173/24 to eth2
Pinging addr 192.168.169.173 from dev eth2
[ip] Pinging addr 192.168.169.173 from dev eth2
Sending gratuitous ARP: 192.168.169.173 c8:1f:66:e8:bb:70 brd ff:ff:ff:ff:ff:ff
[ip] Sending gratuitous ARP: 192.168.169.173 c8:1f:66:e8:bb:70 brd ff:ff:ff:ff:ff:ff
Verifying Configuration Of mysql:mysql
[mysql] Verifying Configuration Of mysql:mysql
Verifying Configuration Of mysql:mysql > Succeed
[mysql] Verifying Configuration Of mysql:mysql > Succeed
Monitoring Service mysql:mysql
[mysql] Monitoring Service mysql:mysql
Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed
[mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed
Monitoring Service mysql:mysql > Service Is Not Running
[mysql] Monitoring Service mysql:mysql > Service Is Not Running
Starting Service mysql:mysql
[mysql] Starting Service mysql:mysql
Looking For IP Address > Succeed - IP Address Found
[mysql] Looking For IP Address > Succeed - IP Address Found
Starting Service mysql:mysql > Succeed
[mysql] Starting Service mysql:mysql > Succeed
Monitoring Service mysql:mysql
[mysql] Monitoring Service mysql:mysql
Monitoring Service mysql:mysql > Service Is Running
[mysql] Monitoring Service mysql:mysql > Service Is Running
Loading resource rule from /usr/share/cluster/checkquorum
Loading resource rule from /usr/share/cluster/mysql.sh
Loading resource rule from /usr/share/cluster/ip.sh
Loading resource rule from /usr/share/cluster/SAPDatabase
Loading resource rule from /usr/share/cluster/fs.sh
Loading resource rule from /usr/share/cluster/fence_scsi_check.pl
Loading resource rule from /usr/share/cluster/script.sh
Loading resource rule from /usr/share/cluster/fence_scsi_check_hardreboot.pl
Loading resource rule from /usr/share/cluster/samba.sh
Loading resource rule from /usr/share/cluster/oradg.sh
Loading resource rule from /usr/share/cluster/nfsexport.sh
Loading resource rule from /usr/share/cluster/orainstance.sh
Loading resource rule from /usr/share/cluster/service.sh
Loading resource rule from /usr/share/cluster/postgres-8.sh
Loading resource rule from /usr/share/cluster/named.sh
Loading resource rule from /usr/share/cluster/bind-mount.sh
Loading resource rule from /usr/share/cluster/svclib_nfslock
Loading resource rule from /usr/share/cluster/lvm_by_lv.sh
Loading resource rule from /usr/share/cluster/ASEHAagent.sh
Loading resource rule from /usr/share/cluster/netfs.sh
Loading resource rule from /usr/share/cluster/tomcat-6.sh
Loading resource rule from /usr/share/cluster/clusterfs.sh
Loading resource rule from /usr/share/cluster/lvm.sh
Loading resource rule from /usr/share/cluster/openldap.sh
Loading resource rule from /usr/share/cluster/vm.sh
Loading resource rule from /usr/share/cluster/ocf-shellfuncs
Loading resource rule from /usr/share/cluster/lvm_by_vg.sh
Loading resource rule from /usr/share/cluster/SAPInstance
Loading resource rule from /usr/share/cluster/nfsserver.sh
Loading resource rule from /usr/share/cluster/nfsclient.sh
Loading resource rule from /usr/share/cluster/apache.sh
Loading resource rule from /usr/share/cluster/oracledb.sh
Loading resource rule from /usr/share/cluster/oralistener.sh
Starting mysql...
Start of mysql complete
---------------------------------------------------------------------------------
--
Mark Bergman
From lists at alteeve.ca Tue Aug 9 03:02:21 2016
From: lists at alteeve.ca (Digimer)
Date: Mon, 8 Aug 2016 23:02:21 -0400
Subject: [Linux-cluster] unable to start mysql as a clustered service,
OK stand-alone
In-Reply-To: <20160808184002.6d23fa73@.>
References: <20160808184002.6d23fa73@.>
Message-ID: <79633422-3585-ada7-3c19-53b0eaa13e66@alteeve.ca>
Please ask again on the Clusterlabs - Users list. This list is (quite)
deprecated now.
http://clusterlabs.org/mailman/listinfo/users
digimer
On 08/08/16 06:40 PM, bergman at merctech.com wrote:
> I've got a 3-node CentOS6 cluster and I'm trying to add mysql 5.1 as a new service. Other cluster
> services (IP addresses, Postgresql, applications) work fine.
>
> The mysql config file and data files are located on shared, cluster-wide storage (GPFS).
>
> On each node, I can successfully start mysql via:
> service mysqld start
> and via:
> rg_test test /etc/cluster/cluster.conf start service mysql
>
> (in each case, the corresponding command with the 'stop' option will also successfully shut down mysql).
>
> However, attempting to start the mysql service with clusvcadm results in the service failing over
> from one node to the next, and being marked as "stopped" after the last node.
>
> Each failover happens very quickly, in about 5 seconds. I suspect that rgmanager isn't waiting long
> enough for mysql to start before checking if it is running and I have added startup delays in
> cluster.conf, but they don't seem to be honored. Nothing is written into the mysql log file at this
> time -- no startup or failure messages. The only log entries (/var/log/messages, /var/log/cluster/*,
> etc) reference rgmanager, not the mysql process itself.
>
>
> Any suggestions?
>
>
> RHCS components:
> cman-3.0.12.1-78.el6.x86_64
> luci-0.26.0-78.el6.centos.x86_64
> rgmanager-3.0.12.1-26.el6_8.3.x86_64
> ricci-0.16.2-86.el6.x86_64
>
>
> --------------------- /etc/cluster/cluster.conf (edited) -----------------
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --------------------------------------------------------------------------
>
>
> --------------------- /var/log/cluster/rgmanager.log from attempt to start mysql with clusvcadm -----------------------
> Aug 08 11:58:16 rgmanager Recovering failed service service:mysql
> Aug 08 11:58:16 rgmanager [ip] Link for eth2: Detected
> Aug 08 11:58:16 rgmanager [ip] Adding IPv4 address 192.168.169.173/24 to eth2
> Aug 08 11:58:16 rgmanager [ip] Pinging addr 192.168.169.173 from dev eth2
> Aug 08 11:58:18 rgmanager [ip] Sending gratuitous ARP: 192.168.169.173 c8:1f:66:e8:bb:34 brd ff:ff:ff:ff:ff:ff
> Aug 08 11:58:19 rgmanager [mysql] Verifying Configuration Of mysql:mysql
> Aug 08 11:58:19 rgmanager [mysql] Verifying Configuration Of mysql:mysql > Succeed
> Aug 08 11:58:19 rgmanager [mysql] Monitoring Service mysql:mysql
> Aug 08 11:58:19 rgmanager [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed
> Aug 08 11:58:19 rgmanager [mysql] Monitoring Service mysql:mysql > Service Is Not Running
> Aug 08 11:58:19 rgmanager [mysql] Starting Service mysql:mysql
> Aug 08 11:58:19 rgmanager [mysql] Looking For IP Address > Succeed - IP Address Found
> Aug 08 11:58:20 rgmanager [mysql] Starting Service mysql:mysql > Succeed
> Aug 08 11:58:21 rgmanager [mysql] Monitoring Service mysql:mysql
> Aug 08 11:58:21 rgmanager 1 events processed
> Aug 08 11:58:21 rgmanager [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed
> Aug 08 11:58:21 rgmanager [mysql] Monitoring Service mysql:mysql > Service Is Not Running
> Aug 08 11:58:21 rgmanager start on mysql "mysql" returned 7 (unspecified)
> Aug 08 11:58:21 rgmanager #68: Failed to start service:mysql; return value: 1
> Aug 08 11:58:21 rgmanager Stopping service service:mysql
> Aug 08 11:58:21 rgmanager [mysql] Verifying Configuration Of mysql:mysql
> Aug 08 11:58:21 rgmanager [mysql] Verifying Configuration Of mysql:mysql > Succeed
> Aug 08 11:58:21 rgmanager [mysql] Stopping Service mysql:mysql
> Aug 08 11:58:21 rgmanager [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed - File Doesn't Exist
> Aug 08 11:58:21 rgmanager [mysql] Stopping Service mysql:mysql > Succeed
> --------------------------------------------------------------------------------
>
>
> ------------------------ rg_test output -----------------------------------------
> Running in test mode.
> Link for eth2: Detected
> [ip] Link for eth2: Detected
> Adding IPv4 address 192.168.169.173/24 to eth2
> [ip] Adding IPv4 address 192.168.169.173/24 to eth2
> Pinging addr 192.168.169.173 from dev eth2
> [ip] Pinging addr 192.168.169.173 from dev eth2
> Sending gratuitous ARP: 192.168.169.173 c8:1f:66:e8:bb:70 brd ff:ff:ff:ff:ff:ff
> [ip] Sending gratuitous ARP: 192.168.169.173 c8:1f:66:e8:bb:70 brd ff:ff:ff:ff:ff:ff
> Verifying Configuration Of mysql:mysql
> [mysql] Verifying Configuration Of mysql:mysql
> Verifying Configuration Of mysql:mysql > Succeed
> [mysql] Verifying Configuration Of mysql:mysql > Succeed
> Monitoring Service mysql:mysql
> [mysql] Monitoring Service mysql:mysql
> Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed
> [mysql] Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed
> Monitoring Service mysql:mysql > Service Is Not Running
> [mysql] Monitoring Service mysql:mysql > Service Is Not Running
> Starting Service mysql:mysql
> [mysql] Starting Service mysql:mysql
> Looking For IP Address > Succeed - IP Address Found
> [mysql] Looking For IP Address > Succeed - IP Address Found
> Starting Service mysql:mysql > Succeed
> [mysql] Starting Service mysql:mysql > Succeed
> Monitoring Service mysql:mysql
> [mysql] Monitoring Service mysql:mysql
> Monitoring Service mysql:mysql > Service Is Running
> [mysql] Monitoring Service mysql:mysql > Service Is Running
> Loading resource rule from /usr/share/cluster/checkquorum
> Loading resource rule from /usr/share/cluster/mysql.sh
> Loading resource rule from /usr/share/cluster/ip.sh
> Loading resource rule from /usr/share/cluster/SAPDatabase
> Loading resource rule from /usr/share/cluster/fs.sh
> Loading resource rule from /usr/share/cluster/fence_scsi_check.pl
> Loading resource rule from /usr/share/cluster/script.sh
> Loading resource rule from /usr/share/cluster/fence_scsi_check_hardreboot.pl
> Loading resource rule from /usr/share/cluster/samba.sh
> Loading resource rule from /usr/share/cluster/oradg.sh
> Loading resource rule from /usr/share/cluster/nfsexport.sh
> Loading resource rule from /usr/share/cluster/orainstance.sh
> Loading resource rule from /usr/share/cluster/service.sh
> Loading resource rule from /usr/share/cluster/postgres-8.sh
> Loading resource rule from /usr/share/cluster/named.sh
> Loading resource rule from /usr/share/cluster/bind-mount.sh
> Loading resource rule from /usr/share/cluster/svclib_nfslock
> Loading resource rule from /usr/share/cluster/lvm_by_lv.sh
> Loading resource rule from /usr/share/cluster/ASEHAagent.sh
> Loading resource rule from /usr/share/cluster/netfs.sh
> Loading resource rule from /usr/share/cluster/tomcat-6.sh
> Loading resource rule from /usr/share/cluster/clusterfs.sh
> Loading resource rule from /usr/share/cluster/lvm.sh
> Loading resource rule from /usr/share/cluster/openldap.sh
> Loading resource rule from /usr/share/cluster/vm.sh
> Loading resource rule from /usr/share/cluster/ocf-shellfuncs
> Loading resource rule from /usr/share/cluster/lvm_by_vg.sh
> Loading resource rule from /usr/share/cluster/SAPInstance
> Loading resource rule from /usr/share/cluster/nfsserver.sh
> Loading resource rule from /usr/share/cluster/nfsclient.sh
> Loading resource rule from /usr/share/cluster/apache.sh
> Loading resource rule from /usr/share/cluster/oracledb.sh
> Loading resource rule from /usr/share/cluster/oralistener.sh
> Starting mysql...
> Start of mysql complete
> ---------------------------------------------------------------------------------
>
--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
From jpokorny at redhat.com Thu Aug 11 09:10:49 2016
From: jpokorny at redhat.com (Jan =?utf-8?Q?Pokorn=C3=BD?=)
Date: Thu, 11 Aug 2016 11:10:49 +0200
Subject: [Linux-cluster] [Cluster-devel] git repo of fence-virt
In-Reply-To: <20150323224838.GC22309@redhat.com>
References: <20150209.171912.399832125393796182.yamato@redhat.com>
<20150209131424.GA12489@redhat.com>
<28403ACB-026C-404A-9CCA-8B9CE03A705C@beekhof.net>
<20150223142934.GA766919@redhat.com>
<41143705-C6AB-4A65-9EA2-DD84532D4D10@beekhof.net>
<20150323224838.GC22309@redhat.com>
Message-ID: <20160811091049.GD8666@redhat.com>
On 23/03/15 23:48 +0100, Jan Pokorn? wrote:
> On 24/02/15 07:13 +1100, Andrew Beekhof wrote:
>>> On 24 Feb 2015, at 1:29 am, Ryan McCabe wrote:
>>> On Mon, Feb 23, 2015 at 07:15:42AM +1100, Andrew Beekhof wrote:
>>>> Ryan, since we're trying to consolidate everything else in the
>>>> clusterlabs org area of github, perhaps consider moving it there.
>>>> You'll still have complete control over it.
>>>>
>>>
>>> Sure, works for me. Could you or somebody else who has the access add me
>>> to the clusterlabs org on github?
>>
>> done :)
>
> Maybe worth updating https://fedorahosted.org/cluster/wiki/FenceVirt
> once again, then. Would save a "301 Moved Permanently" redirection
> to https://github.com/ClusterLabs/fence-virt ;)
There, fixed:
https://fedorahosted.org/cluster/wiki/FenceVirt?action=diff&version=7
--
Jan (Poki)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL:
From amjadcsu at gmail.com Mon Aug 29 13:23:33 2016
From: amjadcsu at gmail.com (Amjad Syed)
Date: Mon, 29 Aug 2016 16:23:33 +0300
Subject: [Linux-cluster] Two node failover with shared storage and Luns in
active/passive
Message-ID:
Hello,
We have implemented a two node failover cluster using RHEL pcs in active
passive mode
We have shared storage with one LUN,
Our concern is what if this LUN fails or there is hardware problem
accessing the data in it.
What is the best way to synchronize data between two LUNS ? any suggestion
will help.
Thanks,
Amjad
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
From ricks at alldigital.com Mon Aug 29 17:17:18 2016
From: ricks at alldigital.com (Rick Stevens)
Date: Mon, 29 Aug 2016 10:17:18 -0700
Subject: [Linux-cluster] Two node failover with shared storage and Luns
in active/passive
In-Reply-To:
References:
Message-ID: <08e0c0da-2538-c17b-3196-760e844483a9@alldigital.com>
On 08/29/2016 06:23 AM, Amjad Syed wrote:
> Hello,
> We have implemented a two node failover cluster using RHEL pcs in
> active passive mode
> We have shared storage with one LUN,
> Our concern is what if this LUN fails or there is hardware problem
> accessing the data in it.
> What is the best way to synchronize data between two LUNS ? any
> suggestion will help.
If you have two identical LUNs, you could run something like drbd to
mirror one LUN to a backup:
https://www.drbd.org/en/comp/drbd-linux-driver
Note this is done at the block level. It doesn't look at filesystem-
specific stuff, it just watches block-level disk writes to the primary
device and replicates those to the backup over the network.
There are other things you can do at a filesystem level, such as
rsyncing often, corosync, etc.
----------------------------------------------------------------------
- Rick Stevens, Systems Engineer, AllDigital ricks at alldigital.com -
- AIM/Skype: therps2 ICQ: 226437340 Yahoo: origrps2 -
- -
- Squawk! Pieces of Seven! Pieces of Seven! Parity Error! -
----------------------------------------------------------------------
From debjyoti.mail at gmail.com Mon Aug 29 18:53:32 2016
From: debjyoti.mail at gmail.com (Debjyoti Banerjee)
Date: Tue, 30 Aug 2016 00:23:32 +0530
Subject: [Linux-cluster] Two node failover with shared storage and Luns
in active/passive
In-Reply-To:
References:
Message-ID:
Hi Amjad,
You should try glusterfs.
Regards,
DJ
On Aug 29, 2016 6:59 PM, "Amjad Syed" wrote:
> Hello,
> We have implemented a two node failover cluster using RHEL pcs in active
> passive mode
> We have shared storage with one LUN,
> Our concern is what if this LUN fails or there is hardware problem
> accessing the data in it.
> What is the best way to synchronize data between two LUNS ? any
> suggestion will help.
>
> Thanks,
> Amjad
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: