[Linux-cluster] Course of action if Cluster Manager cannot stop a Percona Mysql application/service

[David John Capstick]

Hi,

I am investigating a problem that occurred some time ago with a two node cluster. It would appear that rgmanager was unable to stop the application (percona mysql) cleanly according to /var/log/messages. After a while it would appear that rgmanager did start the service again. Does this mean that despite the messages it was indeed able to shut the service down first ?

If a service cannot be stopped cleanly I would have thought that rgmanager does not try and start it again - is this view wrong ?

Also the logs show that rgmanager tried to stop the service at 05:06:04 but how do you discover why this action was taken ?

I have included an excerpt of /var/log/messages.

Many Thanks

David



Nov 17 22:43:03 db1 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="2202" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Nov 20 05:06:04 db1 rgmanager[11672]: Stopping service service:mysql-master
Nov 20 05:06:04 db1 rgmanager[14368]: [mysqld] Stopping Service mysqld:mysql-master
Nov 20 05:06:26 db1 rgmanager[14463]: [mysqld] Stopping Service mysqld:mysql-master > Failed - Application Is Still Running
Nov 20 05:06:26 db1 rgmanager[14485]: [mysqld] Stopping Service mysqld:mysql-master > Failed
Nov 20 05:06:26 db1 rgmanager[11672]: stop on mysqld "mysql-master" returned 1 (generic error)
Nov 20 05:06:26 db1 rgmanager[14559]: [fs] unmounting /srv/mysql-master/mnt
Nov 20 05:06:31 db1 rgmanager[14637]: [fs] unmounting /srv/mysql-master/mnt
Nov 20 05:06:37 db1 rgmanager[14713]: [fs] unmounting /srv/mysql-master/mnt
Nov 20 05:06:37 db1 rgmanager[14758]: [fs] 'umount /srv/mysql-master/mnt' failed, error=1
Nov 20 05:06:37 db1 rgmanager[11672]: stop on fs "mysql-master" returned 1 (generic error)
Nov 20 05:06:37 db1 rgmanager[14811]: [ip] Removing IPv4 address 192.168.249.120/24 from eth0
Nov 20 05:06:38 db1 ntpd[8006]: Deleting interface #28 eth0, 192.168.249.120#123, interface stats: received=0, sent=0, dropped=0, active_time=5767950 secs
Nov 20 05:06:47 db1 rgmanager[11672]: #12: RG service:mysql-master failed to stop; intervention required
Nov 20 05:06:47 db1 rgmanager[11672]: Service service:mysql-master is failed
Nov 20 05:07:32 db1 rgmanager[11672]: #43: Service service:mysql-master has failed; can not start.
Nov 20 05:07:32 db1 rgmanager[11672]: #13: Service service:mysql-master failed to stop cleanly
Nov 20 05:09:46 db1 rgmanager[11672]: #43: Service service:mysql-master has failed; can not start.
Nov 20 05:09:46 db1 rgmanager[11672]: #13: Service service:mysql-master failed to stop cleanly
Nov 20 05:10:37 db1 rgmanager[11672]: #43: Service service:mysql-master has failed; can not start.
Nov 20 05:10:37 db1 rgmanager[11672]: #13: Service service:mysql-master failed to stop cleanly
Nov 20 05:11:06 db1 rgmanager[11672]: #43: Service service:mysql-master has failed; can not start.
Nov 20 05:11:06 db1 rgmanager[11672]: #13: Service service:mysql-master failed to stop cleanly
Nov 20 05:16:50 db1 rgmanager[11672]: Starting stopped service service:mysql-master
Nov 20 05:16:50 db1 rgmanager[15291]: [ip] Adding IPv4 address 192.168.249.120/24 to eth0
Nov 20 05:16:53 db1 ntpd[8006]: Listening on interface #29 eth0, 192.168.249.120#123 Enabled
Nov 20 05:16:53 db1 rgmanager[15516]: [mysqld] Checking Existence Of File /var/run/cluster/mysqld/mysqld:mysql-master.pid [mysqld:mysql-master] > Failed
Nov 20 05:16:54 db1 rgmanager[15538]: [mysqld] Monitoring Service mysqld:mysql-master > Service Is Not Running
Nov 20 05:16:54 db1 rgmanager[15560]: [mysqld] Starting Service mysqld:mysql-master
Nov 20 05:16:58 db1 rgmanager[11672]: Service service:mysql-master started
Nov 20 10:42:01 db1 auditd[7280]: Audit daemon rotating log files

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20130121/04b7d121/attachment.htm>