<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
What I don't understand is what changed between RHEL-5 and RHEL-6
that has made HA NFS failover so difficult?<br>
<br>
I have been running a 3-node CentOS-5 cluster serving home
directories (via NFS) to 200+ users for several years now and have
been able to fail over home directories without significant issues.<br>
<br>
With CentOS-5, most of the time I am able to avoid stale filehandle
issues. So far, I've been unwilling to use CentOS-6 for HA NFS as I
can't get failover to work properly.<br>
<br>
What I'm hearing so far on this list is that it's impossible to use
RHEL/CentOS-6 clusters for seamless NFS services. My question is
this, does the problem go away if you disable NFSv4 functionality?
Because my old servers are exporting as NFSv3 and I don't have the
issues you guys are complaining about...<br>
<br>
Here is a sanitized version of my configuration:<br>
<blockquote>
<pre><?xml version="1.0"?>
<cluster alias="ha_nfs" config_version="371" name="ha_nfs">
<fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>
<clusternodes>
<clusternode name="node01.arlut.utexas.edu" nodeid="1" votes="1">
<fence>
<method name="1">
<device name="node01-ilo"/>
</method>
<method name="2">
<device name="fc-switch03" port="4"/>
</method>
</fence>
</clusternode>
<clusternode name="node02.arlut.utexas.edu" nodeid="2" votes="1">
<fence>
<method name="1">
<device name="node02-ilo"/>
</method>
<method name="2">
<device name="fc-switch03" port="0"/>
</method>
</fence>
</clusternode>
<clusternode name="node03.arlut.utexas.edu" nodeid="3" votes="1">
<fence>
<method name="1">
<device name="node03-ilo"/>
</method>
<method name="2">
<device name="fc-switch04" port="0"/>
</method>
</fence>
</clusternode>
</clusternodes>
<cman/>
<fencedevices>
<fencedevice agent="fence_fc-switch01" ipaddr="fc-switch02" login="dummy" name="fc-switch02" passwd="password"/>
<fencedevice agent="fence_fc-switch01" ipaddr="fc-switch03" login="dummy" name="fc-switch03" passwd="password"/>
<fencedevice agent="fence_fc-switch01" ipaddr="fc-switch04" login="dummy" name="fc-switch04" passwd="password"/>
<fencedevice agent="fence_fc-switch01" ipaddr="fc-switch05" login="dummy" name="fc-switch05" passwd="password"/>
<fencedevice agent="fence_ilo" hostname="node01-ilo" login="dummy" name="node01-ilo" passwd="password"/>
<fencedevice agent="fence_ilo" hostname="node02-ilo" login="dummy" name="node02-ilo" passwd="password"/>
<fencedevice agent="fence_ilo" hostname="node03-ilo" login="dummy" name="node03-ilo" passwd="password"/>
</fencedevices>
<rm>
<failoverdomains>
<failoverdomain name="nfs1-domain" nofailback="1" ordered="1" restricted="1">
<failoverdomainnode name="node01.arlut.utexas.edu" priority="1"/>
<failoverdomainnode name="node02.arlut.utexas.edu" priority="2"/>
<failoverdomainnode name="node03.arlut.utexas.edu" priority="3"/>
</failoverdomain>
<failoverdomain name="nfs2-domain" nofailback="1" ordered="1" restricted="1">
<failoverdomainnode name="node02.arlut.utexas.edu" priority="1"/>
<failoverdomainnode name="node03.arlut.utexas.edu" priority="2"/>
<failoverdomainnode name="node01.arlut.utexas.edu" priority="3"/>
</failoverdomain>
<failoverdomain name="nfs3-domain" nofailback="1" ordered="1" restricted="1">
<failoverdomainnode name="node03.arlut.utexas.edu" priority="1"/>
<failoverdomainnode name="node01.arlut.utexas.edu" priority="2"/>
<failoverdomainnode name="node02.arlut.utexas.edu" priority="3"/>
</failoverdomain>
</failoverdomains>
<resources>
<ip address="10.8.3.39" monitor_link="1"/>
<ip address="10.8.3.40" monitor_link="1"/>
<ip address="10.8.3.41" monitor_link="1"/>
<fs device="/dev/cvg/data01" force_fsck="0" force_unmount="1" fsid="34791" fstype="ext3" mountpoint="/lvm/data-1" name="cvg-data01" self_fence="0"/>
<fs device="/dev/cvg/data02" force_fsck="0" force_unmount="1" fsid="64936" fstype="ext3" mountpoint="/lvm/data-2" name="cvg-data02" self_fence="0"/>
<fs device="/dev/cvg/data03" force_fsck="0" force_unmount="1" fsid="22685" fstype="ext3" mountpoint="/lvm/data-3" name="cvg-data03" self_fence="0"/>
<fs device="/dev/cvg/data04" force_fsck="0" force_unmount="1" fsid="4676" fstype="ext3" mountpoint="/lvm/data-4" name="cvg-data04" self_fence="0"/>
<fs device="/dev/cvg/home01" force_fsck="0" force_unmount="1" fsid="38301" fstype="ext3" mountpoint="/lvm/home-1" name="cvg-home01" self_fence="0"/>
<fs device="/dev/cvg/home02" force_fsck="0" force_unmount="1" fsid="50361" fstype="ext3" mountpoint="/lvm/home-2" name="cvg-home02" self_fence="0"/>
<fs device="/dev/cvg/home03" force_fsck="0" force_unmount="1" fsid="62641" fstype="ext3" mountpoint="/lvm/home-3" name="cvg-home03" self_fence="0"/>
<fs device="/dev/cvg/home04" force_fsck="0" force_unmount="1" fsid="24850" fstype="ext3" mountpoint="/lvm/home-4" name="cvg-home04" self_fence="0"/>
<nfsclient allow_recover="1" name="global-nfs-clients" options="rw,insecure" target="10.0.0.0/8"/>
<nfsclient allow_recover="1" name="local-nfs-clients" options="rw,insecure" target="10.8.0.0/16"/>
</resources>
<service autostart="1" domain="nfs1-domain" exclusive="0" name="nfs1-svc" nfslock="1" recovery="relocate">
<ip ref="10.8.3.39">
<fs __independent_subtree="1" ref="cvg-data01">
<nfsexport name="nfs-data01">
<nfsclient ref="local-nfs-clients"/>
</nfsexport>
</fs>
<fs __independent_subtree="1" ref="cvg-home01">
<nfsexport name="nfs-home01">
<nfsclient ref="global-nfs-clients"/>
</nfsexport>
</fs>
<fs __independent_subtree="1" ref="cvg-home02">
<nfsexport name="nfs-home02">
<nfsclient ref="global-nfs-clients"/>
</nfsexport>
</fs>
</ip>
</service>
<service autostart="1" domain="nfs2-domain" exclusive="0" name="nfs2-svc" nfslock="1" recovery="relocate">
<ip ref="10.8.3.40">
<fs __independent_subtree="1" ref="cvg-data02">
<nfsexport name="nfs-data02">
<nfsclient name=" " ref="local-nfs-clients"/>
</nfsexport>
</fs>
<fs __independent_subtree="1" ref="cvg-data03">
<nfsexport name="nfs-data03">
<nfsclient name=" " ref="local-nfs-clients"/>
</nfsexport>
</fs>
<fs __independent_subtree="1" ref="cvg-home03">
<nfsexport name="nfs-home03">
<nfsclient name=" " ref="global-nfs-clients"/>
</nfsexport>
</fs>
</ip>
</service>
<service autostart="1" domain="nfs3-domain" exclusive="0" name="nfs3-svc" nfslock="1" recovery="relocate">
<ip ref="10.8.3.41">
<fs __independent_subtree="1" ref="cvg-data04">
<nfsexport name="nfs-data04">
<nfsclient name=" " ref="local-nfs-clients"/>
</nfsexport>
</fs>
<fs __independent_subtree="1" ref="cvg-home04">
<nfsexport name="nfs-home04">
<nfsclient name=" " ref="global-nfs-clients"/>
</nfsexport>
</fs>
</ip>
</service>
</rm>
</cluster>
</pre>
</blockquote>
On 08/30/2012 11:00 AM,Colin Simpson
<a class="moz-txt-link-rfc2396E" href="mailto:Colin.Simpson@iongeo.com"><Colin.Simpson@iongeo.com></a>
wrote:
<blockquote
cite="mid:mailman.39.1346342403.18903.linux-cluster@redhat.com"
type="cite">
<pre wrap="">Send Linux-cluster mailing list submissions to
<a class="moz-txt-link-abbreviated" href="mailto:linux-cluster@redhat.com">linux-cluster@redhat.com</a>
To subscribe or unsubscribe via the World Wide Web, visit
<a class="moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/linux-cluster">https://www.redhat.com/mailman/listinfo/linux-cluster</a>
or, via email, send a message with subject or body 'help' to
<a class="moz-txt-link-abbreviated" href="mailto:linux-cluster-request@redhat.com">linux-cluster-request@redhat.com</a>
You can reach the person managing the list at
<a class="moz-txt-link-abbreviated" href="mailto:linux-cluster-owner@redhat.com">linux-cluster-owner@redhat.com</a>
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Linux-cluster digest..."
Today's Topics:
1. Re: RHEL/CentOS-6 HA NFS Configuration Question (Colin Simpson)
2. Re: RHEL/CentOS-6 HA NFS Configuration Question
(Fabio M. Di Nitto)
3. Re: Problems with relocation of apache and fence_vmware
(PARAM KRISH)
----------------------------------------------------------------------
Message: 1
Date: Thu, 30 Aug 2012 10:39:03 +0000
From: Colin Simpson <a class="moz-txt-link-rfc2396E" href="mailto:Colin.Simpson@iongeo.com"><Colin.Simpson@iongeo.com></a>
To: "Fabio M. Di Nitto" <a class="moz-txt-link-rfc2396E" href="mailto:fdinitto@redhat.com"><fdinitto@redhat.com></a>
Cc: <a class="moz-txt-link-rfc2396E" href="mailto:linux-cluster@redhat.com">"linux-cluster@redhat.com"</a> <a class="moz-txt-link-rfc2396E" href="mailto:linux-cluster@redhat.com"><linux-cluster@redhat.com></a>
Subject: Re: [Linux-cluster] RHEL/CentOS-6 HA NFS Configuration
Question
Message-ID: <a class="moz-txt-link-rfc2396E" href="mailto:1346323144.10055.12.camel@bhac.iouk.ioroot.tld"><1346323144.10055.12.camel@bhac.iouk.ioroot.tld></a>
Content-Type: text/plain; charset="utf-8"
Did this fix make it as yet?
Thanks
Colin
On Thu, 2012-05-17 at 11:57 +0200, Fabio M. Di Nitto wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Colin,
On 5/17/2012 11:47 AM, Colin Simpson wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Thanks for all the useful information on this.
I realise the bz is not for this issue, I just included it as it has the
suggestion that nfsd should actually live in user space (which seems
sensible).
</pre>
</blockquote>
<pre wrap="">
Understood. I can?t really say if userland or kernel would make any
difference in this specific unmount issue, but for "safety reasons" I
need to assume their design is the same and behave the same way. when/if
there will be a switch, we will need to look more deeply into it. With
current kernel implementation we (cluster guys) need to use this approach.
</pre>
<blockquote type="cite">
<pre wrap="">
Out of interest is there a bz # for this issue?
</pre>
</blockquote>
<pre wrap="">
Yes one for rhel5 and one for rhel6, but they are both private at the
moment because they have customer data in it.
I expect that the workaround/fix (whatever you want to label it) will be
available via RHN in 2/3 weeks.
Fabio
</pre>
<blockquote type="cite">
<pre wrap="">
Colin
On Thu, 2012-05-17 at 10:26 +0200, Fabio M. Di Nitto wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On 05/16/2012 08:19 PM, Colin Simpson wrote:
</pre>
<blockquote type="cite">
<pre wrap="">This is interesting.
We very often see the filesystems fail to umount on busy clustered NFS
servers.
</pre>
</blockquote>
<pre wrap="">
Yes, I am aware the issue since I have been investigating it in details
for the past couple of weeks.
</pre>
<blockquote type="cite">
<pre wrap="">
What is the nature of the "real fix"?
</pre>
</blockquote>
<pre wrap="">
First, the bz you mention below is unrelated to the unmount problem we
are discussing. clustered nfsd locks are a slightly different story.
There are two issues here:
1) cluster users expectations
2) nfsd internal design
(and note I am not blaming either cluster or nfsd here)
Generally cluster users expect to be able to do things like (fake meta
config):
<service1..
<fs1..
<nfsexport1..
<nfsclient1..
<ip1..
....
<service2
<fs2..
<nfsexport2..
<nfsclient2..
<ip2..
and be able to move services around cluster nodes without problem. Note
that it is irrelevant of the fs used. It can be clustered or not.
This setup does unfortunately clash with nfsd design.
When shutdown of a service happens (due to stop or relocation is
indifferent):
ip is removed
exportfs -u .....
(and that's where we hit the nfsd design limitation)
umount fs..
By design (tho I can't say exactly why it is done this way without
speculating), nfsd will continue to serve open sessions via rpc.
exportfs -u will only stop new incoming requests.
If nfsd is serving a client, it will continue to hold a lock on the
filesystem (in kernel) that would prevent the fs to be unmounted.
The only way to effectively close the sessions are:
- drop the VIP and wait for connections timeout (nfsd would effectively
also drop the lock on the fs) but it is slow and not always consistent
on how long it would take
- restart nfsd.
The "real fix" here would be to wait for nfsd containers that do support
exactly this scenario. Allowing unexport of single fs and lock drops
etc. etc. This work is still in very early stages upstream, that doesn't
make it suitable yet for production.
The patch I am working on, is basically a way to handle the clash in the
best way as possible.
A new nfsrestart="" option will be added to both fs and clusterfs, that,
if the filesystem cannot be unmounted, if force_unmount is set, it will
perform an extremely fast restart of nfslock and nfsd.
We can argue that it is not the final solution, i think we can agree
that it is more of a workaround, but:
1) it will allow service migration instead of service failure
2) it will match cluster users expectations (allowing different exports
and live peacefully together).
The only negative impact that we have been able to evaluate so far (the
patch is still under heavy testing phase), beside having to add a config
option to enable it, is that there will be a small window in which all
clients connect to a certain node for all nfs services, will not be
served because nfsd is restarting.
So if you are migrating export1 and there are clients using export2,
export2 will also be affected for those few ms required to restart nfsd.
(assuming export1 and 2 are running on the same node of course).
Placing things in perspective for a cluster, I think that it is a lot
better to be able to unmount a fs and relocate services as necessary vs
a service failing completely and maybe node being fenced.
</pre>
<blockquote type="cite">
<pre wrap="">
I like the idea of NFSD fully being in user space, so killing it would
definitely free the fs.
Alan Brown (who's on this list) recently posted to a RH BZ that he was
one of the people who moved it into kernel space for performance reasons
in the past (that are no longer relevant):
<a class="moz-txt-link-freetext" href="https://bugzilla.redhat.com/show_bug.cgi?id=580863#c9">https://bugzilla.redhat.com/show_bug.cgi?id=580863#c9</a>
, but I doubt this is the fix you have in mind.
</pre>
</blockquote>
<pre wrap="">
No that's a totally different issue.
</pre>
<blockquote type="cite">
<pre wrap="">
Colin
On Tue, 2012-05-15 at 20:21 +0200, Fabio M. Di Nitto wrote:
</pre>
<blockquote type="cite">
<pre wrap="">This solves different issues at startup, relocation and recovery
Also note that there is known limitation in nfsd (both rhel5/6) that
could cause some problems in some conditions in your current
configuration. A permanent fix is being worked on atm.
Without extreme details, you might have 2 of those services running on
the same node and attempting to relocate one of them can fail because
the fs cannot be unmounted. This is due to nfsd holding a lock (at
kernel level) to the FS. Changing config to the suggested one, mask the
problem pretty well, but more testing for a real fix is in progress.
Fabio
--
Linux-cluster mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Linux-cluster@redhat.com">Linux-cluster@redhat.com</a>
<a class="moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/linux-cluster">https://www.redhat.com/mailman/listinfo/linux-cluster</a>
</pre>
</blockquote>
<pre wrap="">
________________________________
This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original.
--
Linux-cluster mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Linux-cluster@redhat.com">Linux-cluster@redhat.com</a>
<a class="moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/linux-cluster">https://www.redhat.com/mailman/listinfo/linux-cluster</a>
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<pre wrap="">
________________________________
This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original.
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<pre wrap="">
________________________________
This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original.
------------------------------
Message: 2
Date: Thu, 30 Aug 2012 12:54:53 +0200
From: "Fabio M. Di Nitto" <a class="moz-txt-link-rfc2396E" href="mailto:fdinitto@redhat.com"><fdinitto@redhat.com></a>
To: Colin Simpson <a class="moz-txt-link-rfc2396E" href="mailto:Colin.Simpson@iongeo.com"><Colin.Simpson@iongeo.com></a>
Cc: <a class="moz-txt-link-rfc2396E" href="mailto:linux-cluster@redhat.com">"linux-cluster@redhat.com"</a> <a class="moz-txt-link-rfc2396E" href="mailto:linux-cluster@redhat.com"><linux-cluster@redhat.com></a>
Subject: Re: [Linux-cluster] RHEL/CentOS-6 HA NFS Configuration
Question
Message-ID: <a class="moz-txt-link-rfc2396E" href="mailto:503F467D.7020108@redhat.com"><503F467D.7020108@redhat.com></a>
Content-Type: text/plain; charset=UTF-8
Hi Colin,
the fix is out for rhel5.8.z in rgmanager-2.0.52-28.el5_8.2 and/or higher.
rhel6.4 fix has been built but not verified by our QA team yet.
Fabio
On 8/30/2012 12:39 PM, Colin Simpson wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Did this fix make it as yet?
Thanks
Colin
On Thu, 2012-05-17 at 11:57 +0200, Fabio M. Di Nitto wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Colin,
On 5/17/2012 11:47 AM, Colin Simpson wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Thanks for all the useful information on this.
I realise the bz is not for this issue, I just included it as it has the
suggestion that nfsd should actually live in user space (which seems
sensible).
</pre>
</blockquote>
<pre wrap="">
Understood. I can?t really say if userland or kernel would make any
difference in this specific unmount issue, but for "safety reasons" I
need to assume their design is the same and behave the same way. when/if
there will be a switch, we will need to look more deeply into it. With
current kernel implementation we (cluster guys) need to use this approach.
</pre>
<blockquote type="cite">
<pre wrap="">
Out of interest is there a bz # for this issue?
</pre>
</blockquote>
<pre wrap="">
Yes one for rhel5 and one for rhel6, but they are both private at the
moment because they have customer data in it.
I expect that the workaround/fix (whatever you want to label it) will be
available via RHN in 2/3 weeks.
Fabio
</pre>
<blockquote type="cite">
<pre wrap="">
Colin
On Thu, 2012-05-17 at 10:26 +0200, Fabio M. Di Nitto wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On 05/16/2012 08:19 PM, Colin Simpson wrote:
</pre>
<blockquote type="cite">
<pre wrap="">This is interesting.
We very often see the filesystems fail to umount on busy clustered NFS
servers.
</pre>
</blockquote>
<pre wrap="">
Yes, I am aware the issue since I have been investigating it in details
for the past couple of weeks.
</pre>
<blockquote type="cite">
<pre wrap="">
What is the nature of the "real fix"?
</pre>
</blockquote>
<pre wrap="">
First, the bz you mention below is unrelated to the unmount problem we
are discussing. clustered nfsd locks are a slightly different story.
There are two issues here:
1) cluster users expectations
2) nfsd internal design
(and note I am not blaming either cluster or nfsd here)
Generally cluster users expect to be able to do things like (fake meta
config):
<service1..
<fs1..
<nfsexport1..
<nfsclient1..
<ip1..
....
<service2
<fs2..
<nfsexport2..
<nfsclient2..
<ip2..
and be able to move services around cluster nodes without problem. Note
that it is irrelevant of the fs used. It can be clustered or not.
This setup does unfortunately clash with nfsd design.
When shutdown of a service happens (due to stop or relocation is
indifferent):
ip is removed
exportfs -u .....
(and that's where we hit the nfsd design limitation)
umount fs..
By design (tho I can't say exactly why it is done this way without
speculating), nfsd will continue to serve open sessions via rpc.
exportfs -u will only stop new incoming requests.
If nfsd is serving a client, it will continue to hold a lock on the
filesystem (in kernel) that would prevent the fs to be unmounted.
The only way to effectively close the sessions are:
- drop the VIP and wait for connections timeout (nfsd would effectively
also drop the lock on the fs) but it is slow and not always consistent
on how long it would take
- restart nfsd.
The "real fix" here would be to wait for nfsd containers that do support
exactly this scenario. Allowing unexport of single fs and lock drops
etc. etc. This work is still in very early stages upstream, that doesn't
make it suitable yet for production.
The patch I am working on, is basically a way to handle the clash in the
best way as possible.
A new nfsrestart="" option will be added to both fs and clusterfs, that,
if the filesystem cannot be unmounted, if force_unmount is set, it will
perform an extremely fast restart of nfslock and nfsd.
We can argue that it is not the final solution, i think we can agree
that it is more of a workaround, but:
1) it will allow service migration instead of service failure
2) it will match cluster users expectations (allowing different exports
and live peacefully together).
The only negative impact that we have been able to evaluate so far (the
patch is still under heavy testing phase), beside having to add a config
option to enable it, is that there will be a small window in which all
clients connect to a certain node for all nfs services, will not be
served because nfsd is restarting.
So if you are migrating export1 and there are clients using export2,
export2 will also be affected for those few ms required to restart nfsd.
(assuming export1 and 2 are running on the same node of course).
Placing things in perspective for a cluster, I think that it is a lot
better to be able to unmount a fs and relocate services as necessary vs
a service failing completely and maybe node being fenced.
</pre>
<blockquote type="cite">
<pre wrap="">
I like the idea of NFSD fully being in user space, so killing it would
definitely free the fs.
Alan Brown (who's on this list) recently posted to a RH BZ that he was
one of the people who moved it into kernel space for performance reasons
in the past (that are no longer relevant):
<a class="moz-txt-link-freetext" href="https://bugzilla.redhat.com/show_bug.cgi?id=580863#c9">https://bugzilla.redhat.com/show_bug.cgi?id=580863#c9</a>
, but I doubt this is the fix you have in mind.
</pre>
</blockquote>
<pre wrap="">
No that's a totally different issue.
</pre>
<blockquote type="cite">
<pre wrap="">
Colin
On Tue, 2012-05-15 at 20:21 +0200, Fabio M. Di Nitto wrote:
</pre>
<blockquote type="cite">
<pre wrap="">This solves different issues at startup, relocation and recovery
Also note that there is known limitation in nfsd (both rhel5/6) that
could cause some problems in some conditions in your current
configuration. A permanent fix is being worked on atm.
Without extreme details, you might have 2 of those services running on
the same node and attempting to relocate one of them can fail because
the fs cannot be unmounted. This is due to nfsd holding a lock (at
kernel level) to the FS. Changing config to the suggested one, mask the
problem pretty well, but more testing for a real fix is in progress.
Fabio
--
Linux-cluster mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Linux-cluster@redhat.com">Linux-cluster@redhat.com</a>
<a class="moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/linux-cluster">https://www.redhat.com/mailman/listinfo/linux-cluster</a>
</pre>
</blockquote>
<pre wrap="">
________________________________
This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original.
--
Linux-cluster mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Linux-cluster@redhat.com">Linux-cluster@redhat.com</a>
<a class="moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/linux-cluster">https://www.redhat.com/mailman/listinfo/linux-cluster</a>
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<pre wrap="">
________________________________
This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original.
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<pre wrap="">
________________________________
This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original.
</pre>
</blockquote>
<pre wrap="">
------------------------------
Message: 3
Date: Thu, 30 Aug 2012 20:58:17 +0530
From: PARAM KRISH <a class="moz-txt-link-rfc2396E" href="mailto:mkparam@gmail.com"><mkparam@gmail.com></a>
To: Digimer <a class="moz-txt-link-rfc2396E" href="mailto:lists@alteeve.ca"><lists@alteeve.ca></a>, <a class="moz-txt-link-abbreviated" href="mailto:linux-cluster@redhat.com">linux-cluster@redhat.com</a>
Subject: Re: [Linux-cluster] Problems with relocation of apache and
fence_vmware
Message-ID:
<a class="moz-txt-link-rfc2396E" href="mailto:CAA1zgja0QfxzTuTJS_6-5tvjkmxAvb=qCp_Poj+cGGG4a+_UuQ@mail.gmail.com"><CAA1zgja0QfxzTuTJS_6-5tvjkmxAvb=qCp_Poj+cGGG4a+_UuQ@mail.gmail.com></a>
Content-Type: text/plain; charset="iso-8859-1"
Never mind. I am good now. I have figured out the syntax for fence_vmware
and it works beautifully now.
Here it is, just in case someone breaks his head to get this done in future
..
...
<clusternodes>
<clusternode name="node1.localdomain" nodeid="1" votes="1">
<fence>
<method name="fence_vmware">
<device name="vmware"
port="node1.localdomain"/>
</method>
</fence>
</clusternode>
<clusternode name="node2.localdomain" nodeid="2" votes="1">
<fence>
<method name="fence_vmware">
<device name="vmware"
port="node2.localdomain"/>
</method>
</fence>
</clusternode>
..
<fencedevices>
<fencedevice agent="fence_vmware" ipaddr="a.b.c.d"
login="xxx" name="vmware" passwd="xxx"/>
</fencedevices>
...
I will be doing some series of fail-over scenarios ( node and service
failures have worked very well so far) and will get back with the results
if there are any concerns. Thanks for helping me thus far. I really
appreciate.
Param
On Thu, Aug 30, 2012 at 1:37 PM, PARAM KRISH <a class="moz-txt-link-rfc2396E" href="mailto:mkparam@gmail.com"><mkparam@gmail.com></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">*Background : *
I am using two VM's hosted in my internal lab that has two interfaces one
configured with a valid IP and other being down. I have kept the VIP also
in the same network. My intention is to have a Apache configured as cluster
service in these two nodes and do a fail-over when the node or the
interface goes down. I try to use fence_vmware as fencing device. These two
VM's are now part of a ESX 4.1 host and the GuestOS in my VM's are RHEL6.0
32-bit.
I am seeing the following problems in my setup now ...
1. When starting a apache service from LUCI, it starts fine in a node.
But, if i kill httpd process from that node manually, it does not detect
the service is down to restart or to relocate
2. -same- case if i do "ip adds del <VIP>" ; it just detects the node is
down but does not do a restart or relocate of the service
3. Whenever i reboot the nodes, it comes online and the service properly
starts fine in either of the node and both nodes perfectly in Quorum but
the fail-over never happens if i stop that active node.
4. I am not sure what format of fence that i must put in the cluster.conf,
since there is no way i can test that out if at all it works fine.
Manual tests :
1. I manually run something like this
"fence_vmware --action=status --ip=10.72.145.145 --username=<login>
--password=<password> --plug=<vm-name>" which works fine on both the nodes.
2. Apache starts/stops just particularly fine from both nodes when i do
"rg_test test /etc/cluster/cluster.conf start service WEB"
Cluster.conf is attached herewith.
rgmanager.log is attached herewith.
Please let me know any specific debug commands that i can run manually to
find out the issues going on here, more particularly the "relocation" of
service and the "fencing"; both consistently fails.
Please help. I have been spending more than 10 days now to set this up in
my internal lab to show it as Proof of Concept to my business heads to buy
RHEL cluster indeed works for our production requirement.
-Param
</pre>
</blockquote>
<pre wrap="">-------------- next part --------------
An HTML attachment was scrubbed...
URL: <a class="moz-txt-link-rfc2396E" href="https://www.redhat.com/archives/linux-cluster/attachments/20120830/2a00c23c/attachment.html"><https://www.redhat.com/archives/linux-cluster/attachments/20120830/2a00c23c/attachment.html></a>
------------------------------
--
Linux-cluster mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Linux-cluster@redhat.com">Linux-cluster@redhat.com</a>
<a class="moz-txt-link-freetext" href="https://www.redhat.com/mailman/listinfo/linux-cluster">https://www.redhat.com/mailman/listinfo/linux-cluster</a>
End of Linux-cluster Digest, Vol 100, Issue 42
**********************************************
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Randy Zagar Sr. Unix Systems Administrator
E-mail: <a class="moz-txt-link-abbreviated" href="mailto:zagar@arlut.utexas.edu">zagar@arlut.utexas.edu</a> Applied Research Laboratories
Phone: 512 835-3131 Univ. of Texas at Austin
</pre>
</body>
</html>