[Linux-cluster] Fence_ilo issue

[Srija]

Hi,
 
I am facing  issues  with the fence_ilo. We have sixteen nodes clusters.  One of the node have
some issue and from other node tried to fence that node  but, could not, the error is:
 
    node1 fenced[12698]: fencing node "node2-priv.xxx.xxx"
    node1 fenced[12698]: agent "fence_ilo" reports: Unable to connect/login to fencing device
    node1 fenced[12698]: fence "node2-priv.xxx.xxx" failed
    node1 openais[12590]: [TOTEM] entering GATHER state from 11.
 
The  fencing node is  out of the cluster now.  but  still from node1 it is trying to fence.
Here is the cluster config file..
 
<clusternodes>
<clusternode name="node1-priv.xxx.xxx" nodeid="1" votes="1">
                 <fence><method name="1">
                 <device name="ilo-node1r" action="reboot"/></method>
                 </fence>
</clusternode>
<clusternode name="node2-priv.xxx.xxx" nodeid="2" votes="1">
        <fence><method name="1">
        <device name="ilo-node2r" action="reboot"/></method>
        </fence>
</clusternode>
----------<snip>------
<fencedevices>
        <fencedevice agent="fence_ilo" hostname="node1r" login="Admin" name="ilo-node1r" passwd="xxxxxxxx"/>
        <fencedevice agent="fence_ilo" hostname="node3r" login="Admin" name="ilo-node3r" passwd="xxxxxxxx"/>
        <fencedevice agent="fence_ilo" hostname="node2r" login="Admin" name="ilo-node2r" passwd="xxxxxxxx"/>
       ------------<snip>------------
</fencedevices>
<rm>
<failoverdomains/>
<resources/>
I tried  to login  ILO of node2  with ssh , using the ILO user/passw  as  Admin/xxxxxxxx, it worked
Here the report:
[root at node1 tmp]# ssh Admin at node2r
Admin at node2r's password: 
User:Admin logged-in to node2R.(xxx.xx.xx.xx)
iLO 2 Standard Blade Edition 2.01 at 11:16:29 Aug 04 2010
Server Name: node2
Server Power: On
</>hpiLO-> 

One more information : The dns entries  of  ILO has no reverse  entry .  only the ip in the dns. 
I have  tested the ilo fencing in the lab  .. and it works fine...

Any advice will be appreciated , where to look, why the fence did not work. 

Thanks again
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20120210/dd7ef74d/attachment.htm>