[Linux-cluster] gfs2, kvm setup

[J. Bruce Fields]

On Thu, Jul 10, 2008 at 02:27:14PM +0100, Steven Whitehouse wrote:
> Hi,
> 
> On Wed, 2008-07-09 at 12:32 -0400, J. Bruce Fields wrote:
> > On Wed, Jul 09, 2008 at 04:50:14PM +0100, Christine Caulfield wrote:
> > > J. Bruce Fields wrote:
> > >> On Wed, Jul 09, 2008 at 09:51:02AM +0100, Christine Caulfield wrote:
> > >>> Steven Whitehouse wrote:
> > >>>> Hi,
> > >>>>
> > >>>> On Tue, 2008-07-08 at 18:15 -0400, J. Bruce Fields wrote:
> > >>>>> On Mon, Jul 07, 2008 at 02:49:28PM -0400, bfields wrote:
> > >>>>>> On Mon, Jul 07, 2008 at 10:48:28AM -0500, David Teigland wrote:
> > >>>>>>> On Sun, Jul 06, 2008 at 05:51:05PM -0400, J. Bruce Fields wrote:
> > >>>>>>>> -	write(control_fd, in, sizeof(struct gdlm_plock_info));
> > >>>>>>>> +	write(control_fd, in, sizeof(struct dlm_plock_info));
> > >>>>>>> Gah, sorry, I keep fixing that and it keeps reappearing.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> Jul  1 14:06:42 piglet2 kernel: dlm: connect from non cluster node
> > >>>>>>>> It looks like dlm_new_workspace() is waiting on dlm_recoverd, which is
> > >>>>>>>> in "D" state in dlm_rcom_status(), so I guess the second node isn't
> > >>>>>>>> getting some dlm reply it expects?
> > >>>>>>> dlm inter-node communication is not working here for some reason.  There
> > >>>>>>> must be something unusual with the way the network is configured on the
> > >>>>>>> nodes, and/or a problem with the way the cluster code is applying the
> > >>>>>>> network config to the dlm.
> > >>>>>>>
> > >>>>>>> Ah, I just remembered what this sounds like; we see this kind of thing
> > >>>>>>> when a network interface has multiple IP addresses, and/or routing is
> > >>>>>>> configured strangely.  Others cc'ed could offer better details on exactly
> > >>>>>>> what to look for.
> > >>>>>> OK, thanks!  I'm trying to run gfs2 on 4 kvm machines, I'm an expert on
> > >>>>>> neither, and it's entirely likely there's some obvious misconfiguration.
> > >>>>>> On the kvm host there are 4 virtual interfaces bridged together:
> > >>>>> I ran wireshark on vnet0 while doing the second mount; what I saw was
> > >>>>> the second machine opened a tcp connection to port 21064 on the first
> > >>>>> (which had already completed the mount), and sent it a single message
> > >>>>> identified by wireshark as "DLM3" protocol, type recovery command:
> > >>>>> status command.  It got back an ACK then a RST.
> > >>>>>
> > >>>>> Then the same happened in the other direction, with the first machine
> > >>>>> sending a similar message to port 21064 on the second, which then reset
> > >>>>> the connection.
> > >>>>>
> > >>> That's a symptom of the "connect from non-cluster node" error in the  
> > >>> DLM.
> > >>
> > >> I think I am getting a message to that affect in my logs.
> > >>
> > >>> It's got a connection from an IP address that is not known to cman.   
> > >>> So it closes it as a spoofer
> > >>
> > >> OK.  Is there an easy way to see the list of ip addresses known to cman?
> > >
> > > yes,
> > >
> > >   cman_tool nodes -a
> > >
> > > will show you all the nodes and their known IP addresses
> > 
> > piglet2:~# cman_tool nodes -a
> > Node  Sts   Inc   Joined               Name
> >    1   M    376   2008-07-09 12:30:32  piglet1
> >        Addresses: 192.168.122.129 
> >    2   M    368   2008-07-09 12:30:31  piglet2
> >        Addresses: 192.168.122.130 
> >    3   M    380   2008-07-09 12:30:33  piglet3
> >        Addresses: 192.168.122.131 
> >    4   M    372   2008-07-09 12:30:31  piglet4
> >        Addresses: 192.168.122.132 
> > 
> > These addresses are correct (and are the same addresses that show up in the
> > packet trace).
> > 
> > I must be overlooking something very obvious....
> > 
> > --b.
> > 
> There is something v. odd in the packet trace you sent:
> 
> 16:31:25.513487 00:16:3e:2a:e6:4b (oui Unknown) > 00:16:3e:16:4d:61 (oui
> Unknown
> ), ethertype IPv4 (0x0800), length 74: 192.168.122.130.41170 >
> 192.168.122.129.2
> 1064: S 1424458172:1424458172(0) win 5840 <mss 1460,sackOK,timestamp
> 140931 0,no
> p,wscale 4>
> 
> here we have a packet from .130 (00:16:3e:2a:e6:4b) to .129
> (00:16:3e:16:4d:61) but next we see:
> 
> 16:31:25.513880 00:ff:1d:e9:b9:a3 (oui Unknown) > 00:16:3e:2a:e6:4b (oui
> Unknown
> ), ethertype IPv4 (0x0800), length 74: 192.168.122.129.21064 >
> 192.168.122.130.4
> 1170: S 1340956343:1340956343(0) ack 1424458173 win 5792 <mss
> 1460,sackOK,timest
> amp 140842 140931,nop,wscale 4>
> 
> a packet thats supposedly from .129 except that its mac address is now
> 0:ff:1d:e9:b9:a3. So it looks like the .129 address might be configured
> on two different nodes, either that or there is something odd going on
> with bridging.

Th mystery mac address 00:ff:1d:e9:b9:a3 of both vnet0 and vnet4.  vnet0
is the bridge, which has ip .1 on the host, and which is also the
interface that wireshark is being run on.  The other two addresses are
the mac addresses of the (virtual) ethernet interfaces inside the two
kvm's, with ip's .129 and .130 respectively.  So .130 is sending to the
expected mac address for .129, but responses from .130 are getting the
mac address of vnet0/vnet4.

I'm running wireshark on the host on vnet0.  Just out of curiosity, I
ran it on the host on vnet1 instead, and this time saw the first DLM
connection made from ip .1 and piglet2's mac address.  Erp.  Ok, I'll
experiment some more and look at the /sbin/ip output.

--b.

> If that still doesn't help you solve the problem, can you
> do a:
> 
> /sbin/ip addr list
> /sbin/ip route list
> /sbin/ip neigh list
> 
> on each node and the "host" after an failed attempt so that we can try
> and match up the mac addresses with the interfaces in the trace?
> 
> I don't think that we are too far away from a solution now,