[Linux-cluster] gfs2, kvm setup

[J. Bruce Fields]

On Fri, Jul 11, 2008 at 11:33:08PM -0400, bfields wrote:
> On Fri, Jul 11, 2008 at 07:25:29PM -0400, bfields wrote:
> > On Thu, Jul 10, 2008 at 02:27:14PM +0100, Steven Whitehouse wrote:
> > > a packet thats supposedly from .129 except that its mac address is now
> > > 0:ff:1d:e9:b9:a3. So it looks like the .129 address might be configured
> > > on two different nodes, either that or there is something odd going on
> > > with bridging.
> > 
> > Th mystery mac address 00:ff:1d:e9:b9:a3 of both vnet0 and vnet4.  vnet0
> > is the bridge, which has ip .1 on the host, and which is also the
> > interface that wireshark is being run on.  The other two addresses are
> > the mac addresses of the (virtual) ethernet interfaces inside the two
> > kvm's, with ip's .129 and .130 respectively.  So .130 is sending to the
> > expected mac address for .129, but responses from .130 are getting the
> > mac address of vnet0/vnet4.
> > 
> > I'm running wireshark on the host on vnet0.  Just out of curiosity, I
> > ran it on the host on vnet1 instead, and this time saw the first DLM
> > connection made from ip .1 and piglet2's mac address.  Erp.  Ok, I'll
> > experiment some more and look at the /sbin/ip output.
> 
> Bah, yes, I clearly got the network configuration completely screwed up
> at some point--it must be trying to do some kind of NAT, though that
> clearly makes no sense.  I'll get this untangled and then I think it
> should be OK....

Problem found.  So the network configuration that libvirt sets up has 4
interfaces (one for each of the 4 kvm guests) all bridged together on
the host, with NAT setup to give the guests access to the outside world.
That looks like this:

root at pig:~# iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  192.168.122.0/24     0.0.0.0/0           

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

OK, fine, except that packets exchanged between the hosts on the bridge
also seem to be going through that POSTROUTING chain, so tcp
connectsions between the guests work--sort of--but they're all getting
NAT'd so they appear to come from 192.168.122.1, so the dlm complains
about connection from a non cluster host".

So my gfs2 mount finally succeeds after:

root at pig:~# iptables -t nat -I POSTROUTING -s 192.168.122.0/24 -d 192.168.122.0/24 -j ACCEPT

ptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  192.168.122.0/24     192.168.122.0/24    
MASQUERADE  all  --  192.168.122.0/24     0.0.0.0/0           

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

I don't know if that's the right fix.  In any case, the original
behavior certainly looks to me like a bug in libvirt.

Thanks for your patience!  I should have caught that much sooner....

--b.