[Linux-cluster] virtualization on top of Red Hat cluster

[Gunther Schlegel]

Geoffrey wrote:

> Before going into the actual issues we are running into, I thought we 
> should first find out if there are others who are doing this, or 
> attempting to do this.
> 
> This includes a separate virtual server for the following services: 
> squid, ldap, mail server, dns, samba, email and a handful of 
> Xservers/application servers.
> 
> The specific hardware includes:
> 
> 8 Dell 19150 nodes w/ rh5.2 xen
> Each node has 32GB of ram and 8 cores (xeon 2.66)

you need a 64-bit distro to have xen support more than 16GB RAM.

> EMC San CX-310
> 2 Brocade 5000 fibre switches
> 
> Any feedback from anyone attempting or currently running a similar 
> solution (virtualization on top of cluster) would be greatly appreciated.

We are running something like that for more than a year now (tests on 
rhel5.0, live on rhel5.1, now running rhel5.2).

45+ paravirtualized VMs on 2 clusters.
VMs are located on clustered logical volumes, not gfs.
gfs is used for /etc/xen, though.

performance is good, even with io-intensive apps inside the VMs.
stability is fair (but improving over time).

The main issue has always been the cluster loosing quorum without 
apparent reason. Improved after we added a Quorum Disk. Improved further 
after RH Support recommended a couple of not so well documented ;) 
configuration parameters, which also cannot be maintained with 
conga/luci/ricci or system-config-cluster. Improved even further after 
we complained again and RH came up with even more undocumented setting 
to solve the race-conditions we experienced when a node left the cluster 
(even intentionally).

Another big issue was live migration, turned out that the bridge in Dom0 
  has a default forward delay of 15 seconds. We may have experienced 
that because we use a different xen network-script than delivered by RH. 
Though, the original one could not deal with neither bonding nor Vlans, 
so we had to do that. This has changed according to the rhel5.3 
changelog, but we have not tested it yet.

We also had issues with the DomU time jumping around after live 
migration, but RH fixed that recently and as a workaround one could run 
ntpd inside the DomU.


best regards, Gunther

-- 
Gunther Schlegel
Manager IT Infrastructure


.............................................................
Riege Software International GmbH  Fon: +49 (2159) 9148 0
Mollsfeld 10                       Fax: +49 (2159) 9148 11
40670 Meerbusch                    Web: www.riege.com
Germany                            E-Mail: schlegel at riege.com
---                                ---
Handelsregister:                   Managing Directors:
Amtsgericht Neuss HRB-NR 4207      Christian Riege
USt-ID-Nr.: DE120585842            Gabriele  Riege
                                   Johannes  Riege
.............................................................
           YOU CARE FOR FREIGHT, WE CARE FOR YOU          



-------------- next part --------------
A non-text attachment was scrubbed...
Name: schlegel.vcf
Type: text/x-vcard
Size: 346 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20090123/60f29a28/attachment.vcf>