[Linux-cluster] Advice on suitability

[Radu Rendec]

On Fri, 2010-11-19 at 11:24 +0000, Ashley Large wrote:
> My thoughts have been to have 3 “front end” physical host machines
> with a rough 50/50 split of the current 25 odd servers into VMs on
> these physical hosts. I will then have redundancy across 3 machines so
> that if machine 1 dies, the vms are moved to machine 2 or 3 pending
> fix of machine 1. Then I want to create two “back end” storage servers
> which are replicated and provide redundant storage to the front ends
> in a primary/secondary way such that the front ends can failover to
> either available storage server. I am anticipating having only VM host
> booting off machine1-3 and using the back end storage servers for
> booting the vms and storing their data. Each storage server will have
> 6 x 2Tb sata disks set up as a 6Tb raid1 array although I have been
> reading that more spindles is better so maybe I need more disks for
> optimal performance. Front ends and back ends would have multiple
> bonded Gig e ports to a dedicated switch which I think should provide
> enough bandwidth and the net facing NICs running off a separate
> switch.

I've built a very similar system on top of Centos 5 / RHCS. I used drbd
on top of LVM to replicate the data across the two storage machines, and
then iscsi to export the storage to the VM nodes. I used xen for
virtualization.

While my system seemed to be pretty much ok, I experienced random
crashes on virtual machines that (unfortunately) were windows. The crash
however resided not in the virtual machine itself, but in the
virtualization system (segfault in qemu-dm).

The crashes seem to be related to moving storage resources across the
two nodes, but however they cannot be 100% replicated. I also
experienced file corruption inside the VMs. I'm pretty sure that under
certain circumstances some block writes are lost, but for now I didn't
manage to figure out exactly where.

Best regards,

Radu Rendec