[Fedora-xen] Xen - large virtual machines

[Alex Davies]

Hi Everyone,
I'm looking for a solution to run some Xen VM's with fairly significant(up
to ~30TB) local storage volumes. We plan to use two Sun "Thumper" x4540's
with 48*1TB drives (8 controllers).

I have a few questions, and would appreciate feedback before I try to
achieve something dumb or impossible :)

   - Has Xen been tested / does Xen support very large volumes? We will
   probably end up using ext3 (as it seems its the only supported option), so
   no single volume in a VM will exceed ~16TB, but we will be at this end of
   the spectrum on several volumes.
   - Can anyone suggest a better option than creating 7 RAID5 arrays of 6
   disks each (1 per controller); using the remaining 6 drives for RAID-10 for
   DomU and hot spares and then adding the 7 RAID5 volumes to a LVM group,
   creating a LV per Xen VM block device? Has anybody used mdadm and/or LVM to
   handle this number of drives?
   - We are required to be able to bring up VMs on one machine on another
   within a reasonable period of time (this is defined as days, not hours) and
   to keep the backup copy reasonably in sync (again, days not hours). Can
   anyone comment on the suitability of rsync to copy such massive block
   devices? We would use version 3, which I know has improved performance for
   this sort of move, but is it going to literally take forever to checksum
   48TB? If anyone is able to give example of rsync performance (same switch or
   crossover - whichever we conclude is fastest) over gigabit Ethernet, that
   would be a bonus. We expect there to be significant spare resources
   available to domU at night for checksumming if required.
   - I know XFS is not supported by RHEL, but does it work - and/or well?
   I've seen several reports that suggest it performs far better (best summary
   http://hepix.caspur.it/storage/hep_pdf/2007/Spring/Petkus_HEPiX_Spring06.storageeval.pdf)
    but what is it like for reliability?
   - Does anyone has any suggestions / improvements to this plan? Has
   anybody done this before - with similar hardware?
   - [An alternative scenario, if we can get it, would be a third Thumper -
   in which case we would run half the VMs on each of two Thumpers running
   RAID-10 (8 RAID-1 volumes, each a PV) for ~24TB per machine, and use the
   third Thumper, partitioned as above with RAID-5 for more like RAID-5 levels
   of storage efficiency, for backups of VMs and disaster recovery. Comments on
   this would be fantastic too.]

The VMs are expected to be quite IO-heavy, hence the requirement for local
storage (not iSCSI) and para virtualization.

Any comments much appreciated.

Many thanks,

Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/fedora-xen/attachments/20080905/ed1b24d8/attachment.htm>