<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head> <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type"> <title></title> </head> <body bgcolor="#ffffff" text="#000000"> [Not sure if this is the right forum - apologies in advance if it's not...] I'm running some LVM/DM scaling performance runs on a 4-way IA64 box (HP RX2600), that is equipped with four dual-U320 SCSI host bus adapters. Each bus has 7 U320 disk drives (15K) attached for a total of 56 drives. [Each dual-U320 HBA is capable of doing on the order of 512MB/sec sustained throughput, I've measured the system using direct IO (no LVM) at being able to read data at up to 1.8GB/sec.] The system has RHEL4 Update 1 Beta bits installed (2.6.9 based kernel). The test program is the SPEW benchmark, and was run with the DIRECT IO flag set. There were ten (10) LVM volumes used, they were all constructed with a 16KB stripe size. The volumes were - Volume 1 - 1 disk Volume 2 - 2 disks, each disk on a separate U320 bus Volume 3 - 3 disks, each disk on a separate U320 bus ... Volume 8 - 8 disks, each disk on a separate U320 bus Volume 9 - 9 disks, 1 disk on 7 buses, 2 disks each on 1 U320 bus Volume 10 - 10 disks, 1 disk on 6 buses, 2 disks each on 2 U320 buses I then ran 4 groups of tests, with each group consisting of a separate run against each of the 10 volumes. The groups had a block size associated with it based upon the number of disks in a volume. Thus the first group of runs had a block size of 16KB times the number of disks in the volume. [The idea being to have a single transfer split into one 16KB against each of the disks in the volume.] The second group had a block size of 32KB times the number of disks in the volume. [The idea here was to see the effects of the wrapping - for example, for Volume 2 with 2 disks (striped at 16KB) we would expect each 64KB initial transfer to be split into four 16KB-transfers, with the first and third delivered to the first drive in the volume, and the second and fourth transfer going to the second drive in the volume.] The third group had a block size of 48KB times the number of disks, and the fourth had a block size of 64KB times the number of disks in the volume. The expectation was to see that the overall performance would be at least as good or hopefully even better as the overall transfer size increased. [If for no other reasons except that (a) the amount of time spent doing IO transfers at the device level should dwarf any overhead at the application/LVM(DM) layers, and (b) the targetted devices could be given multiple commands and thus reduce idle time.] However, that is *not* what was seen. The accompanying xmgrace-generated chart illustrates this (I've also attached the xmgrace file itself, in case you are having problems with the .png file included...): <img alt="" src="cid:part1.09000705.09040005@hp.com" height="612" width="792"> Here we see the black line showing scaling along the first group (16KB per segmenet/disk per transfer) - in fact we see good scaling up to 4 disks (pretty linear), with a slight hiccup, and then it continues onwards at less of an increase. Something strange at 5 disks happens, and again at 9/10 disks - the latter perhaps due to the fact that we're wrapping on the disks now (some adapters have two disks busy at a time). I'm going to look into the actual bus/target mappings for the volumes to see if they play a role. But note, when we go to group 2 (32KB per segment per transfer), the performance is always slightly lower up to 3 disks, and then starting at 4 disks we see almost a flattening out (with a big dip at volume 6). We do see some (slight) scaling again from 32KB to 48KB and then 64KB, but we never again attain the rate of simply issuing the 16KB per segment initial requests. It's interesting to see the dip at 6-disks per volume for all the transfer sizes that wrap. I've been investigating this, but was wondering if anyone had noticed something like this - or perhaps had a ready explanation for this... Alan D. Brunelle HP </body> </html>