[Linux-cluster] GFS2 performance on large files / mkfs tuning

[Gordan Bobic]

On Thu, 23 Apr 2009 13:16:28 +0300, "Pasi Kärkkäinen" <pasik at iki.fi>
wrote:


>> > Yes, I have (by the way, do you know how long ext3 takes to create a
>> > 6TB filesystem???).
>> 
>> It depends on what you consider to be "long". The last box I built had
>> 8x 1TB 7200rpm disks in software md RAID6 = 6 TB usable, DAS, consumer
>> grade motherboard, and 2 of the 8 ports were massively bottlenecked by
>> a 32-bit 33MHz PCI SATA controller, but all 8 ports had NCQ. This took
>> about 10-20 minutes (I haven't timed it exactly, about a cup of coffee
>> long ;)) to mkfs ext3 when the parameters were properly set up. With
>> default settings, it was taking around 10x longer, maybe even more.
>> 
> 
> Interesting information.. what settings did you tweak for faster mkfs?
Just
> the things mentioned below?

Yes, all of the ones mentioned.

>> My findings are that the default settings and old wisdom often taken
>> as axiomatically true are actually completely off the mark.
>> 
>> Here is the line of reasoning that I have found to lead to best results.
>> 
>> RAID block size of 64-256KB is way, way too big. It will kill
>> the performance of small IOs without yielding a noticeable increase
>> in performance for large IOs, and sometimes in fact hurting large
>> IOs, too.
>> 
>> To pick the optimum RAID block size, look at the disks. What is the
>> multi-sector transfer size they can handle? I have not seen any disks
>> to date that have this figure at anything other than 16, and
>> 16sectors * 512 bytes/sector = 8KB.
>> 
> 
> Hmm.. how can you determine multi-sector transfer size from a disk? 

hdparm -i /dev/[hs]d[a-z]

Gordan