ext3 with maildir++ = huge disk latency and high load
Andrey
basketboy at bk.ru
Thu Sep 29 15:13:48 UTC 2011
Hello,
This is standard Debian Squeeze(6.0.2) kernel:
# uname -a
Linux debian 2.6.32-5-686 #1 SMP Fri Sep 9 20:51:05 UTC 2011 i686 GNU/Linux
29.09.2011 18:44, Eric Sandeen пишет:
> On 9/29/11 9:36 AM, Andrey wrote:
>> Let me to share some testing RAID 5 results with bonnie++:
>
> What kernel version was this tested on?
>
> Thanks,
> -Eric
>
>> ext3 (defaults,noatime):
>>
>> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
>> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
>> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
>> debian 2G 242 96 22458 10 8826 2 1854 98 120985 11 317.1 3
>> Latency 211ms 896ms 720ms 22258us 18733us 622ms
>> Version 1.96 ------Sequential Create------ --------Random Create--------
>> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
>> 16 12857 33 +++++ +++ 15377 34 13585 33 +++++ +++ 15404 35
>> Latency 12284us 992us 1029us 432us 140us 76us
>>
>> ext3 (-T small,defaults,noatime):
>>
>> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
>> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
>> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
>> debian 2G 229 98 4989 5 3862 1 1762 97 91111 9 266.6 6
>> Latency 79046us 22858ms 2577ms 19253us 12120us 767ms
>> Version 1.96 ------Sequential Create------ --------Random Create--------
>> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
>> 16 6422 16 +++++ +++ 10319 25 8934 21 +++++ +++ 10347 26
>> Latency 9968us 977us 964us 482us 144us 178us
>>
>> ext3 (-T news,defaults,noatime):
>> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
>> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
>> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
>> debian 2G 237 95 22807 11 8807 2 1897 99 121893 11 324.6 5
>> Latency 223ms 808ms 523ms 13765us 11049us 831ms
>> Version 1.96 ------Sequential Create------ --------Random Create--------
>> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
>> 16 12826 33 +++++ +++ 15900 35 14548 36 +++++ +++ 15460 35
>> Latency 417us 984us 1024us 430us 140us 175us
>>
>> ext4 (defaults,noatime):
>>
>> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
>> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
>> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
>> debian 2G 256 98 21495 6 9896 2 1771 99 125775 11 349.7 5
>> Latency 37738us 992ms 3490ms 10811us 12045us 495ms
>> Version 1.96 ------Sequential Create------ --------Random Create--------
>> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
>> 16 14766 43 +++++ +++ 18026 46 16094 46 +++++ +++ 17428 45
>> Latency 424us 982us 1023us 367us 139us 174us
>>
>> xfs(defaults,noatime,logbufs=8,logbsize=131072):
>>
>> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
>> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
>> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
>> debian 2G 476 96 35129 9 12524 3 1417 99 124716 12 445.9 9
>> Latency 19798us 420ms 721ms 14122us 9394us 131ms
>> Version 1.96 ------Sequential Create------ --------Random Create--------
>> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
>> 16 1552 8 +++++ +++ 1705 11 1675 9 +++++ +++ 1346 8
>> Latency 104ms 291us 48604us 109ms 45us 227ms
>>
>> It seems that latency is big in whole results, best is for XFS. It is tempting me to think that there are some RAID 5 issues here. It's really strange that block writing for SCSI server disks in RAID5 is no more than 30MB/sec(XFS). I guess I should consider XFS file system or different RAID configuration. May be someone can comment this strange benchmark result? Will very appreciate that.
>>
>> With regards, Andrey.
>>
>> 23.09.2011 11:31, Janne Pikkarainen пишет:
>>> Hello,
>>>
>>> On 09/23/2011 08:51 AM, Andrey wrote:
>>>> Hello,
>>>>
>>>> I have a production mail server with maildir++ structure and about
>>>> 250GB (~10 millions) of files on the ext3 partition on RAID5. It's
>>>> mounted with noatime option. These mail server is responsible to local
>>>> delivery and storing mail messages.
>>>>
>>>> System has Debian Squeeze installed and Exim as MDA + Dovecot as
>>>> IMAP+POP3 server.
>>>>
>>>> Bonnie results are terrible. Sequential output for Block and Rewrite
>>>> are 10722ms and 9232ms. So if there is a 1000 messages in the mail
>>>> queue load is extremely high, delivery time is very big and server can
>>>> hang. I did not see such problems with UFS on FreeBSD server.
>>>>
>>>> As I understand ext3 file system is really bad for such configurations
>>>> with Maildir++ (many smaill files)? Is there a way to decrease disk
>>>> latency on ext3 or speed up it?
>>>>
>>>> With regards, Andrey
>>>>
>>>> ___
>>>
>>
>> _______________________________________________
>> Ext3-users mailing list
>> Ext3-users at redhat.com
>> https://www.redhat.com/mailman/listinfo/ext3-users
>
>
More information about the Ext3-users
mailing list