ext3 with maildir++ = huge disk latency and high load
Eric Sandeen
sandeen at redhat.com
Thu Sep 29 14:44:45 UTC 2011
On 9/29/11 9:36 AM, Andrey wrote:
> Let me to share some testing RAID 5 results with bonnie++:
What kernel version was this tested on?
Thanks,
-Eric
> ext3 (defaults,noatime):
>
> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> debian 2G 242 96 22458 10 8826 2 1854 98 120985 11 317.1 3
> Latency 211ms 896ms 720ms 22258us 18733us 622ms
> Version 1.96 ------Sequential Create------ --------Random Create--------
> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> 16 12857 33 +++++ +++ 15377 34 13585 33 +++++ +++ 15404 35
> Latency 12284us 992us 1029us 432us 140us 76us
>
> ext3 (-T small,defaults,noatime):
>
> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> debian 2G 229 98 4989 5 3862 1 1762 97 91111 9 266.6 6
> Latency 79046us 22858ms 2577ms 19253us 12120us 767ms
> Version 1.96 ------Sequential Create------ --------Random Create--------
> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> 16 6422 16 +++++ +++ 10319 25 8934 21 +++++ +++ 10347 26
> Latency 9968us 977us 964us 482us 144us 178us
>
> ext3 (-T news,defaults,noatime):
> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> debian 2G 237 95 22807 11 8807 2 1897 99 121893 11 324.6 5
> Latency 223ms 808ms 523ms 13765us 11049us 831ms
> Version 1.96 ------Sequential Create------ --------Random Create--------
> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> 16 12826 33 +++++ +++ 15900 35 14548 36 +++++ +++ 15460 35
> Latency 417us 984us 1024us 430us 140us 175us
>
> ext4 (defaults,noatime):
>
> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> debian 2G 256 98 21495 6 9896 2 1771 99 125775 11 349.7 5
> Latency 37738us 992ms 3490ms 10811us 12045us 495ms
> Version 1.96 ------Sequential Create------ --------Random Create--------
> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> 16 14766 43 +++++ +++ 18026 46 16094 46 +++++ +++ 17428 45
> Latency 424us 982us 1023us 367us 139us 174us
>
> xfs(defaults,noatime,logbufs=8,logbsize=131072):
>
> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> debian 2G 476 96 35129 9 12524 3 1417 99 124716 12 445.9 9
> Latency 19798us 420ms 721ms 14122us 9394us 131ms
> Version 1.96 ------Sequential Create------ --------Random Create--------
> debian -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> 16 1552 8 +++++ +++ 1705 11 1675 9 +++++ +++ 1346 8
> Latency 104ms 291us 48604us 109ms 45us 227ms
>
> It seems that latency is big in whole results, best is for XFS. It is tempting me to think that there are some RAID 5 issues here. It's really strange that block writing for SCSI server disks in RAID5 is no more than 30MB/sec(XFS). I guess I should consider XFS file system or different RAID configuration. May be someone can comment this strange benchmark result? Will very appreciate that.
>
> With regards, Andrey.
>
> 23.09.2011 11:31, Janne Pikkarainen пишет:
>> Hello,
>>
>> On 09/23/2011 08:51 AM, Andrey wrote:
>>> Hello,
>>>
>>> I have a production mail server with maildir++ structure and about
>>> 250GB (~10 millions) of files on the ext3 partition on RAID5. It's
>>> mounted with noatime option. These mail server is responsible to local
>>> delivery and storing mail messages.
>>>
>>> System has Debian Squeeze installed and Exim as MDA + Dovecot as
>>> IMAP+POP3 server.
>>>
>>> Bonnie results are terrible. Sequential output for Block and Rewrite
>>> are 10722ms and 9232ms. So if there is a 1000 messages in the mail
>>> queue load is extremely high, delivery time is very big and server can
>>> hang. I did not see such problems with UFS on FreeBSD server.
>>>
>>> As I understand ext3 file system is really bad for such configurations
>>> with Maildir++ (many smaill files)? Is there a way to decrease disk
>>> latency on ext3 or speed up it?
>>>
>>> With regards, Andrey
>>>
>>> ___
>>
>
> _______________________________________________
> Ext3-users mailing list
> Ext3-users at redhat.com
> https://www.redhat.com/mailman/listinfo/ext3-users
More information about the Ext3-users
mailing list