ext3 with maildir++ = huge disk latency and high load

Eric Sandeen sandeen at redhat.com
Thu Sep 29 14:44:45 UTC 2011


On 9/29/11 9:36 AM, Andrey wrote:
> Let me to share some testing RAID 5 results with bonnie++:

What kernel version was this tested on?

Thanks,
-Eric

> ext3 (defaults,noatime):
> 
> Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
> Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> debian           2G   242  96 22458  10  8826   2  1854  98 120985  11 317.1   3
> Latency               211ms     896ms     720ms   22258us   18733us 622ms
> Version  1.96       ------Sequential Create------ --------Random Create--------
> debian              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
>                  16 12857  33 +++++ +++ 15377  34 13585  33 +++++ +++ 15404  35
> Latency             12284us     992us    1029us     432us     140us  76us
> 
> ext3 (-T small,defaults,noatime):
> 
> Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
> Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> debian           2G   229  98  4989   5  3862   1  1762  97 91111   9 266.6   6
> Latency             79046us   22858ms    2577ms   19253us   12120us 767ms
> Version  1.96       ------Sequential Create------ --------Random Create--------
> debian              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
>                  16  6422  16 +++++ +++ 10319  25  8934  21 +++++ +++ 10347  26
> Latency              9968us     977us     964us     482us     144us 178us
> 
> ext3 (-T news,defaults,noatime):
> Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
> Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> debian           2G   237  95 22807  11  8807   2  1897  99 121893  11 324.6   5
> Latency               223ms     808ms     523ms   13765us   11049us 831ms
> Version  1.96       ------Sequential Create------ --------Random Create--------
> debian              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
>                  16 12826  33 +++++ +++ 15900  35 14548  36 +++++ +++ 15460  35
> Latency               417us     984us    1024us     430us     140us 175us
> 
> ext4 (defaults,noatime):
> 
> Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
> Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> debian           2G   256  98 21495   6  9896   2  1771  99 125775  11 349.7   5
> Latency             37738us     992ms    3490ms   10811us   12045us 495ms
> Version  1.96       ------Sequential Create------ --------Random Create--------
> debian              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
>                  16 14766  43 +++++ +++ 18026  46 16094  46 +++++ +++ 17428  45
> Latency               424us     982us    1023us     367us     139us 174us
> 
> xfs(defaults,noatime,logbufs=8,logbsize=131072):
> 
> Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
> Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
> debian           2G   476  96 35129   9 12524   3  1417  99 124716  12 445.9   9
> Latency             19798us     420ms     721ms   14122us    9394us 131ms
> Version  1.96       ------Sequential Create------ --------Random Create--------
> debian              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
>                  16  1552   8 +++++ +++  1705  11  1675   9 +++++ +++ 1346   8
> Latency               104ms     291us   48604us     109ms      45us 227ms
> 
> It seems that latency is big in whole results, best is for XFS. It is tempting me to think that there are some RAID 5 issues here. It's really strange that block writing for SCSI server disks in RAID5 is no more than 30MB/sec(XFS). I guess I should consider XFS file system or different RAID configuration. May be someone can comment this strange benchmark result? Will very appreciate that.
> 
> With regards, Andrey.
> 
> 23.09.2011 11:31, Janne Pikkarainen пишет:
>> Hello,
>>
>> On 09/23/2011 08:51 AM, Andrey wrote:
>>> Hello,
>>>
>>> I have a production mail server with maildir++ structure and about
>>> 250GB (~10 millions) of files on the ext3 partition on RAID5. It's
>>> mounted with noatime option. These mail server is responsible to local
>>> delivery and storing mail messages.
>>>
>>> System has Debian Squeeze installed and Exim as MDA + Dovecot as
>>> IMAP+POP3 server.
>>>
>>> Bonnie results are terrible. Sequential output for Block and Rewrite
>>> are 10722ms and 9232ms. So if there is a 1000 messages in the mail
>>> queue load is extremely high, delivery time is very big and server can
>>> hang. I did not see such problems with UFS on FreeBSD server.
>>>
>>> As I understand ext3 file system is really bad for such configurations
>>> with Maildir++ (many smaill files)? Is there a way to decrease disk
>>> latency on ext3 or speed up it?
>>>
>>> With regards, Andrey
>>>
>>> ___
>>
> 
> _______________________________________________
> Ext3-users mailing list
> Ext3-users at redhat.com
> https://www.redhat.com/mailman/listinfo/ext3-users




More information about the Ext3-users mailing list