[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: ext3 with maildir++ = huge disk latency and high load



Hello,

This is standard Debian Squeeze(6.0.2) kernel:

# uname -a
Linux debian 2.6.32-5-686 #1 SMP Fri Sep 9 20:51:05 UTC 2011 i686 GNU/Linux

29.09.2011 18:44, Eric Sandeen пишет:
On 9/29/11 9:36 AM, Andrey wrote:
Let me to share some testing RAID 5 results with bonnie++:

What kernel version was this tested on?

Thanks,
-Eric

ext3 (defaults,noatime):

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
debian           2G   242  96 22458  10  8826   2  1854  98 120985  11 317.1   3
Latency               211ms     896ms     720ms   22258us   18733us 622ms
Version  1.96       ------Sequential Create------ --------Random Create--------
debian              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
                  16 12857  33 +++++ +++ 15377  34 13585  33 +++++ +++ 15404  35
Latency             12284us     992us    1029us     432us     140us  76us

ext3 (-T small,defaults,noatime):

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
debian           2G   229  98  4989   5  3862   1  1762  97 91111   9 266.6   6
Latency             79046us   22858ms    2577ms   19253us   12120us 767ms
Version  1.96       ------Sequential Create------ --------Random Create--------
debian              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
                  16  6422  16 +++++ +++ 10319  25  8934  21 +++++ +++ 10347  26
Latency              9968us     977us     964us     482us     144us 178us

ext3 (-T news,defaults,noatime):
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
debian           2G   237  95 22807  11  8807   2  1897  99 121893  11 324.6   5
Latency               223ms     808ms     523ms   13765us   11049us 831ms
Version  1.96       ------Sequential Create------ --------Random Create--------
debian              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
                  16 12826  33 +++++ +++ 15900  35 14548  36 +++++ +++ 15460  35
Latency               417us     984us    1024us     430us     140us 175us

ext4 (defaults,noatime):

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
debian           2G   256  98 21495   6  9896   2  1771  99 125775  11 349.7   5
Latency             37738us     992ms    3490ms   10811us   12045us 495ms
Version  1.96       ------Sequential Create------ --------Random Create--------
debian              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
                  16 14766  43 +++++ +++ 18026  46 16094  46 +++++ +++ 17428  45
Latency               424us     982us    1023us     367us     139us 174us

xfs(defaults,noatime,logbufs=8,logbsize=131072):

Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
debian           2G   476  96 35129   9 12524   3  1417  99 124716  12 445.9   9
Latency             19798us     420ms     721ms   14122us    9394us 131ms
Version  1.96       ------Sequential Create------ --------Random Create--------
debian              -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
                  16  1552   8 +++++ +++  1705  11  1675   9 +++++ +++ 1346   8
Latency               104ms     291us   48604us     109ms      45us 227ms

It seems that latency is big in whole results, best is for XFS. It is tempting me to think that there are some RAID 5 issues here. It's really strange that block writing for SCSI server disks in RAID5 is no more than 30MB/sec(XFS). I guess I should consider XFS file system or different RAID configuration. May be someone can comment this strange benchmark result? Will very appreciate that.

With regards, Andrey.

23.09.2011 11:31, Janne Pikkarainen пишет:
Hello,

On 09/23/2011 08:51 AM, Andrey wrote:
Hello,

I have a production mail server with maildir++ structure and about
250GB (~10 millions) of files on the ext3 partition on RAID5. It's
mounted with noatime option. These mail server is responsible to local
delivery and storing mail messages.

System has Debian Squeeze installed and Exim as MDA + Dovecot as
IMAP+POP3 server.

Bonnie results are terrible. Sequential output for Block and Rewrite
are 10722ms and 9232ms. So if there is a 1000 messages in the mail
queue load is extremely high, delivery time is very big and server can
hang. I did not see such problems with UFS on FreeBSD server.

As I understand ext3 file system is really bad for such configurations
with Maildir++ (many smaill files)? Is there a way to decrease disk
latency on ext3 or speed up it?

With regards, Andrey

___


_______________________________________________
Ext3-users mailing list
Ext3-users redhat com
https://www.redhat.com/mailman/listinfo/ext3-users




[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]