ext3 performance with hardware RAID5

Simon Oliver simon.oliver at umist.ac.uk
Wed Jul 14 12:56:34 UTC 2004 

I'm setting up a new fileserver.  It has two RAID controllers, a PERC 3/DI
providing mirrored system disks and a PERC 3/DC providing a 1TB RAID5 volume
consisting of eight 144GB U160 drives.  This will serve NFS, Samba and sftp
clients for about 200 users.

The logical drive was created with the following settings:

RAID         = 5
stripe size  = 32kb
write policy = wrback
read policy  = adaptive
cache policy = CachedIO
#stripes     = 8

I intend to revisit the effects of stripe size and read/write/cache policies
at a later stage, but any advice on selecting the appropriate parameters is
greatly appreciated.

I considered other file systems but have chosen ext3 for various reasons not
discussed here.

When creating the file system I used the following options:
  -O sparse_super -O dir_index -R stride=8

-O sparse_super because I don't want to waste too much space
-O dir_index because this is supposed to help file listing performance with
Samba shares -R stride=8 because of the 32kb stripe size (4kb*8=32kb)

The next stage in my plan is to optimise file system performance.  I've
decided to use bonnie++ (local execution) for the initial tests.  I will
then extend the tests to factor in network file system performance (via NFS
and Samba) with multiple simultaneous loads.

I decided to investigate the use of an external log device.  I have read
that it can improve performance for RAID 5 volumes, especially when the log
file is on a separate mirrored volume.  I have also read that using
data=journal can further improve performance for applications such as NFS.

For these tests I created a 256MB partition in the mirrored disk set for the
log device: /dev/sdb6.  The RAID5 volume is a 500GB partition: /dev/sda1.

Here is a summary of the results:

Log location:
- log location does not affect the file system performance as measured by
the sequential output and sequential input tests
- random seek rate improves with an external log (25%)
- log location affects the random create "create" test results and has a
massive impact on the random create "read" and "delete" test results

Data journaling:
- data journaling does not affect sequential input or random seek test
results
- data journaling significantly reduces the performance of the file system
with regard to sequential output, sequential create and random create
(between 25% and 56% decrease).
- data journaling improves file system performance in only one area, the
random create "delete" rate.

Writeback journaling:
- data=writeback has few performance benefits over data=ordered:
 - sequential create "read" (50% faster)
 - random create "delete" (100% faster)
- data=writeback under performed for the random create "create" and "read"
tests

Please provide feedback on these results, especially on the significance of
the random create tests.  

I would also like to hear any suggestions on how I might further improve
file system performance or my test methodology.

The tests...

# internal log
mkfs.ext3 -v -O sparse_super -O dir_index -R stride=8 /dev/sda1 mount -t
ext3 /dev/sda1 /mnt mkdir -m 777 /mnt/t
bonnie++ -u nobody:nobody -d /mnt/t -s 4g -m ss1 -n128:20000:16:512

Version  1.03       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
%CP
ss1              4G 28235  94 38522  28 17222   8 22257  61 40274  10 598.0
1
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
%CP
   128:20000:16/512  1566  17 11216  24  9255  43  1872  20   500   1   101
1
ss1,4G,28235,94,38522,28,17222,8,22257,61,40274,10,598.0,1,128:20000:16/512,
1566,17,11216,24,9255,43,1872,20,500,1,101,1

# external log
mke2fs -v -s 4096 -O journal_dev /dev/sdb6
mkfs.ext3 -v -j -J device=/dev/sdb6 -O sparse_super -O dir_index -R stride=8
/dev/sda1 mount -t ext3 /dev/sda1 /mnt mkdir -m 777 /mnt/t
bonnie++ -u nobody:nobody -d /mnt/t -s 4g -m ss1 -n128:20000:16:512

Version  1.03       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
%CP
ss1              4G 28557  95 38721  28 17354   8 22077  60 40036  10 742.7
1
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
%CP
   128:20000:16/512  1558  17 24247  51 20063  92  2512  28 39603 100  8696
48
ss1,4G,28557,95,38721,28,17354,8,22077,60,40036,10,742.7,1,128:20000:16/512,
1558,17,24247,51,20063,92,2512,28,39603,100,8696,48

# external data log
mke2fs -v -s 4096 -O journal_dev /dev/sdb6
mkfs.ext3 -v -j -J device=/dev/sdb6 -O sparse_super -O dir_index -R stride=8
/dev/sda1 mount -t ext3 -o data=journal /dev/sda1 /mnt mkdir -m 777 /mnt/t
bonnie++ -u nobody:nobody -d /mnt/t -s 4g -m ss1 -n128:20000:16:512

Version  1.03       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
%CP
ss1              4G 12369  47 17023  21 12894  14 22120  61 40720  10 736.0
2
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
%CP
   128:20000:16/512  1128  16  5103  12 20367  95  1144  16   725   2 17172
94
ss1,4G,12369,47,17023,21,12894,14,22120,61,40720,10,736.0,2,128:20000:16/512
,1128,16,5103,12,20367,95,1144,16,725,2,17172,94

# external data log with writeback
mke2fs -v -s 4096 -O journal_dev /dev/sdb6
mkfs.ext3 -v -j -J device=/dev/sdb6 -O sparse_super -O dir_index -R stride=8
/dev/sda1 mount -t ext3 -o data=writeback /dev/sda1 /mnt mkdir -m 777 /mnt/t
bonnie++ -u nobody:nobody -d /mnt/t -s 4g -m ss1 -n128:20000:16:512

Version  1.03       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
%CP
ss1              4G 28205  95 37493  25 17134   7 21320  59 40395  10 756.3
2
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
%CP
   128:20000:16/512  1637  17 36370  76 20284  94  2109  22 27351  70 17057
94
ss1,4G,28205,95,37493,25,17134,7,21320,59,40395,10,756.3,2,128:20000:16/512,
1637,17,36370,76,20284,94,2109,22,27351,70,17057,94

-- 
  Simon Oliver
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ext3.pdf
Type: application/pdf
Size: 16811 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/ext3-users/attachments/20040714/e773bf4a/attachment.pdf>

More information about the Ext3-users mailing list