Limited write bandwidth from ext3

[Sinha_Himanshu at emc.com]

We tried the extents+mballoc+delalloc patches suggested by Andreas and found
that it made a significant improvement in our benchmark - write bandwidth
increased from 144 MBps to 214 MBps. We are at about 85% of the bandwidth
that one can get writing to an  ext2 file which in turn is about 82% of the
bandwidth one can get writing to the block device. We are analyzing our
traces to determine the cause of these differences. So far, we see that
during writes to the ext3 file lun writes periodically wait for 5 reads
while in the case of writes to ext2 file lun writes periodically wait for
only one read.

Workload: Single threaded 512 KB writes to a new file.
				RedHat 4 U1			2.6.16.8
kernel
			(2.6.9 based kernel)		
Block Device		308 MBps			306 MBps
Ext2 file			267				255
Ext3				138				144
Ext3 with patches		N/A				216 
Ext3 with patches, journal on separate LUN	215

Himanshu


-----Original Message-----
From: Andreas Dilger [mailto:adilger at clusterfs.com] 
Sent: Wednesday, June 21, 2006 4:54 PM
To: Sinha, Himanshu
Cc: ext3-users at redhat.com
Subject: Re: Limited write bandwidth from ext3

On Jun 19, 2006  14:18 -0400, Sinha_Himanshu at emc.com wrote:
> We measured the write bandwidth for writes to the block device
> corresponding to the lun (e.g. /dev/sdb), a file in an ext2 filesystem
> and to a file in an ext3 file system.
> 		Write b/w for 512 KB writes
> Block device	312 MBps
> Ext2 file		247 MBps
> Ext3 file		130 MBps
> 
> We are looking for ways to improve the ext3 file write bandwidth.

Have a look at the extents+mballoc+delalloc patches from Alex Tomas:

ftp://ftp.lustre.org/pub/people/alex/2.6.16.8/

Mount the filesystem with "-o extents,mballoc,delalloc" to enable this.

They noticably improve IO performance while also reducing the CPU load
for ext3.  The extent patches are approved by all of the ext3 developers
and will be supported upstream fairly soon (in the kernel and e2fsprogs),
and mballoc+delalloc will follow on afterward.

NOTE: the extents on-disk format is incompatible with older kernels, so
      at this stage consider it "for benchmarking only".

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.