Disk IO issues

[Mike McGrath]

On Thu, 1 Jan 2009, Kostas Georgiou wrote:

> On Thu, Jan 01, 2009 at 01:17:38AM -0600, Mike McGrath wrote:
>
> > On Thu, 1 Jan 2009, James Antill wrote:
> >
> > > On Wed, 2008-12-31 at 14:42 -0600, Mike McGrath wrote:
> > > > Lets pool some knowledge together because at this point, I'm missing
> > > > something.
> > > >
> > > > I've been doing all measurements with sar as bonnie, etc, causes builds to
> > > > timeout.
> > > >
> > > > Problem: We're seeing slower then normal disk IO.  At least I think we
> > > > are.  This is a PERC5/E and MD1000 array.
> > > >
> > > > When I try to do a normal copy "cp -adv /mnt/koji/packages /tmp/" I get
> > > > around 4-6MBytes/s
> > >
> > >  This _might_ not be "IO" in a normal sense, -a to cp means:
> > >
> > >  file data + file inode + ACLs + selinux + xattrs [+ file capabilities]
> > >
> > > ...esp. given that you aren't getting large IOWait times, you might want
> > > to strace -T the cp and do some perl/whatever on the result to see what
> > > is eating up the time.
> >
> > Even with non cp type things (like a bacula backup) it just doesn't seem
> > as fast as I would expect it to be.  I've never actually done trending at
> > this level / scale on a filesystem / drive before.  So I really don't have
> > a good baseline except that it just seems slow to me.
> >
> > Other then the much faster direct block access and the large file reads, I
> > don't have much else to go on that makes me think its slow.
>
> Do writes show the same pattern? If you use selinux/ACLs/xattrs the default
> inode size of 128 can cause slowdowns (#205161 for example).
>
> Can you run blktrace+seekwatcher (both in EPEL) to get an idea on
> what is going on? An iostat -x -k /dev/sde 1 output will also be
> helpfull.
>

Here's a seekwatcher of a find I ran:

http://mmcgrath.fedorapeople.org/find2.png

I had to kill it, I'll have a more full run soon.  Doing some other tests
now.

	-Mike