Disk IO issues

Mike McGrath mmcgrath at redhat.com
Thu Jan 1 18:48:13 UTC 2009


On Thu, 1 Jan 2009, Kostas Georgiou wrote:

> On Thu, Jan 01, 2009 at 01:17:38AM -0600, Mike McGrath wrote:
>
> > On Thu, 1 Jan 2009, James Antill wrote:
> >
> > > On Wed, 2008-12-31 at 14:42 -0600, Mike McGrath wrote:
> > > > Lets pool some knowledge together because at this point, I'm missing
> > > > something.
> > > >
> > > > I've been doing all measurements with sar as bonnie, etc, causes builds to
> > > > timeout.
> > > >
> > > > Problem: We're seeing slower then normal disk IO.  At least I think we
> > > > are.  This is a PERC5/E and MD1000 array.
> > > >
> > > > When I try to do a normal copy "cp -adv /mnt/koji/packages /tmp/" I get
> > > > around 4-6MBytes/s
> > >
> > >  This _might_ not be "IO" in a normal sense, -a to cp means:
> > >
> > >  file data + file inode + ACLs + selinux + xattrs [+ file capabilities]
> > >
> > > ...esp. given that you aren't getting large IOWait times, you might want
> > > to strace -T the cp and do some perl/whatever on the result to see what
> > > is eating up the time.
> >
> > Even with non cp type things (like a bacula backup) it just doesn't seem
> > as fast as I would expect it to be.  I've never actually done trending at
> > this level / scale on a filesystem / drive before.  So I really don't have
> > a good baseline except that it just seems slow to me.
> >
> > Other then the much faster direct block access and the large file reads, I
> > don't have much else to go on that makes me think its slow.
>
> Do writes show the same pattern? If you use selinux/ACLs/xattrs the default
> inode size of 128 can cause slowdowns (#205161 for example).
>

One reason I'm trying to ramp this up now is because the koji share is
still under 50% utilized.  If it turns out to be something in the
filesystem, its not too late for us to shrink the main filesystem, create
the new, copy, and grow the new.

> Can you run blktrace+seekwatcher (both in EPEL) to get an idea on
> what is going on? An iostat -x -k /dev/sde 1 output will also be
> helpfull.
>

I'll take a look at those two applications as well, here's the iostat:

Linux 2.6.18-92.1.18.el5xen (xen2.fedora.phx.redhat.com)        01/01/2009

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.55    0.01    1.35    0.10    6.28   91.71

Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sde            1389.22    95.13 161.74 270.46  6693.75  1670.16    38.70   1.09    2.51   1.48  64.04


	-Mike




More information about the Fedora-infrastructure-list mailing list