Disk IO issues
Sascha Thomas Spreitzer
sascha at spreitzer.name
Wed Dec 31 21:56:24 UTC 2008
Hello Mike,
maybe the RAID mirror failed and is auto healing in background. Is
there a way to determine the RAID state?
Is the RAID controller showing any errors? Physical disks reporting seek errors?
I assume a problem with either hardware buffers of disk or RAID
controller or faulty disk or RAID hardware.
regards,
Sascha
2008/12/31 Mike McGrath <mmcgrath at redhat.com>:
> Lets pool some knowledge together because at this point, I'm missing
> something.
>
> I've been doing all measurements with sar as bonnie, etc, causes builds to
> timeout.
>
> Problem: We're seeing slower then normal disk IO. At least I think we
> are. This is a PERC5/E and MD1000 array.
>
> When I try to do a normal copy "cp -adv /mnt/koji/packages /tmp/" I get
> around 4-6MBytes/s
>
> When I do a cp of a large file "cp /mnt/koji/out /tmp/" I get
> 30-40MBytes/s.
>
> Then I "dd if=/dev/sde of=/dev/null" I get around 60-70 MBytes/s read.
>
> If I "cat /dev/sde > /dev/null" I get between 225-300MBytes/s read.
>
> The above tests are pretty consistent. /dev/sde is a raid5 array,
> hardware raid.
>
> So my question here is, wtf? I've been working to do a backup which I
> would think would either cause network utilization to max out, or disk io
> to max out. I'm not seeing either. Sar says the disks are 100% utilized
> but I can cause major increases in actual disk reads and writes by just
> running additional commands. Also, if the disks were 100% utilized I'd
> expect we would see lots more iowait. We're not though, iowait on the box
> is only %0.06 today.
>
> So, long story short, we're seeing much better performance when just
> reading or writing lots of data (though dd is many times slower then cat).
> But with our real-world traffic, we're just seeing crappy crappy IO.
>
> Thoughts, theories or opinions? Some of the sysadmin noc guys have access
> to run diagnostic commands, if you want more info about a setting, let me
> know.
>
> I should also mention there's lots going on with this box, for example its
> hardware raid, lvm and I've got xen running on it (though the tests above
> were not in a xen guest).
>
> -Mike
>
> _______________________________________________
> Fedora-infrastructure-list mailing list
> Fedora-infrastructure-list at redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
>
--
Mit freundlichen Grüßen, / with kind regards,
Sascha Thomas Spreitzer
http://spreitzer.name/
More information about the Fedora-infrastructure-list
mailing list