[linux-lvm] LVM commands extremely slow during raid check/resync

[James Candelaria]

Larkin,

This is likely due to the way the scheduler and MD driver are interacting.  The MD driver has likely filled the queue of the devices pretty deep with read requests, then when you attempt to run a LVM command such as lvcreate your request gets inserted.  You must "wait-in-line" for this command (likely only a few sectors worth of IO) to get its turn on the media.  The MD driver realizes that there is another request to the media and slows itself briefly, but since there is no further IO after the LVM command it then goes back to its job of resyncing your array in earnest.

James


-----Original Message-----
From: linux-lvm-bounces at redhat.com [mailto:linux-lvm-bounces at redhat.com] On Behalf Of Larkin Lowrey
Sent: Sunday, March 25, 2012 3:56 AM
To: linux-lvm at redhat.com
Subject: [linux-lvm] LVM commands extremely slow during raid check/resync

I've been suffering from an extreme slowdown of the various lvm commands
during high I/O load ever since updating from Fedora 15 to 16.

I notice this particularly Sunday AMs when Fedora kicks of a raid-check.
What is normally near instantaneous, commands like lvs and lvcreate
--snapshot take minutes to complete (literally). This causes my backup
jobs to timeout and fail.

While all this is going on, the various filesystems are reasonably
responsive (considering the raid-check is running) and I can read/write
to files without problems. It seems that this slow-down is unique to lvm.

I have three raid 5 arrays of 8, 6, and 6 drives. The root fs sits
entirely within the 8 disk array as does the spare area used for snapshots.

Interestingly, perhaps, if I can coax a backup into running, the lvs
command, for example, will complete in just 15-30 seconds instead of
120-180s. It would seem that the random I/O of the backup is able to
break things up enough for the lvm commands to squeeze in.

I'm at a loss for what to do about this or what data to scan for clues.
Any suggestions?

kernel 3.2.10-3.fc16.x86_64

lvm> version
  LVM version:     2.02.86(2) (2011-07-08)
  Library version: 1.02.65 (2011-07-08)
  Driver version:  4.22.0

--Larkin

_______________________________________________
linux-lvm mailing list
linux-lvm at redhat.com
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/