[linux-lvm] advice for curing terrible snapshot performance?
chris (fool) mccraw
gently at gmail.com
Fri Nov 12 21:52:25 UTC 2010
i'm new to linux lvm but a longtime user of lvm's on other commercial
unices. i love some of the features like r/w snapshots! and indeed
snapshots are the primary reason i'm even interested in using LVM on
however, snapshots really shoot my system in the foot. in my (12
processor, 12GB RAM, x86_64 centos 5.5) server, i have two pricey
areca hardware raid cards that give me ridiculous write performance:
i can sustain over 700MByte/sec write speeds (writing a file with dd
if=/dev/zero bs=1M, twice the size of the raid card's onboard memory
and timing the write + a sync afterwards) to the volumes on either
card (both backed by 7 or more fast disks). enabling a single
snapshot reduces that speed by a factor of 10! enabling more
snapshots isn't as drastic but still doesn't really scale well:
no snapshot = ~11sec (727MB/sec)
1 snapshot = ~102sec (78MB/sec)
2 snapshots = ~144sec (55MB/sec)
3 snapshots = ~313sec (25MB/sec)
4 snapshots = ~607sec (15MB/sec)
i have my snapshots set up a separate array from the master
filesystem, on a separate raid card. i did not change the default
parameters for the setup (ie blocksize) because our typical workload
is reading and writing small (<64k) files. i can copy non-snapshotted
files from an LVM on array1 to array2 at a good clip, 307MByte/sec
(including sync). copies from the parent array to itself (no
snapshots enabled) go at about 220MByte/sec.
all of my measurements were repeated 4x and averaged--occasionally
there was one that was a good 30% faster than the other 3, but it was
always an outlier. typically all measurements for a given scenario
were within 10% of eachother.
so i guess that as a replacement for a netapp, setup with a few hourly
& daily, and even one weekly snapshot isn't something people do with
stock linux LVM? or am i just doing it wrong?
in searching the archives i heard about zumastor. is that really
production-ready? the no-new-releases in the last 2 years and not
being in the mainstream kernel makes me leery of it. i think we can
live with the factor-of-10 performance degradation on a daily
basis--we can turn off all the snapshots in case we really have to
hammer the server (which has a 4Gbit uplink to a render farm, so it is
possible for us to actually write over 70MByte/sec when things are
humming, via NFS), and in general it serves SMB at closer to 400Mbit
than 4000, so all the desktop users will not notice a difference.
it seems that others have seen these problems:
http://www.nikhef.nl/~dennisvd/lvmcrap.html as an example.
thanks in advance for your input!
More information about the linux-lvm