[Linux-cluster] Announce: Higher level cluster raid

[Daniel Phillips]

Hi all,

OK, it took quite a bit longer to get basic throughput numbers than I 
had hoped, because I ran into a nasty performance bug and thought I'd 
better do something about it first.  This is solved, and performance
is now looking pretty good.

I don't have a suitable high performance cluster at hand so I am working 
with a scsi array on a local machine.  However, the full cluster 
synchronization stack is used, including a cluster-aware persistent 
dirty log (remembers which regions had in-flight writes in order to 
limit post-crash resyncing).  Though I have five disks available, I 
used one for the persistent dirty log, so I am restricted to an 
order-one ddraid array of three disks at the moment.  One of the disks 
is a dedicated parity disk, so the best I could hope for is a factor of 
two throughput increase versus a single raw disk.

For larger transfer sizes, throughput does in fact double compared to 
raw IO to one of the scsi disks.  For small transfer sizes, the 
overhead of parity calculations, dirty logging and bio cloning become 
relatively large versus raw disk IO, so breakeven occurs at about 64K 
transfer size.  Below that, a single raw disk is faster and above, the 
ddraid array is faster.  With 1 MB transfers, the ddraid array is 
nearly twice as fast.

I tried various combinations disabling the parity calculations and dirty 
logging to see how the write overheads break down:

   No persistent dirty log, no parity calculations:
      Tie at 8K
      Almost twice as fast at 32K and above

   Parity calculations, no persistent dirty log:
      Tie between 8K and 16K

   Persistent dirty log, no parity calculations:
      Tie at 16K

   Parity calculations and persistent dirty log:
      Tie at 64K

We see from this that dirty logging is the biggest overhead, which is no 
surprise.  After that, the overhead parity calculation and basic 
bookkeeping seem about the same.  The parity calculations can easily be 
optimized, the bio bookkeeping overhead will be a little harder.  There 
are probably a few tricks remaining to reduce the dirty log overhead.

The main point, though, is that even before a lot of optimization, 
performance looks good enough for production use.  Some loads will be a 
little worse, some loads will perform dramatically better.  Over time, 
I suspect various optimizations will reduce the per-transfer overhead 
considerably, so that the array is always faster.

Whether or not the array is faster, it certainly is more redundant than 
a raw disk.  This was my primary objective, and increased performance 
is just a nice fringe benefit.  Of course, with no other cluster raid 
implemention to compete with, this is by default the fastest one :-)

Performance notes:

  - Scatter/gather worries turned out to be unfounded.  The scsi
    controller handles the thousands of sg regions I throw at it per
    second without measureable overhead.  Just in case, I investigated
    the overhead of copying the IO into linear chunks via memcpy,
    which turned out to be small but noticeable.  Gnbd and iscsi do
    network scatter/gather, which I haven't tested yet, but I would be
    surprised if there is a problem.

  - Read transfers have no dirty log overhead.

  - For reliability, I check parity on read.  Later this will be an
    option, so read transfers don't necessarily have parity overhead
    either.

  - I think I may be able to increase read throughput to N times single
    disk throughput, instead of N-1.

  - I haven't determined where the bookkeeping overhead comes from,
    but I suspect most of it can be eliminated.

  - The nasty performance bug: releasing a dirty region immediately
    after all writes complete is a bad idea that really hammers the
    performance of back to back writes.  There is now a timer delay
    on release for each region, which cures this.

This code is also capable of running an n-way cluster mirror, but that 
feature is broken at the moment.  I'll restore it next week and we can 
check mirror performance as well.

I need to add degraded mode IO and parity reconstruction before this is 
actually useful, which I must put off until after LCA.  Code cleanup is 
in progress and should land in cvs Monday or Tuesday.  More benchmark 
numbers and hopefully some pretty charts are on the way.

Regards,

Daniel