[dm-devel] Device-mapper cluster locking

[Jonathan Brassow]

Thanks for your response.  Comments in-lined.

On Apr 7, 2010, at 12:49 PM, Mikulas Patocka wrote:

> Hi
>
> On Wed, 7 Apr 2010, Jonathan Brassow wrote:
>
>> I've been working on a cluster locking mechanism to be primarily  
>> used by
>> device-mapper targets.  The main goals are API simplicity and an  
>> ability
>> to tell if a resource has been modified remotely while a lock for the
>> resource was not held locally.  (IOW, Has the resource I am  
>> acquiring the
>> lock for changed since the last time I held the lock.)
>>
>> The original API (header file below) required 4 locking modes:  
>> UNLOCK,
>> MONITOR, SHARED, and EXCLUSIVE.  The unfamiliar one, MONITOR, is  
>> similar to
>> UNLOCK; but it keeps some state associated with the lock so that  
>> the next
>> time the lock is acquired it can be determined whether the lock was
>> acquired EXCLUSIVE by another machine.
>>
>> The original implementation did not cache cluster locks.  Cluster  
>> locks
>> were simply released (or put into a non-conflicting state) when the  
>> lock
>> was put into the UNLOCK or MONITOR mode.  I now have an  
>> implementation
>> that always caches cluster locks - releasing them only if needed by  
>> another
>> machine.  (A user may want to choose the appropriate implementation  
>> for
>> their workload - in which case, I can probably provide both  
>> implementations
>> through one API.)
>
> Maybe you can think about autotuning it --- i.e. count how many times
> caching "won" (the lock was taken by the same node) or "lost" (the  
> lock
> was acquired by another node) and keep or release the lock based on  
> the
> ratio of these two counts. Decay the counts over time, so that it  
> adjusts
> on workload change.

Certainly, that sounds like a sensible thing to do; and I think there  
is some precedent out there for this.

>
> How does that dlm protocol work? When a node needs a lock, what  
> happens?
> It sends all the nodes message about the lock? Or is there some master
> node as an arbiter?

Yes, all nodes receive a message.  No, there is no central arbiter.   
For example, if 4 nodes have a lock SHARED and the 5th one wants the  
lock EXCLUSIVE, the 4 nodes will get a notice requesting them to drop  
(or at least demote) the lock.

>
>> The interesting thing about the new caching approach is
>> that I probably do not need this extra "MONITOR" state.  (If a lock  
>> that
>> is cached in the SHARED state is revoked, then obviously someone is  
>> looking
>> to alter the resource.  We don't need to have extra state to give  
>> us what
>> can already be inferred and returned from cached resources.)
>
> Yes, MONITOR and UNLOCK could be joined.
>
>> I've also been re-thinking some of my assumptions about whether we
>> /really/ need separate lockspaces and how best to release resources
>> associated with each lock (i.e. get rid of a lock and its memory
>> because it will not be used again, rather than caching  
>> unnecessarily).
>> The original API (which is the same between the cached and non- 
>> caching
>> implementations) only operates by way of lock names.  This means a
>> couple of things:
>> 1) Memory associated with a lock is allocated at the time the lock is
>>   needed instead of at the time the structure/resource it is  
>> protecting
>>   is allocated/initialized.
>> 2) The locks will have to be tracked by the lock implementation.   
>> This
>>   means hash tables, lookups, overlapping allocation checks, etc.
>> We can avoid these hazards and slow-downs if we separate the  
>> allocation
>> of a lock from the actual locking action.  We would then have a lock
>> life-cycle as follows:
>> - lock_ptr = dmcl_alloc_lock(name, property_flags)
>> - dmcl_write_lock(lock_ptr)
>> - dmcl_unlock(lock_ptr)
>> - dmcl_read_lock(lock_ptr)
>> - dmcl_unlock(lock_ptr)
>> - dmcl_free_lock(lock_ptr)
>
> I think it is good --- way better than passing the character string on
> every call, parsing the string, hashinh it and comparing.
>
> If you do it this way, you speed up lock acquires and releases.
>
>> where 'property flags' is, for example:
>> PREALLOC_DLM: Get DLM lock in an unlocked state to prealloc  
>> necessary structs
>
> How would it differ from non-PREALLOC_DLM behavior?

When a cluster lock is allocated, it could also acquire the DLM lock  
in the UNLOCKed state.  This forces the dlm to create the necessary  
structures for the lock and create entries in the global index.  This  
involves memory allocation (on multiple machines) and inter-machine  
communication.  The only reason you wouldn't want to do this is if the  
DLM module or the cluster infrastructure was not available at the time  
you are allocating the lock.

I could envision something like this if you were allocating the lock  
on module init for some reason.  In this case, you would want to delay  
the actions of the DLM until you needed the lock.

This seems like it would be a rare occurrence, so perhaps I could  
negate that flag to 'DELAY_DLM_INTERACTION' or some such thing.

>
>> CACHE_RD_LK: Cache DLM lock when unlocking read locks for later  
>> acquisitions
>
> OK.
>
>> CACHE_WR_LK: Cache DLM lock when unlocking write locks for later  
>> acquisitions
>
> OK.
>
>> USE_SEMAPHORE: also acquire a semaphore when acquiring cluster lock
>
> Which semaphore? If the user needs a specific semaphore, he can just
> acquire it with down() --- there is no need to overload dm-locking  
> with
> that. Or is there any other reason why it is needed?

Ok, I thought this might bring a degree of convenience; but I will  
happily not include this option if it makes things bloated.  I will  
simply leave this out in any initial version.

>> Since the 'name' of the lock - which is used to uniquely identify a  
>> lock by
>> name cluster-wide - could conflict with the same name used by  
>> someone else,
>> we could allow locks to be allocated from a new lockspace as well.   
>> So, the
>> option of creating your own lockspace would be available in  
>> addition to the
>> default lockspace.
>
> What is the exact lockspace-lockname relationship? You create
> locspace "dm-snap" and lockname will be UUID of the logical volume?

The lockspace can be thought of as the location from which you acquire  
locks.  When simply using UUIDs as names of locks, a single default  
lockspace would suffice.  However, if you are using block numbers or  
inode numbers as your lock names, these names may conflict if you were  
locking the same block number on two different devices.  In that case,  
you might create a lockspace for each device (perhaps named by the  
UUID) and acquire locks from these independent lock spaces based on  
block numbers.  Since the locks are being sourced from independent  
lockspaces, there is no chance of overloading/conflict.

IOW, if your design uses names for locks that could be used by other  
users of the DLM, you should consider creating your own lockspace.  In  
fact, the default lockspace that would be available through the this  
API would actually be a lockspace created specifically for the users  
of this new API - to prevent any possible conflict with other DLM  
users.  So in actuality, you would only need to create a new lockspace  
if you thought your lock names might conflict with those from other  
device-mapper target instances (including your own if you are using  
block numbers as the lock names).

>
>> The code has been written, I just need to arrange it into the right  
>> functional
>> layout...  Would this new locking API make more sense to people?   
>> Mikulas,
>> what would you prefer for cluster snapshots?
>>
>> brassow
>
> I think using alloc/free interface is good.
>
> BTW. also, think about failure handling. If there is a communication
> problem, the lock may fail. What to do? Detach the whole exception  
> store
> and stop touching it? Can unlock fail?

Yes, dlm operations can fail or stall (due to quorum issues or network  
outage).  I'll talk with some of the other users (GFS) to see how they  
cope with these issues.

The caching aspect may help limit some of the failure cases.  If you  
unlock, the DLM lock will not be release until it is needed.  A  
machine will be notified of the need to release the lock only if  
communication is working properly.

The locking API can always return failure from the functions and leave  
the decision up to the user; but perhaps there are better solutions  
already in play by other users of the DLM.  I will ask them.

  brassow