[Cluster-devel] cluster4 gfs_controld

[Steven Whitehouse]

Hi,

On Thu, 2011-10-13 at 10:20 -0400, David Teigland wrote:
> Here's the outline of my plan to remove/replace the essential bits of
> gfs_controld in cluster4.  I expect it'll go away entirely, but there
> could be one or two minor things it would still handle on the side.
> 
> kernel dlm/gfs2 will continue to be operable with either
> . cluster3 dlm_controld/gfs_controld combination, or
> . cluster4 dlm_controld only
> 
> Two main things from gfs_controld need replacing:
> 
> 1. jid allocation, first mounter
> 
> cluster3
> . both from gfs_controld
> 
> cluster4
> . jid from dlm-kernel "slots" which will be assigned similarly
What is the actual algorithm used to assign these slots?

> . first mounter using a dlm lock in lock_dlm
> 
That sounds good to me. The thing we need to resolve is how do we get
from one to the other. We may have to introduce a new name for the lock
protocol to avoid people accidentally using both schemes in the same
cluster.

> 2. recovery coordination, failure notification
> 
> cluster3
> . coordination of dlm-kernel/gfs-kernel recovery is done
>   indirectly in userspace between dlm_controld/gfs_controld,
>   which then toggle sysfs files.
> . write("sysfs block", 0) -> block_store(1)
>   write("sysfs recover", jid) -> recover_store(jid)
>   write("sysfs block", 1) -> block_store(0)
> 
> cluster4
> . coordination of dlm-kernel/gfs-kernel recovery is done
>   directly in kernel using callbacks from dlm-kernel to gfs-kernel.
> . gdlm_mount(struct gfs2_sbd *sdp, const char *table, int *first, int *jid)
>   calls dlm_recover_register(dlm, &jid, &recover_callbacks)
Can we not just pass the extra functions to dlm_create_lockspace? That
seems a bit simpler than adding an extra function just to register the
callbacks,

Steve.

> . gdlm_recover_prep() -> block_store(1)
>   gdlm_recover_slot(jid) -> recover_store(jid)
>   gdlm_recover_done() -> block_store(0)
> 
> cluster3 dlm/gfs recovery
> . dlm_controld sees nodedown                      (libcpg)
> . gfs_controld sees nodedown                      (libcpg)
> . dlm_controld stops dlm-kernel                   (sysfs control 0)
> . gfs_controld stops gfs-kernel                   (sysfs block 1)
> . dlm_controld waits for gfs_controld kernel stop (libdlmcontrol)
> . gfs_controld waits for dlm_controld kernel stop (libdlmcontrol)
> . dlm_controld syncs state among all nodes        (libcpg)
> . gfs_controld syncs state among all nodes        (libcpg)
> . dlm_controld starts dlm-kernel recovery         (sysfs control 1)
> . gfs_controld starts gfs-kernel recovery         (sysfs recover jid)
> . gfs_controld starts gfs-kernel                  (sysfs block 0)
> 
> cluster4 dlm/gfs recovery
> . dlm_controld sees nodedown                      (libcpg)
> . dlm_controld stops dlm-kernel                   (sysfs control 0)
> . dlm-kernel stops gfs-kernel                     (callback block 1)
> . dlm_controld syncs state among all nodes        (libcpg)
> . dlm_controld starts dlm-kernel recovery         (sysfs control 1)
> . dlm-kernel starts gfs-kernel recovery           (callback recover jid)
> . dlm-kernel starts gfs-kernel                    (callback block 0)
>