[Linux-cluster] CLVMD without GFS

Tue Jul 21 11:16:19 UTC 2009

Hi Chrissie,

Indeed, by default when creating the VG, it is clustered, thus when creating
the LV it is active on all nodes.

To avoid data corruption, I have re created the VG as non clustered
(vgcreate -c n vgXX) then created the LV which got activated only on the
node where it got created.

Then changed the VG to clustered (vgchange -c y VGXX) and activated it
exclusively on this node.

But, I could reproduce the behaviour of bypassing the exclusive flag:

On node B, re change the VG to non clustered though it is activated
exclusively on node A.

and then activate it on node B and it works.

The thing I'm trying to point is that simply by erasing the clustered flag
you can bypass the exclusive activation.

I think a barrier is necessary to prevent this to happen, removing the
clustered flag from a VG should be possible only if the node holding the VG
exclusively is down (does the lock manager  DLM report which node holds
exclusively a VG ?)

Thanks

Brem

2009/7/21, Christine Caulfield <ccaulfie at redhat.com>:
>
> Hiya,
>
> I've just tried this on my cluster and it works fine.
>
> What you need to remember is that lvcreate on one node will also activate
> the LV on all nodes in the cluster - it does an implicit lvchange -ay when
> you create it. What I can't explain is why vgchange -ae seemed to work fine
> on node A, it should give the same error as on node B because LVs are open
> shared on both nodes.
>
> Its not clear to me when you tagged the VG as clustered, so that might be
> contributing to the problem. When I create a new VG on shared storage it
> automatically gets labelled clustered so I have never needed to do this
> explicitly. If you create a non-clustered VG you probably ought to
> deactivate it on all nodes first as it could mess up the locking otherwise.
> This *might* be the cause of your troubles.
>
> The error on clvmd startup can be ignored. It's caused by clvmd ussing a
> background command with --no_locking so that it can check which LVs (if any)
> are already active and re-acquire locks for them
>
> Sorry this isn't conclusive, The exact order in which things are happening
> is not clear to me.
>
> Chrissie.
>
> On 07/21/2009 10:21 AM, brem belguebli wrote:
>
>> Hi all,
>> I think there is something to clarify about using CLVM across a cluster
>> in a active/passive mode without GFS.
>>  From my understanding, CLVM keeps LVM metadata coherent among the
>> cluster nodes and provides a cluster wide locking mechanism that can
>> prevent any node from trying to activate a volume group if it has been
>> activated exclusively (vgchange -a e VGXXX)  by another node (which
>> needs to be up).
>> I have been playing with it to check this behaviour but it doesn't seem
>> to make what is expected.
>> I have 2 nodes (RHEL 5.3 X86_64, cluster installed and configured) , A
>> and B using a SAN shared storage.
>> I  have a LUN from this SAN seen by both nodes, pvcreate'd
>> /dev/mpath/mpath0 , vgcreate'd vg10 and lvcreate'd lvol1 (on one node),
>> created an ext3 FS on /dev/vg10/lvol1
>> CLVM is running in debug mode (clvmd -d2 ) (but it complains about
>> locking disabled though locking set to 3 on both nodes)
>> On node A:
>>           vgchange -c y vg10 returns OK (vgs -->  vg10     1   1   0
>> wz--nc)
>>           vgchange -a e --> OK
>>           lvs returns lvol1   vg10   -wi-a-
>> On node B (while things are active on A, A is UP and member of the
>> cluster ):
>>           vgchange -a e --> Error locking on node B: Volume is busy on
>> another node
>>                                    1 logical volume(s) in volume group
>> "vg10" now active
>> It activates vg10 even if it sees it busy on another node .
>> on B, lvs returns lvol1   vg10   -wi-a-
>> as well as on A.
>> I think the main problem comes from the fact that, as it is said when
>> starting CLVM in debug mode,  WARNING: Locking disabled. Be careful!
>> This could corrupt your metadata.
>> IMHO, the algorithm should be as follows:
>> VG is tagged as clustered (vgchange -c y VGXXX)
>> if a node (node P) tries to activate the VG exclusively (vgchange -a
>> VGXXX)
>> ask the lock manager to check if VG is not already locked by another
>> node (node X)
>> if so, check if node X is up
>> if node X is down, return OK to node P
>> else
>> return NOK to node P (explicitely that VG is held exclusively by node X)
>> Brem
>> PS: this shouldn't be a problem with GFS or other clustered FS (OCFS,
>> etc...) as no node should try to activate exclusively any VG.
>>
>>
>> ------------------------------------------------------------------------
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20090721/f2699fe9/attachment.htm>