[linux-lvm] Why do lvcreate with clvmd insist on VG being available on all nodes?

[Jacek Konieczny]

On Thu, Nov 15, 2012 at 12:01:10PM +0100, Zdenek Kabelac wrote:
> Dne 15.11.2012 11:08, Jacek Konieczny napsal(a):
> > On Thu, Nov 15, 2012 at 10:09:35AM +0100, Zdenek Kabelac wrote:
> >>> work properly, as I would expect (make the volume available/unavailable
> >>> on the node). But an attempt to create a new volume:
> >>>
> >>> lvcreate -n new_volume -L 1M shared_vg
> >>>
> >>> fails with:
> >>>
> >>> Error locking on node 1: Volume group for uuid not found: Hlk5NeaVF0qhDF20RBq61EZaIj5yyUJgGyMo5AQcLfZpJS0DZUcgj7QMd3QPWICL
> >>>

> >> Haven't really tried to understand what are you trying to achieve,
> >> but if you want to have node being activated only on one cluster node,
> >> you may easily use    lvcreate -aey  option.
> >>
> > My stupid mistake, indeed.
> >
> > 'lvcreate -an -Z n' and 'lvcreate -aey' do work in such case.


> > Though, LVM have some problems with tracking the exclusive activations
> > later…
> 
> If you know about any such bug - just open rhbz with full description of such 
> erroneous case.

It was just another mistake of mine. LVM properly tracks the exclusive
locks – the volumes were being deactivated by something else.

> > Clusters do not have to be symmetrical. Cluster when different nodes
> > have a bit different set of resources available are still clusters.
> 
> You want to support different scheme - thus you need to probably write your 
> own clvmd-like daemon to cover all new cases you bring in with non-symmetrical 
> cases.

I think this will not be needed.

> clvmd typical use case is  'vg' used on couple cluster nodes.

I see.
 
> While you are probably trying to use  N:M mapping of vg and clustered nodes.

Exactly. But it seems it should not be a problem in my case.


After knowing my mistake I can see LVM already provides the
On Thu, Nov 15, 2012 at 12:01:10PM +0100, Zdenek Kabelac wrote:
> Dne 15.11.2012 11:08, Jacek Konieczny napsal(a):
> > On Thu, Nov 15, 2012 at 10:09:35AM +0100, Zdenek Kabelac wrote:
> >>> work properly, as I would expect (make the volume available/unavailable
> >>> on the node). But an attempt to create a new volume:
> >>>
> >>> lvcreate -n new_volume -L 1M shared_vg
> >>>
> >>> fails with:
> >>>
> >>> Error locking on node 1: Volume group for uuid not found: Hlk5NeaVF0qhDF20RBq61EZaIj5yyUJgGyMo5AQcLfZpJS0DZUcgj7QMd3QPWICL
> >>>

> >> Haven't really tried to understand what are you trying to achieve,
> >> but if you want to have node being activated only on one cluster node,
> >> you may easily use    lvcreate -aey  option.
> >>
> > My stupid mistake, indeed.
> >
> > 'lvcreate -an -Z n' and 'lvcreate -aey' do work in such case.


> > Though, LVM have some problems with tracking the exclusive activations
> > later…
> 
> If you know about any such bug - just open rhbz with full description of such 
> erroneous case.

It was just another mistake of mine. LVM properly tracks the exclusive
locks – the volumes were being deactivated by something else.

> > Clusters do not have to be symmetrical. Cluster when different nodes
> > have a bit different set of resources available are still clusters.
> 
> You want to support different scheme - thus you need to probably write your 
> own clvmd-like daemon to cover all new cases you bring in with non-symmetrical 
> cases.

I think this will not be needed.

> clvmd typical use case is  'vg' used on couple cluster nodes.

I see.
 
> While you are probably trying to use  N:M mapping of vg and clustered nodes.

Exactly. But it seems it should not be a problem in my case.

After knowing my mistake I can see LVM already provides the
functionality I need.

To summarize:

- The default LV activation mode is '-ay', which means, for clustered
  volume groups, that the volume is to be active on every node in the
  cluster.

– The activation is not always explicit. 'lvcreate' uses '-ay' when no
  other '-a' option is given.

– The '-ay' activation won't work if any node in the cluster cannot
  access the volume group (e.g. when the DRBD device holding it is
  Secondary or not configured at that node).

– However, my use case doesn't need more than one node using any of the
  volumes at any time. In fact, it is very important that only a single
  machine uses each LV at a time.

– For this scenario '-aey' should be always used.

– Explicitly using '-aey' with 'lvcreate' fixes the 'Volume group for
  uuid not found' for me.

– Other tests made showed that the locking works as expected when
  a volume group becomes available and unavailable on various cluster
  nodes. A node having access to a VG can create and/or activate LVs
  there in exclusive node and all other nodes will comply with that
  lock whenever they gain access to this VG.

So, it seems that clvmd is not that bound to the 'symmetrical cluster'
scenario, provided no more than one node needs to access a volume at
a time.

Does this make sense?

Greets,
        Jacek