[Linux-cluster] Deadlock when using clvmd + OpenAIS + Corosync

[Evan Broder]

On Mon, Jan 11, 2010 at 4:03 AM, Christine Caulfield
<ccaulfie at redhat.com> wrote:
> On 08/01/10 22:58, Evan Broder wrote:
>>
>> [please preserve the CC when replying, thanks]
>>
>> Hi -
>>    We're attempting to setup a clvm (2.02.56) cluster using OpenAIS
>> (1.1.1) and Corosync (1.1.2). We've gotten bitten hard in the past by
>> crashes leaving DLM state around and forcing us to reboot our nodes,
>> so we're specifically looking for a solution that doesn't involve
>> in-kernel locking.
>>
>> We're also running the Pacemaker OpenAIS service, as we're hoping to
>> use it for management of some other resources going forward.
>>
>> We've managed to form the OpenAIS cluster, and get clvmd running on
>> both of our nodes. Operations using LVM succeed, so long as only one
>> operation runs at a time. However, if we attempt to run two operations
>> (say, one lvcreate on each host) at a time, they both hang, and both
>> clvmd processes appear to deadlock.
>>
>> When they deadlock, it doesn't appear to affect the other clustering
>> processes - both corosync and pacemaker still report a fully formed
>> cluster, so it seems the issue is localized to clvmd.
>>
>> I've looked at logs from corosync and pacemaker, and I've straced
>> various processes, but I don't want to blast a bunch of useless
>> information at the list. What information can I provide to make it
>> easier to debug and fix this deadlock?
>>
>
> To start with, the best logging to produce is the clvmd logs which can be
> got with clvmd -d (see the man page for details). Ideally these should be
> from all nodes in the cluster so they can be correlated. If you're still
> using DLM then a dlm lock dump from all nodes is often helpful in
> conjunction with the clvmd logs.

Sure, no problem. I've posted the logs from clvmd on both processes in
<http://web.mit.edu/broder/Public/clvmd/>. I've annotated them at a
few points with what I was doing - the annotations all start with "
>> ", so they should be easy to spot.

One interesting thing was the output from lvcreate when I SIGKILLed
the clvmd process:

root at black-mesa:~# lvcreate -L 1G -n broder-test-1 xenvg
  Error reading data from clvmd: Connection reset by peer
  Aborting. Failed to activate new LV to wipe the start of it.
  Error writing data to clvmd: Broken pipe
  Unable to deactivate failed new LV. Manual intervention required.
  Error writing data to clvmd: Broken pipe
  Internal error: Volume Group xenvg was not unlocked
  Device '/dev/sdb' has been left open.
  Device '/dev/sdc' has been left open.
  Device '/dev/sdc' has been left open.
  Device '/dev/sdb' has been left open.

I'm not entirely sure what to make of that, or how worried I should
be. After I killed clvmd and restarted corosync, broder-test-1 (the
first lvcreate I started) did seem to exist, although I didn't look
too hard at it.

> Also, did you know it's possible to use clvmd without the DLM? The -I
> openais option will tell it to use the Lck service in userspace - though if
> there are DLM bugs I think we'd like to fix them if possible ;-)

I suspect that the DLM bugs we ran into were all from using old
software - it just left a bit of a bitter taste in our mouth when we
were left with no way to recover from problems. We  built the clvmd
we're using now without support for any clustering infrastructure but
openais, so we should always be using the Lck service.

>
> Chrissie
>

Let us know if there's any other information we can provide.

- Evan