[libvirt] [PATCH] Firewall : let libvirtd proceed after verifying locking args

[Daniel P. Berrange]

On Fri, Dec 26, 2014 at 03:54:12PM +0530, Prerna Saxena wrote:
> I recently encountered a situation where an unclean ebtables shutdown
> caused /var/lib/ebtables/lock to be left behind. When libvirtd was
> started on such a system, it caused libvirtd to "hang". Reason:

So you are saying that the lockfile exists, but no ebtables process
is still running ?

> While probing to check if locking was supported, libvirt runs this
> command synchronously :
>   #  /usr/sbin/ebtables --concurrent -L
> And this seemed to go on with msgs like :
>  Trying to obtain lock /var/lib/ebtables/lock
>  Trying to obtain lock /var/lib/ebtables/lock
>  Trying to obtain lock /var/lib/ebtables/lock
> 
> Result:
> Libvirtd never recovered from this scenario, and the system was
> essentially unable to start any VMs.
> 
> The following patch fixes this scenario:
> -----------------------------------------------------------
> >From ec245eccc03e8a69dc2c2e6edbf30a7b34eb74d0 Mon Sep 17 00:00:00 2001
> From: Prerna Saxena <prerna at linux.vnet.ibm.com>
> Date: Fri, 26 Dec 2014 15:24:45 -0500
> Subject: [PATCH] Firewall : let libvirtd proceed after verifying valid locking
>  args.
> 
> Commit dc33e6e4a5a5d42 introduces locking args to be run with [eb/ip]tables to
> determine whether locking is supported. However, this command needs to be
> run asynchronously ( as against its present synchronous run), and needs to be
> gracefully terminated once the job is done. Else it can potentially stall libvirtd
> with messages like :
>  "Trying to acquire lock ..."

This doesn't really fully fix the problem - the stale lockfile still
exists, so as soon as you try to start a VM with filters, we're still
going to get stuck. You're just delaying the point at which anyone
will see the problem, which I'm not convinced is good as the admin
is far more likely to see the problem during libvirtd startup, than
when launching a VM.

Really, the fault lies with ebtables for just using the existance of
the lock file as its mutex. It should be fixed to use fcntl locks
on the file as its mutex, which guarantees you can never get any stale
lock files even when exiting uncleanly.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|