[libvirt] heisenbug in command.c

[Daniel P. Berrange]

On Fri, Mar 16, 2012 at 04:42:42PM -0500, Serge Hallyn wrote:
> On 03/16/2012 03:52 PM, Jim Paris wrote:
> >Serge Hallyn wrote:
> >>On 03/16/2012 11:50 AM, Eric Blake wrote:
> >>>On 03/16/2012 10:36 AM, Serge Hallyn wrote:
> >>>>Hi,
> >>>>
> >>>>It seems I've run into quite the heisenbug, reported at
> >>>>https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/922628
> >>>>
> >>>>It manifests itself as virPidWait returning status=4 for iptables (which
> >>>>should never exit with status=4).
> >>>
> >>>Maybe iptables isn't documented as exiting with $? of 4, but that's what
> >>>is happening.  The libvirt code in question is quite clear that it
> >>>grabbed an accurate exit status from the child process.
> >>>
> >>
> >>Well, yes.  I figured that either (1) iptables actually got -EINTR
> >>from the kernel and passed that along as its exit code, or (2)
> >>something went wrong with memory being overwritten in libvirt,
> >>however unlikely. Stranger things have happened.  If (1), I was
> >>wondering if it was being ignored on purpose.
> >
> >Why do you bring up EINTR at all?  Just because EINTR is 4?
> 
> Yup.
> 
> >  That
> >seems very much unrelated.
> 
> I didn't think so, but looks like you're right :)
> 
> >This is from iptables:
> >
> >enum xtables_exittype {
> >         OTHER_PROBLEM = 1,
> >         PARAMETER_PROBLEM,
> >         VERSION_PROBLEM,
> >         RESOURCE_PROBLEM,
> >         XTF_ONLY_ONCE,
> >         XTF_NO_INVERT,
> >         XTF_BAD_VALUE,
> >         XTF_ONE_ACTION,
> >};
> >
> >So it looks like iptables is returning RESOURCE_PROBLEM (which could
> >explain why it's intermittent).
> 
> That makes a lot more sense then, yes.  When scripting a bunch of
> parallel iptables rules adds (followed by waits), I do once in
> awhile get
> 
> iptables: Resource temporarily unavailable.
> 
> (and sometimes a -EINVAL message)
> 
> though 'wait' still says status was 0.  I've never gotten 4.  The
> next run then succeeds.
> 
> This still however sounds like src/util/iptables.c might ought to
> re-run the command if it gets 4.

Or do we need some serialization of commands at a higher level perhaps ?


Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|