[libvirt] heisenbug in command.c

[Jim Paris]

Serge Hallyn wrote:
> On 03/16/2012 11:50 AM, Eric Blake wrote:
> >On 03/16/2012 10:36 AM, Serge Hallyn wrote:
> >>Hi,
> >>
> >>It seems I've run into quite the heisenbug, reported at
> >>https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/922628
> >>
> >>It manifests itself as virPidWait returning status=4 for iptables (which
> >>should never exit with status=4).
> >
> >Maybe iptables isn't documented as exiting with $? of 4, but that's what
> >is happening.  The libvirt code in question is quite clear that it
> >grabbed an accurate exit status from the child process.
> >
> 
> Well, yes.  I figured that either (1) iptables actually got -EINTR
> from the kernel and passed that along as its exit code, or (2)
> something went wrong with memory being overwritten in libvirt,
> however unlikely. Stranger things have happened.  If (1), I was
> wondering if it was being ignored on purpose.

Why do you bring up EINTR at all?  Just because EINTR is 4?  That
seems very much unrelated.

This is from iptables:

enum xtables_exittype {
        OTHER_PROBLEM = 1,
        PARAMETER_PROBLEM,
        VERSION_PROBLEM,
        RESOURCE_PROBLEM,
        XTF_ONLY_ONCE,
        XTF_NO_INVERT,
        XTF_BAD_VALUE,
        XTF_ONE_ACTION,
};

So it looks like iptables is returning RESOURCE_PROBLEM (which could
explain why it's intermittent).

-jim

> >     ret = virPidWait(cmd->pid, exitstatus ? exitstatus :&status);
> >     if (ret == 0) {
> >         cmd->pid = -1;
> >         cmd->reap = false;
> >         if (status) {
> >             char *str = virCommandToString(cmd);
> >             char *st = virCommandTranslateStatus(status);
> >             virCommandError(VIR_ERR_INTERNAL_ERROR,
> >                             _("Child process (%s) status unexpected: %s"),
> >                             str ? str : cmd->args[0], NULLSTR(st));
> >
> >>  But it's only been seen on two (very
> >>different) machines, and the slightest shifting of the winds makes it go
> >>away.  Given how sneaky this bug appears to be, there's a slight
> >>temptation to have iptablesAddRemoveRule pass in a int* for status and
> >>better deal with the -EINTR.  But I fear that might be papering over a
> >>worse race.
> >
> >I don't follow how you think there is a -EINTR being encountered in
> >libvirt.
> 
> Yeah I don't really either.
> 
> > I think you'd be better off investigating why iptables really
> >is exiting with status 4.
> 
> Well, given what EINTR means, shouldn't src/util/iptables.c re-try
> the command if it gets that?
> 
> Anyway I'll keep digging, but was wondering if anyone else has seen this.
> 
> -serge
> 
> --
> libvir-list mailing list
> libvir-list at redhat.com
> https://www.redhat.com/mailman/listinfo/libvir-list