[libvirt] Entering freeze for libvirt-3.5.0

[Roman Bogorodskiy]

  Daniel P. Berrange wrote:

> On Mon, Jul 03, 2017 at 05:20:13PM +0400, Roman Bogorodskiy wrote:
> >   Ján Tomko wrote:
> > 
> > > [cc: Guido]
> > > 
> > > On Sat, Jul 01, 2017 at 02:18:58PM +0400, Roman Bogorodskiy wrote:
> > > >  Andrea Bolognani wrote:
> > > >> virnetsockettest also fails pretty often for me, certainly
> > > >> more than your figure; even if that wasn't the case, 1/5
> > > >> failure rate is way too high for a CI job.
> > > >
> > > >I played a little more with virnetsockettest to get real stats and
> > > >figured the following:
> > > >
> > > > 1. On my desktop (i5) and laptop (i3), I didn't get any failures in 50
> > > > 'check' runs
> > > > 2. On a VM that I use to run test builds in Jenkins, out of 50 runs it
> > > > fails from 1 to 6 times; I did this test a couple of times and either I
> > > > was lucky or failure rate is higher when my Jenkins perform regular
> > > > builds.
> > > >
> > > >Anyway, I'll try to find a way to debug what's going on with
> > > >virnetsockettest.
> > > >
> > > 
> > > IIRC Debian disabled this test years ago.
> > > 
> > > Guido, have you ever discovered the cause?
> > > 
> > > Jan
> > 
> > I made some experiments on the weekend, and here are my results:
> > 
> > On a box where test fails from time to time, it fails at this point:
> > 
> >     virObjectUnref(csock);
> > 
> >     for (i = 0; i < nlsock; i++) {
> >         if (virNetSocketAccept(lsock[i], &ssock) != -1 && ssock) {
> >             char c = 'a';
> >             if (virNetSocketWrite(ssock, &c, 1) != -1 &&
> >                 virNetSocketRead(ssock, &c, 1) != -1) {
> >                 VIR_DEBUG("Unexpected client socket present"); <--- HERE
> >                 goto cleanup;
> >             }   
> >         }   
> >         virObjectUnref(ssock);
> >         ssock = NULL;
> >     }  
> > 
> > On a box where this test never fails, it reaches this block, but:
> > 
> >  * virNetSocketWrite(ssock, &c, 1) != -1
> >  * virNetSocketRead(ssock, &c, 1) == -1
> > 
> > It's enough to make the test pass. On a failing box both Write() and Read()
> > return != -1 when the test fails.
> 
> We discussed this on IRC and what is happening here is a race condition.
> 
> The test suite is connecting a client to the server and then immediately
> closing the client connection. When the server tries to accept the
> client, usually it'll get -1 because the client has already gone away.
> Socket termination is a multi-stage process at the network layer, and
> so there is a non-zero chance that Accept will succeed. The test suite
> is assuming that if the accept did succeeed, then we'll get I/O error
> on read & write, but this is also not actually guaranteed. It is
> possible that write may suceed buffering output, and read may simply
> see '0' for EOF. When this happens the test suite will fail. Roman
> debugging a failing run & confirmed this is exactly what's happening.
> 
> IOW, this test suite is just plain broken. It needs rewriting to do
> a more sensible real world test. ie spawn a thread to act as the
> server, and have the server just read from the client & echo it
> back to the client. The main thread acts as the client and tests
> this echo'ing. This is race free and is an real-world example of
> usage.

Yeah, I'll try to re-write this test, this weekend hopefully.

Roman Bogorodskiy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20170706/18356c0b/attachment-0001.sig>