[libvirt] xen/libvirt freeze while attching network-device to vm´s - question about provided patches

Tue Sep 20 13:17:17 UTC 2016

Hi Jim,

thanks for your response ans sorry for my late answer,

i´m sorry, i don´t really understand what you are missing in [2], or from where in the log-output you see that an 
shutdown event is missing.

on our test-system the output looks normaly as follow:

2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1719:devices_destroy_cb: forked pid 14987 for destroy of domain 1375
2016-09-20 15:00:38 CEST libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x7fe63c4b9180 wpath=@releaseDomain token=3/0: event epath=@releaseDomain
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1207:domain_death_xswatch_callback: [evg=0x7fe63c3ac070:0] nentries=200 rc=14 0..1388
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c3ac070:0]   got=domaininfos[0] got->domain=0
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback:  exists shutdown_reported=0 dominf.flags=ffff0020
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c1564d0:1374]   got=domaininfos[0] got->domain=0
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c1564d0:1374]   got=domaininfos[1] got->domain=1374
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback:  exists shutdown_reported=0 dominf.flags=ffff0022
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe67c0be740:1375]   got=domaininfos[1] got->domain=1374
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe67c0be740:1375]   got=domaininfos[2] got->domain=1376
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1163:domain_death_occurred: missing from list
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe67c0c4cd0:1376]   got=domaininfos[2] got->domain=1376
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback:  exists shutdown_reported=0 dominf.flags=ffff0012
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c4937e0:1378]   got=domaininfos[2] got->domain=1376
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c4937e0:1378]   got=domaininfos[3] got->domain=1378
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback:  exists shutdown_reported=0 dominf.flags=ffff0012
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe68407e8d0:1379]   got=domaininfos[3] got->domain=1378
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe68407e8d0:1379]   got=domaininfos[4] got->domain=1379
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback:  exists shutdown_reported=0 dominf.flags=ffff0012
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c497090:1380]   got=domaininfos[4] got->domain=1379
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c497090:1380]   got=domaininfos[5] got->domain=1380
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback:  exists shutdown_reported=0 dominf.flags=ffff0012
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6800c1bc0:1381]   got=domaininfos[5] got->domain=1380
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6800c1bc0:1381]   got=domaininfos[6] got->domain=1381
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback:  exists shutdown_reported=0 dominf.flags=ffff0012
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6780a2c90:1382]   got=domaininfos[6] got->domain=1381
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6780a2c90:1382]   got=domaininfos[7] got->domain=1382
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback:  exists shutdown_reported=0 dominf.flags=ffff0022
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c4c5850:1383]   got=domaininfos[7] got->domain=1382
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c4c5850:1383]   got=domaininfos[8] got->domain=1383
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback:  exists shutdown_reported=0 dominf.flags=ffff0012
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe684080480:1384]   got=domaininfos[8] got->domain=1383
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe684080480:1384]   got=domaininfos[9] got->domain=1384
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback:  exists shutdown_reported=0 dominf.flags=ffff0012
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6800b6d50:1385]   got=domaininfos[9] got->domain=1384
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6800b6d50:1385]   got=domaininfos[10] got->domain=1385
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback:  exists shutdown_reported=0 dominf.flags=ffff0012
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6800a7440:1386]   got=domaininfos[10] got->domain=1385
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6800a7440:1386]   got=domaininfos[11] got->domain=1386
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback:  exists shutdown_reported=0 dominf.flags=ffff0002
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6800b74a0:1387]   got=domaininfos[11] got->domain=1386
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe6800b74a0:1387]   got=domaininfos[12] got->domain=1387
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback:  exists shutdown_reported=0 dominf.flags=ffff0022
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c1550d0:1388]   got=domaininfos[12] got->domain=1387
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1218:domain_death_xswatch_callback: [evg=0x7fe63c1550d0:1388]   got=domaininfos[13] got->domain=1388
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1245:domain_death_xswatch_callback:  exists shutdown_reported=0 dominf.flags=ffff0012
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1211:domain_death_xswatch_callback: [evg=0] all reported
2016-09-20 15:00:38 CEST libxl: debug: libxl.c:1274:domain_death_xswatch_callback: domain death search done
2016-09-20 15:00:38 CEST libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x7fe6340145c0: complete, rc=0
2016-09-20 15:00:38 CEST libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x7fe6340145c0: destroy

the output is from an fedora23 with xen-4.6.1 and libvirt-2.1.0 build with the patches, which you provide last year.
And it has done 1400  vm restores, at the moment, that i write here.

all the best
 guido
________________________________________
Von: Jim Fehlig [jfehlig at suse.com]
Gesendet: Freitag, 16. September 2016 22:28
An: Andrea Bolognani; Rossmueller, Guido; libvir-list at redhat.com
Cc: ustermann78 at web.de
Betreff: Re: [libvirt] xen/libvirt freeze while attching network-device to vm´s - question about provided patches

On 09/09/2016 09:43 AM, Andrea Bolognani wrote:
> On Thu, 2016-09-08 at 14:04 +0000, Guido.Rossmueller at gdata.de wrote:
>> Hello everybody,
>>
>> a colleague of me described last november a problem that we have with libvirt and xen
>>
>> https://www.redhat.com/archives/libvir-list/2015-November/msg00130.html
>>
>> Jim Fehlig provided for this problem a patch a few days later:
>>
>> https://www.redhat.com/archives/libvir-list/2015-November/msg00520.html
>>
>> The provided patch work great and we have no problems until we applied them..
>>
>> Now we a in the situation that we want to migrate to libvirt 2.1, and we saw the same
>> problem that we have last year, again.
>> After we take a look in the source code, we determine that the patches are didn´t go
>> in the source code. So we modify the patches in that way, that we could apply them
>> again, and all looks good.
>>
>> My question, is their any special reason why this patches didn´t go in the source code?
>>
>> I see same posts on the maillinglist, but absolute statement for or again the patches
>> from Jim Fehlig.
> Jim said he would submit a v2 to address some concerns raised during
> review[1]; later, he reported some issues he'd run into[2] while
> working on it.

I had planned to wait a few days to see if Ian had any thoughts on the libxl
side. I'm not familiar with the code and would need some time to understand it
before debugging further. But I've never got back to that work :-(.

>
> So it looks to me like the code was not yet ready for merge, which
> would explain it being tagged as RFC. And then it probably just
> slipped through the cracks :)

Sort of. It is an item that is fermenting on my todo list.

>
> CCing Jim for insights on the current status of the patch.

I still have a branch in my local git tree :-). It needs rebased and tested
against Xen 4.7 (and probably xen.git master).

>
>
> [1] https://www.redhat.com/archives/libvir-list/2015-November/msg00958.html
> [2] https://www.redhat.com/archives/libvir-list/2015-November/msg00974.html

Guido,

With the rebased version of the patch, do you notice the issue described in [2]
during your testing? To trigger it, I used a hacky script that created,
shutdown, destroyed, saved, and restored many domains in parallel.

What version of Xen are you using?

Regards,
Jim

____________
Virus checked by G Data MailSecurity
Version: AVA 25.8356 dated 20.09.2016
Virus news: www.antiviruslab.com