<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="font-size: 14px; color: rgb(0, 0, 0);"><font face="Arial">Hi Folks,</font></div>
<div style="font-size: 14px; color: rgb(0, 0, 0);"><font face="Arial"><br>
</font></div>
<div style="font-size: 14px; color: rgb(0, 0, 0);"><font face="Arial">Using libvirt python bindings we are creating an lxc container.Here is the problem that we see sometimes (say 20 % of the time) when we create a new container.</font></div>
<div style="font-size: 14px; color: rgb(0, 0, 0);"><font face="Arial"><br>
</font></div>
<div style="font-size: 14px; color: rgb(0, 0, 0);"><font face="Arial">1.container gets created, and also starts.However the we are not able to enter the namepace of the container.It throws an error initPid not available.We see that the using netstat command
, socket connection is closed.</font></div>
<div style="font-size: 14px; color: rgb(0, 0, 0);"><font face="Arial">2.To get around this problem we have to stop and start the container again.We see that socket under (/var/run/libvirt/*) connection is established and we are able to enter the namespace.</font></div>
<div style="font-size: 14px; color: rgb(0, 0, 0);"><font face="Arial"><br>
</font></div>
<div style="font-size: 14px; color: rgb(0, 0, 0);"><font face="Arial">Enabled the libvirtd debug logs to debug this issue.</font></div>
<div style="font-size: 14px; color: rgb(0, 0, 0);"><font face="Arial"><br>
</font></div>
<div style="font-size: 14px; color: rgb(0, 0, 0);"><font face="Arial">For <b>success</b> case we see that for new client connection gets created and is able to handle async incoming events, </font></div>
<div style="font-size: 14px; color: rgb(0, 0, 0);"><font face="Arial"><br>
</font></div>
<div style="font-size: 14px;">
<div><i><font color="#007f00" face="Arial">2016-03-12 08:18:55.748+0000: 1247: info : virObjectNew:202 : OBJECT_NEW: obj=0x7fed54005460 classname=virLXCMonitor</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:18:55.748+0000: 1247: debug : virNetSocketNew:159 : localAddr=0x7fed7cd1d170 remoteAddr=0x7fed7cd1d200 fd=28 errfd=-1 pid=0</font></i></div>
<div><i><font color="#007f00" face="Arial">2016-03-12 08:18:55.749+0000: 1247: info : virObjectNew:202 : OBJECT_NEW: obj=0x7fed54009040 classname=virNetSocket</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:18:55.749+0000: 1247: info : virNetSocketNew:209 : RPC_SOCKET_NEW: sock=0x7fed54009040 fd=28 errfd=-1 pid=0 localAddr=127.0.0.1;0, remoteAddr=127.0.0.1;0</font></i></div>
<div><i><font color="#007f00" face="Arial">2016-03-12 08:18:55.749+0000: 1247: info : virObjectNew:202 : OBJECT_NEW: obj=0x7fed54009d10 classname=virNetClient</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:18:55.749+0000: 1247: info : virNetClientNew:327 : RPC_CLIENT_NEW: client=0x7fed54009d10 sock=0x7fed54009040</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:18:55.749+0000: 1247: info : virObjectRef:296 : OBJECT_REF: obj=0x7fed54009d10</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:18:55.749+0000: 1247: info : virObjectRef:296 : OBJECT_REF: obj=0x7fed54009040</font></i></div>
<div><font color="#007f00" face="Arial"><i>2016-03-12 08:18:55.750+0000: 1247: info : virObjectNew:202 : OBJECT_NEW: obj=0x7fed540009a0 classname=virNetClientProgram</i></font></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:18:55.750+0000: 1247: info : virObjectRef:296 : OBJECT_REF: obj=0x7fed540009a0</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:18:55.750+0000: 1247: info : virObjectRef:296 : OBJECT_REF: obj=0x7fed54005460</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:18:55.750+0000: 1247: info : virObjectUnref:259 : OBJECT_UNREF: obj=0x7fed5c168eb0</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:18:55.750+0000: 1247: debug : virLXCProcessCleanInterfaces:475 : Cleared net names: eth0 </font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:18:55.750+0000: 1247: info : virObjectUnref:259 : OBJECT_UNREF: obj=0x7fed5c168eb0</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:18:55.750+0000: 1247: info : virObjectUnref:259 : OBJECT_UNREF: obj=0x7fed5c169600</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:18:55.755+0000: 1244: debug : virNetClientIncomingEvent:1808 : client=0x7fed54009d10 wantclose=0</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:18:55.755+0000: 1244: debug : virNetClientIncomingEvent:1816 : Event fired 0x7fed54009040 1</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:18:55.755+0000: 1244: debug : virNetMessageDecodeLength:151 : Got length, now need 36 total (32 more)</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:18:55.756+0000: 1244: info : virNetClientCallDispatch:1116 : RPC_CLIENT_MSG_RX: client=0x7fed54009d10 len=36 prog=305402420 vers=1 proc=2 type=2 status=0 serial=1</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:18:55.756+0000: 1244: debug : virKeepAliveCheckMessage:377 : ka=(nil), client=0x7fed81fc5ed4, msg=0x7fed54009d78</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:18:55.756+0000: 1244: debug : virNetClientProgramDispatch:220 : prog=305402420 ver=1 type=2 status=0 serial=1 proc=2</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:18:55.756+0000: 1244: debug : virLXCMonitorHandleEventInit:109 : Event init 1420 </font></i></div>
</div>
<div style="font-size: 14px; color: rgb(0, 0, 0);"><font face="Arial"><br>
</font></div>
<div style="font-size: 14px; color: rgb(0, 0, 0);"><font face="Arial">For <b>failure</b> case ,we see that the client socket connection is initiated and gets closed immediately after receiving an incoming event.In this case, I don’t see an object for virNetClientProgram
being created.</font></div>
<div style="font-size: 14px; color: rgb(0, 0, 0);"><font face="Arial">Incoming event comes in and since the its unable to find client->prog it bails out and closes the connection.</font></div>
<div style="font-size: 14px; color: rgb(0, 0, 0);"><font face="Arial">Snaphost of the code,</font></div>
<div style="font-size: 14px; color: rgb(0, 0, 0);"><font face="Arial"><br>
</font></div>
<div style="font-size: 14px;">
<div style="color: rgb(0, 0, 0);"><font face="Arial">static int virNetClientCallDispatchMessage(virNetClientPtr client)</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Arial">{</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Arial"> size_t i;</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Arial"> virNetClientProgramPtr prog = NULL;</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Arial"><br>
</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Arial"> for (i = 0; i < client->nprograms; i++) {</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Arial"> if (virNetClientProgramMatches(client->programs[i],</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Arial"> &client->msg)) {</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Arial"> prog = client->programs[i];</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Arial"> break;</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Arial"> }</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Arial"> } </font></div>
<div style="color: rgb(0, 0, 0);"><font face="Arial"> if (!prog) {</font></div>
<div><b><font color="#ff0000" face="Arial"> VIR_DEBUG("No program found for event with prog=%d vers=%d",</font></b></div>
<div><b><font color="#ff0000" face="Arial"> client->msg.header.prog, client->msg.header.vers);</font></b></div>
<div style="color: rgb(0, 0, 0);"><font face="Arial"> return -1;</font></div>
<div style="color: rgb(0, 0, 0);"><font face="Arial"> } </font></div>
</div>
<div style="font-size: 14px; color: rgb(0, 0, 0);"><font face="Arial"><br>
</font></div>
<div style="font-size: 14px;">
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:19:15.935+0000: 1246: info : virObjectRef:296 : OBJECT_REF: obj=0x7fed5c168eb0</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:19:15.935+0000: 1246: info : virObjectRef:296 : OBJECT_REF: obj=0x7fed82bd7bc0</font></i></div>
<div><i><font color="#ff0000" face="Arial">2016-03-12 08:19:15.935+0000: 1246: info : virObjectNew:202 : OBJECT_NEW: obj=0x7fed82bd8120 classname=virLXCMonitor</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:19:15.935+0000: 1246: debug : virNetSocketNew:159 : localAddr=0x7fed7d51e170 remoteAddr=0x7fed7d51e200 fd=31 errfd=-1 pid=0</font></i></div>
<div><i><font color="#ff0000" face="Arial">2016-03-12 08:19:15.936+0000: 1246: info : virObjectNew:202 : OBJECT_NEW: obj=0x7fed82bd8660 classname=virNetSocket</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:19:15.936+0000: 1246: info : virNetSocketNew:209 : RPC_SOCKET_NEW: sock=0x7fed82bd8660 fd=31 errfd=-1 pid=0 localAddr=127.0.0.1;0, remoteAddr=127.0.0.1;0</font></i></div>
<div><i><font color="#ff0000" face="Arial">2016-03-12 08:19:15.936+0000: 1246: info : virObjectNew:202 : OBJECT_NEW: obj=0x7fed82bd8ca0 classname=virNetClient</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:19:15.936+0000: 1246: info : virNetClientNew:327 : RPC_CLIENT_NEW: client=0x7fed82bd8ca0 sock=0x7fed82bd8660</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:19:15.936+0000: 1246: info : virObjectRef:296 : OBJECT_REF: obj=0x7fed82bd8ca0</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:19:15.936+0000: 1246: info : virObjectRef:296 : OBJECT_REF: obj=0x7fed82bd8660</font></i></div>
<div><i><font face="Arial" color="#ff0000"><b>2016-03-12 08:19:15.942+0000: 1244: debug : virNetClientIncomingEvent:1808 : client=0x7fed82bd8ca0 wantclose=0</b></font></i></div>
<div><i><font face="Arial" color="#ff0000"><b>2016-03-12 08:19:15.942+0000: 1244: debug : virNetClientIncomingEvent:1816 : Event fired 0x7fed82bd8660 1</b></font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:19:15.942+0000: 1244: debug : virNetMessageDecodeLength:151 : Got length, now need 36 total (32 more)</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:19:15.942+0000: 1244: info : virNetClientCallDispatch:1116 : RPC_CLIENT_MSG_RX: client=0x7fed82bd8ca0 len=36 prog=305402420 vers=1 proc=2 type=2 status=0 serial=1</font></i></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:19:15.942+0000: 1244: debug : virKeepAliveCheckMessage:377 : ka=(nil), client=0x7fed81fc5ed4, msg=0x7fed82bd8d08</font></i></div>
<div><b><font color="#ff0000" face="Arial"><i>2016-03-12 08:19:15.942+0000: 1244: debug : virNetClientCallDispatchMessage:1008 : No program found for event with prog=305402420 vers=1</i></font></b></div>
<div style="color: rgb(0, 0, 0);"><i><font face="Arial">2016-03-12 08:19:15.942+0000: 1244: debug : virNetMessageClear:57 : msg=0x7fed82bd8d08 nfds=0</font></i></div>
<div><i><font face="Arial" color="#ff0000"><b>2016-03-12 08:19:15.942+0000: 1244: debug : virNetClientMarkClose:632 : client=0x7fed82bd8ca0, reason=0</b></font></i></div>
<div><i><font face="Arial" color="#ff0000"><b>2016-03-12 08:19:15.942+0000: 1244: debug : virNetClientCloseLocked:648 : client=0x7fed82bd8ca0, sock=0x7fed82bd8660, reason=0</b></font></i></div>
</div>
<font face="Arial"><br>
</font>
<div style="font-size: 14px;"><font face="Arial"><br>
</font></div>
<div style="font-size: 14px;"><font face="Arial">Here is the snapshot of code ,</font></div>
<div style="font-size: 14px;"><font face="Arial"><br>
</font></div>
<div style="font-size: 14px;">
<div><font face="Arial">virLXCMonitorPtr virLXCMonitorNew(virDomainObjPtr vm, </font></div>
<div><font face="Arial"> const char *socketdir,</font></div>
<div><font face="Arial"> virLXCMonitorCallbacksPtr cb) </font></div>
<div><font face="Arial">{</font></div>
<div><font face="Arial"> virLXCMonitorPtr mon;</font></div>
<div><font face="Arial"> char *sockpath = NULL;</font></div>
<div><font face="Arial"><br>
</font></div>
<div><font face="Arial"> if (virLXCMonitorInitialize() < 0)</font></div>
<div><font face="Arial"> return NULL;</font></div>
<div><font face="Arial"><br>
</font></div>
<div><font face="Arial"> if (!(mon = virObjectLockableNew(virLXCMonitorClass)))</font></div>
<div><font face="Arial"> return NULL;</font></div>
<div><font face="Arial"><br>
</font></div>
<div><font face="Arial"> if (virAsprintf(&sockpath, "%s/%s.sock",</font></div>
<div><font face="Arial"> socketdir, vm->def->name) < 0)</font></div>
<div><font face="Arial"> goto error;</font></div>
<div><font face="Arial"><br>
</font></div>
<div><font face="Arial"> if (!(mon->client = virNetClientNewUNIX(sockpath, false, NULL)))</font></div>
<div><font face="Arial"> goto error;</font></div>
<div><font face="Arial"><br>
</font></div>
<div><font face="Arial"> <font color="#7f0000"><b> if (</b></font><b style="color: rgb(127, 0, 0);">virNetClientRegisterAsyncIO</b><font color="#7f0000"><b>(mon->client) < 0)</b></font></font></div>
<div><font color="#7f0000" face="Arial"><b> goto error;</b></font></div>
<div><font color="#7f0000" face="Arial"><b><br>
</b></font></div>
<div><font color="#7f0000" face="Arial"><b> if (!(mon->program = virNetClientProgramNew(VIR_LXC_MONITOR_PROGRAM,</b></font></div>
<div><font color="#7f0000" face="Arial"><b> VIR_LXC_MONITOR_PROGRAM_VERSION,</b></font></div>
<div><font color="#7f0000" face="Arial"><b> virLXCMonitorEvents,</b></font></div>
<div><font color="#7f0000" face="Arial"><b> ARRAY_CARDINALITY(virLXCMonitorEvents),</b></font></div>
<div><font color="#7f0000" face="Arial"><b> mon)))</b></font></div>
<div><font color="#7f0000" face="Arial"><b> goto error;</b></font></div>
<div><font color="#7f0000" face="Arial"><b><br>
</b></font></div>
<div><font face="Arial"><font color="#7f0000"><b> if (</b></font><b style="color: rgb(127, 0, 0);">virNetClientAddProgram</b><font color="#7f0000"><b>(mon->client,</b></font></font></div>
<div><font color="#7f0000" face="Arial"><b> mon->program) < 0)</b></font></div>
<div><font color="#7f0000" face="Arial"><b> goto error;</b></font></div>
<div><font color="#7f0000" face="Arial"><b><br>
</b></font></div>
<div><font color="#7f0000" face="Arial"><b> mon->vm = vm;</b></font></div>
<div><font color="#7f0000" face="Arial"><b> memcpy(&mon->cb, cb, sizeof(mon->cb));</b></font></div>
<div><font face="Arial"><br>
</font></div>
<div><font face="Arial"> virObjectRef(mon);</font></div>
<div><font face="Arial"> virNetClientSetCloseCallback(mon->client, virLXCMonitorEOFNotify, mon,</font></div>
<div><font face="Arial"> virLXCMonitorCloseFreeCallback);</font></div>
</div>
<div style="font-size: 14px;"><font face="Arial"><br>
</font></div>
<div style="font-size: 14px;"><font face="Arial">Is the problem occurring due to invocation of “<b>virNetClientRegisterAsyncIO" api before the </b><b>virNetClientAddProgram.Probably once we register for aysnc IO , immediately an event comes in and that thread
takes priority and bails out since it does not find the client->prog?Also the client is not retrying to establish a new connection.</b></font></div>
<div style="font-size: 14px;"><b><font face="Arial"><br>
</font></b></div>
<div><b><font face="Arial">Please let me any thoughts/comments.Is there any patch already available which has fixed this issue?</font></b></div>
<div><b><font face="Arial"><br>
</font></b></div>
<div><b><font face="Arial">-Thanks,</font></b></div>
<div><b><font face="Arial">Rammohan</font></b></div>
<div style="font-size: 14px;"><font face="Arial"><br>
</font></div>
<div style="font-family: Calibri, sans-serif; font-size: 14px;"><br>
</div>
</body>
</html>