[libvirt] Trying to debug "Received unexpected event 3" from libvirt

[Michal Privoznik]

On 14.01.2016 19:51, Yaniv Kaul wrote:
> On Fri, Jan 8, 2016 at 7:00 PM, Michal Privoznik <mprivozn at redhat.com>
> wrote:
> 
>> > On 30.12.2015 11:30, Yaniv Kaul wrote:
>>> > > Hi,
>>> > >
>> >
>> > Hey,
>> >
>> > sorry for getting to you so late.
>> >
>>> > > I'm trying to debug this issue, which may be affecting my inability to
>>> > > perform live snapshot.
>>> > > 1. I'm not sure what 'Waking up a tragedian" in the debug log means -
>> > what
>>> > > exactly is a tragedian?
>> >
>> >
>> > It's the thread that has issued the change state API (shutdown, reboot,
>> > ..) and it's waiting for confirmation on the monitor. For instance, mgmt
>> > app issues virDomainPMSuspendForDuration() which in qemu driver is
>> > implemented via some agent calls. So the flow is like this:
>> >
>> > 1) the control gets to qemuDomainPMSuspendForDuration()
>> > 2) libvirt does some checks and issues 'guest-suspend-disk' command (or
>> > corresponding command to the selected target).
>> > 3) qemu-ga running inside guest tries (!) to suspend the guest (it may
>> > not necessarily succeed)
>> > 4) meanwhile, as guest is writing something onto disk (saving its RAM -
>> > but one is unable to tell from outside), the libvirt API is blocked
>> > 5) finally, guest kernel calls 'HALT' to which qemu responds by sending
>> > libvirt 'RESET' event
>> > 6) the libvirt event loop finds out that an event occurred on the domain
>> > monitor and calls callback
>> > 7) the callback will wake up the sleeping API if the event the API is
>> > waiting for matches the one obtained on the monitor
>> >
>>> > > 2. In any case, it'd be great if the WARN would mention mon->await_event
>> > -
>>> > > is it the event libvirt is actually waiting for?
>> >
>> > Sure, that would be much more helpful - mind posting a patch?
>> >
> Attached and tested. I can't post it properly, as the git is a bit 'dirty'
> with some po files that I can't clean:
> 
> On branch workbranch
> Your branch is up-to-date with 'origin/master'.
> Changes not staged for commit:
>   (use "git add <file>..." to update what will be committed)
>   (use "git checkout -- <file>..." to discard changes in working directory)
> 
>         modified:   po/ar.po
>         modified:   po/as.po
>         modified:   po/bg.po
>         modified:   po/bn.po
>         modified:   po/bn_IN.po
>         modified:   po/bs.po
>         modified:   po/ca.po
>         modified:   po/cs.po
>         modified:   po/cy.po
>         modified:   po/da.po
>         modified:   po/de.po
>         modified:   po/el.po
>         modified:   po/en_GB.po
>         modified:   po/es.po
>         modified:   po/et.po
>         modified:   po/fi.po
>         modified:   po/fr.po
>         modified:   po/gl.po
>         modified:   po/gu.po
>         modified:   po/he.po
>         modified:   po/hi.po


I guess you were doing 'make dist' or 'make rpm'. Both of them result in
generating of translation strings. Anyway, you can just drop them and
have clean working tree: git checkout po/

> 
> ...
> 
>  Thanks,
> Y.
> 
> 
>>> > >
>>> > > (Both from qemu/qemu_agent.c)
>>> > > 3. I reckon event 3 is QEMU_AGENT_EVENT_RESET ? (from qemu/qemu_agent.h)
>>> > >
>> >
>> > Correct.
>> >
>>> > > 4. I'm also getting 'End of file while reading data: Input/output error'
>>> > > messages, not sure what they mean yet.
>> >
>> >
>> > Usually they mean crashing daemon. If you are able to get a stacktrace
>> > please do share it somewhere.
>> >
>> > Michal
>> >
> 
> 
> qemu_agent.c.diff
> 
> 
> diff --git a/src/qemu/qemu_agent.c b/src/qemu/qemu_agent.c
> index f979f82..924c177 100644
> --- a/src/qemu/qemu_agent.c
> +++ b/src/qemu/qemu_agent.c
> @@ -1241,7 +1241,7 @@ void qemuAgentNotifyEvent(qemuAgentPtr mon,
>          }
>      } else {
>          /* shouldn't happen but one never knows */
> -        VIR_WARN("Received unexpected event %d", event);
> +        VIR_WARN("Received unexpected event %d (expected %d)", event, mon->await_event);
>      }
>  }
>  


ACKed and pushed. I've committed the patch under your name. I hope
that's okay with you.

Michal