[libvirt] Seeking for advice on 'virsh console' problems debugging

[Daniel P. Berrange]

On Fri, Mar 07, 2014 at 10:04:48PM +0400, Roman Bogorodskiy wrote:
>   Daniel P. Berrange wrote:
> 
> > On Fri, Mar 07, 2014 at 07:49:04PM +0400, Roman Bogorodskiy wrote:
> > >   Daniel P. Berrange wrote:
> > > 
> > > > On Fri, Mar 07, 2014 at 06:15:24PM +0400, Roman Bogorodskiy wrote:
> > > > >   Daniel P. Berrange wrote:
> > > > > 
> > > > > > All those mangled messages are being printed out by libvirt client
> > > > > > code, and it is totally expected that log messages libvirt prints
> > > > > > are mangled in this way for the console.
> > > > > > 
> > > > > > What's more important is what data coming from the guest looks
> > > > > > like.
> > > > > 
> > > > > All those spaces come from the stream as well. For example, I've added
> > > > > the following debugging to daemonStreamHandleRead() into
> > > > > daemon/stream.c:
> > > > > 
> > > > > 732     ret = virStreamRecv(stream->st, buffer, bufferLen);
> > > > > 733     VIR_ERROR("buffer = [%s]", buffer);
> > > > > 
> > > > > And it prints stuff like:
> > > > > 
> > > > > 2014-03-07 13:29:34.060+0000: 34489787392: error :
> > > > > daemonStreamHandleRead:733 : buffer = [  File "/usr/bin/cloud    raise
> > > > > DataSourceNotFoundException(msg)
> > > > > ]
> > > > > 2014-03-07 13:29:34.060+0000: 34489787392: error :
> > > > > daemonStreamHandleRead:733 : buffer = [
> > > > > ]
> > > > > 2014-03-07 13:29:34.061+0000: 34489787392: error :
> > > > > daemonStreamHandleRead:733 : buffer = [
> > > > > ]
> > > > > 2014-03-07 13:29:34.061+0000: 34489787392: error :
> > > > > daemonStreamHandleRead:733 : buffer = [
> > > > > ]
> > > > 
> > > > The debug logs are *always* going to be strange when you have the
> > > > terminal in raw mode. If you're debugging the console support you
> > > > really must not try to use stderr for debug logs - you'll never get
> > > > sane results. Set libvirt to log to a file or syslog instead.
> > > 
> > > That's strange because libvirtd and virsh use different terminals for
> > > logging, but I've disabled stderr logging to be on the safe side.
> > > 
> > > Anyway, I've conducted an experiment like you suggested:
> > > 
> > > 1. run 'virsh' on linux to freebsd libvirtd, output is mangled:
> > > 
> > >  http://people.freebsd.org/~novel/misc/ttyweirdness2.png
> > > 
> > >  2. run 'virsh' on freebsd to linux libvirtd, output is good:
> > > 
> > >  http://people.freebsd.org/~novel/misc/ttyweirdness3.png
> > > 
> > > Guest uses the same image and the same domain xml.
> > > 
> > > Also, I removed all my custom debug stuff from virsh except mutex lock
> > > (as it doesn't work otherwise). So the problem seem to be on the
> > > libvirtd side...
> > 
> > Agreed, it is either libvirtd or bhyve that's the source of the
> > mangling then.
> 
> Very unlikely that it's bhyve, as I'm testing with qemu (plan to get to
> bhyve when I'll be able to get it fixed for qemu).

Oh, I didn't realize you were testing this with QEMU on BSD!

One further idea I just had - in QEMU code util/qemu-openpty.c we call
cfmakeraw()/tcsetattr on the PTY slave file descriptor. On Linux at 
least this affects both ends of the device, but I wonder if there's
perhaps a need to call this on the master file descriptor as well as
or instead of on the slave file descriptor ?  Or perhaps the client
(in this case libvirtd) itself has to call cfmakeraw/tcsetattr since
although its opening the same slave, it is a different file descriptor
for the slave

> And considering that 'cu' works properly, it's most likely the daemon.
> I'll continue looking in this direction. Thanks for the pointer, that
> helped to narrow down the problem indeed.

True, but interesting to strace 'cu' and libvirtd to see what differences
they have if any when opening the slave FD.

> 
> Roman Bogorodskiy



Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|