[libvirt] Seeking for advice on 'virsh console' problems debugging

[Daniel P. Berrange]

On Fri, Mar 07, 2014 at 05:49:34PM +0400, Roman Bogorodskiy wrote:
> Hi,
> 
> I've been trying to understand why 'virsh console' doesn't work on
> FreeBSD for some time and now I feel like I got stuck.
> 
> Background of the problem. Some time ago I started implementing console
> support for the bhyve driver. I initially implemented it using nmdm(4)
> device:
> 
> http://www.freebsd.org/cgi/man.cgi?query=nmdm&sektion=4&manpath=FreeBSD+10.0-stable
> 
> But 'virsh console' didn't work on it and exited immediately.
> 
> I though that it could be nmdm compatibility issue and decided to go
> with the plan /dev/pts devices. But I got the same problem with it.
> 
> So I decided to try use console with qemu driver. And... I got the same
> problem.
> 
> After some investigation, I spotted this block in tools/virsh-console.c:376
> 
>     while (!con->quit) {
>         if (virCondWait(&con->cond, &con->lock) < 0) {
>             VIR_ERROR(_("unable to wait on console condition"));
>             goto cleanup;
>         }
>     }
> 
> The strange thing is that it doesn't look con->lock mutex before
> waiting. 
> 
> FreeBSD man page for pthread_cond_wait() says that it errors in that
> situation:
> 
>      [EPERM]            The specified mutex was not locked by the
>      calling thread.
> 
> I replaced VIR_ERROR with virReportSystemError and it returns EPERM
> indeed. So I've added virMutexLock() before calling virCondWait() and it
> partially solved the problem for me -- 'virsh console' stopped exiting
> immediately.
> 
> Actually, it started output for guest, but it's badly formatted: it
> includes lots and lots of whitespaces. Input doesn't work properly
> (cannot type in sensible stuff).
> 
> Here's a sample output, even libvirt messages are badly formatted (I
> added some debug prints along the way).
> 
> http://people.freebsd.org/~novel/misc/ttyweirdness.png

All those mangled messages are being printed out by libvirt client
code, and it is totally expected that log messages libvirt prints
are mangled in this way for the console.

What's more important is what data coming from the guest looks
like.

> So, e.g. dmesg output for the guest looks almost the same.
> 
> Looks like there's something wrong with the terminal settings, but how
> could that be debugged?

The virsh console works remotely. So it would be interesting to run
virsh console on linux, connecting to a FreeBSD hosted guest and see
if it is still a problem. That would help determine whether virsh
console is at fault, or the bhyve side is at fault.

Similarly, try virsh console running on FreeBSD, connecting to a
guest that is running with KVM on Linux. Again that would help
narrow down which part is at fault.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|