[Libguestfs] simplify debugging of guestfsd

[Richard W.M. Jones]

On Fri, Sep 21, 2012 at 02:44:52PM +0200, Olaf Hering wrote:
> 
> Sometimes guestfsd fails to gather info, and the virt-<tool> -v output
> is usually not useful to figure out whats going on within the temporary
> guest. I see the /init script has support to run guestfsd with a debug
> tool, which is currently valgrind.
>
> Granted, valgrind support is a compile time thing. What would be a good
> way to optionally run guestfsd with strace or gdb?

Stepping back a bit, we have a long-open bug about collecting core
dumps from the appliance:

https://bugzilla.redhat.com/show_bug.cgi?id=619334

The problem with collecting core dumps is how you get them out of the
appliance, given that it is short-lived and disappears on exit.  We
looked at various methods including writing them to a block device or
squirting it down a virtio-serial connection, but none of them are
very satisfactory.

strace output can do through the usual console logging mechanism.

gdb seems intractable to me if you want to use gdb interactively.
Just printing stack traces can go via the console logging mechanism.
Note there's very limited development tools in the appliance (no
debuginfo), and I'd be cautious about how much stuff gdb will pull in
via dependencies.  That's why we looked at core dumps.

> Should each tool get a new option, something like 'virt-<tool>
> --debug=strace', or should it be like 'XYZ_STRACE=1 virt-<tool>'?

Adding options to every tool seems to be a lot of work.

The guestfs handle lets you set several useful properties on it using
explicit calls (eg. guestfs_set_attach_method).  However we don't
expose all of these properties in every tool.  Instead we allow the
properties to take default values when the handle is created from
environment variables (eg. $LIBGUESTFS_ATTACH_METHOD), and this means
that every tool doesn't need to support every property.

The '-v' and '-x' options on most tools are convenient, but are
arguably superfluous since people could set LIBGUESTFS_DEBUG=1 and
LIBGUESTFS_TRACE=1 and get almost precisely the same effect.

So environment variables are the way to go.

Another complication is that guestfs_create doesn't have a rich way to
return errors, and thus if the environment variables cannot be parsed
things start to go wrong.  Currently it spits error messages out to
stderr in this case, which is very bad behaviour.

With the benefit of hindsight, splitting handle creation and
environment parsing into two calls would have been better:

  g = guestfs_create ();
  if (!g) error;
  if (guestfs_set_environment (g) == -1) {
    /* with the handle created, richer error methods are now available */
  }

But we cannot change that now without breaking clients.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://et.redhat.com/~rjones/virt-top