[libvirt] [PATCH 00/13] Add multiple trace backend function and add new ftrace backend for libvirt

[Daniel P. Berrange]

On Tue, Apr 01, 2014 at 09:28:18AM +0000, yangzy.fnst at cn.fujitsu.com wrote:
> 
> > -----Original Message-----
> > From: libvir-list-bounces at redhat.com
> > [mailto:libvir-list-bounces at redhat.com] On Behalf Of Daniel P. Berrange
> > Sent: Saturday, March 22, 2014 12:37 AM
> > To: Yang, Zhiyong/杨 志勇
> > Cc: libvir-list at redhat.com; Xinghai Yu
> > Subject: Re: [libvirt] [PATCH 00/13] Add multiple trace backend function
> > and add new ftrace backend for libvirt
> >...
> > If you just want the ability to record a printable formatted string for
> > probes, then I'd much rather just see us improve our logging code, so
> > we can request that the libvirt logging system were able to ouput only
> > log statements associated with probe points. That's be far simpler to
> > do and avoid all the problems with this patchset.
> 
> I sincerely appreciate the analysis and feedback from you to my 
> patch. As you have pointed out, there are a few of drawbacks in the 
> realization of ftrace, such as overhead and the loss of check of 
> systemtap/dtrace, which I think are due to technical reasons and
> which I must contemplate on. It is determined that I take on my
> precious advices and work out a next version of ftrace.
> 
> And on that new start point, it might be advisable to re-assess 
> the integration of multi-tracer framework(ftrace including) into libvirt, 
> which I strongly believe will practically benefit libvirt. 
> 
> A big advantage of using ftrace for libvirt is that we can easily merge 
> all trace data of libvirt, QEMU, and kernel into one memory buffer.
> That makes it really easier for us to investigate a problem. When we 
> worked on a problem, we didn't know in which QEMU or libvirt the
> root cause resided. we needed to look at logging files of both QEMU
> and libvirt in parallel, but it was really hard. The time QEMU and libvirt
> has was different so we can't compare the each line of the log easily.

The timestamp difference is likely to only be related to the timezone
which libvirt always records in UTC, since there's no thread safe way
to log in localtime.

All the comments I had about the proposed ftrace impl in libvirt apply
even more to QEMU, since it is more performance critical. I'm rather
surprised that QEMU accepted the ftrace impl as it is designed currently
since merely turning it on a compile time imposes a permanent performance
burden thereafter.

> As to the dtrace, which is included in libvirt, is more difficult for 
> average users to configurate the D-script. 
> For developer, maybe trace is a powerful, flexible and dynamical tool,
> but in average users' actual occasion, something such as ftrace
> may be more practical.
> 
> The communication with the original author of ftrace has commenced 
> and it is hopeful that the license will soon be legally permitted to me.
> 
> I am looking forward to any exchange of view of whether 
> multi-tracer framework(ftrace including) is necessary.

At the bare minimum there needs to be an impl which doesn't impose a
permanent performance burden once turned on at compile time. There's
also the licensing problems I previously explained. I don't think we
really want to import any of the QEMU tracetool code generator. The
libvirt probe code is already able to generate string format messages
for probe points via our logging APIs. Since ftrace is really nothing
more than a string logging buffer in the kernel, IMHO, it could easily
be done as a libvirt log output. eg so you could turn it on with

  log_filters="2:probe.marker"
  log_outputs="3:ftrace"

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|