[Virtio-fs] [RFC PATCH 0/2] add stat tools for virtiofsd

[Stefan Hajnoczi]

On Thu, Aug 22, 2019 at 10:41:48PM +0800, Gang Deng wrote:
> On 2019/8/22 21:40, Stefan Hajnoczi wrote:
> > On Mon, Aug 19, 2019 at 11:41:12AM +0800, Gang Deng wrote:
> >> There exist two components: vtrace && vstat. vtrace is embeded in virtiofsd,
> >> it will put raw statistics data into share memory. Then the vstat tool could
> >> parse it and do some post processing works. The performance overhead of
> >> vtrace is very small because it does very simple things.
> > 
> > The QEMU source tree already contains support for DTrace/SystemTap,
> > LTTng Userspace Tracer, ftrace, and other tracers via tracetool.  See
> > docs/devel/tracing.txt and scripts/tracetool.py.
> > 
> > It would be good to use that tracing infrastructure instead of writing
> > tracing code from scratch.  Soon someone will want to record FUSE
> > request arguments and other information and then the trace file format
> > and code will become complex and duplicate what tracetool already does.
> > 
> > With tracetool all trace events are defined in a trace-events file
> > (contrib/virtiofsd/trace-events):
> > 
> >   virtiofs_op_begin(int opcode) "opcode 0x%x"
> >   virtiofs_op_end(int opcode, int64_t ns) "opcode 0x%x ns %" PRId64
> > 
> > It would be nice to capture more information: fuse_in_header->unique (to
> > identify the request) and fuse_in_header->nodeid (the inode).
> > 
> > The lowest overhead tracer that tracetool supports is LTTng UST (it uses
> > shared memory) and would be suitable for vstat.
> > 
> > Adding tracetool to virtiofsd will require a little work to verify it
> > works with your tracer of choice (e.g. LTTng UST) despite the process
> > sandboxing, but in the long term I don't think writing tracing code from
> > scratch again makes sense.
> > 
> 
> Thanks for your comments!
> 
> The name of vtrace may be confused. We want the stat tool meets the following
> constraints:
> 
> *). has very small overhead;
> *). can be always on (e.g. detect hang, it's too late to enable trace when hang
>     was already occurred);
> *). can output the latency histogram if needed in future, to detect performance
>     jitter.

If the scope of vstat is well-defined then it may be okay to implement
it from scratch.  My concern is virtiofsd duplicating tracing
infrastructure.  Please rename "trace" in the code so it's clear this is
not a tracing mechanism but specifically just for vstat data collection.

> I think the difference of vtrae && qemu's tracing infrastructure looks like
> that of /proc/diskstats(iostat) && blktrace(blkparse) for linux kernel's block
> device. It depends on the use case. I'll measure the cost later to see whether
> qemu's tracer can meet our needs and whether it's enough to get statistics.

Tracetool supports multiple tracers.  I suggest trying LTTng Userspace
Tracer because it is the lowest overhead tracer (using shared memory).

https://lttng.org/docs/v2.10/#doc-lttng-ust

I don't use LTTng much so I'm not sure how to consume a live trace from
a custom tool like vstat.  I've asked on the LTTng IRC channel and will
let you know what the answer is.

Stefan
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/virtio-fs/attachments/20190823/0c3f5f52/attachment.sig>