[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: Anyone interested in having additional debugging support in NPTLvia a tracing facility?



I think that this would be very useful for anyone who needs to dig into multi-threaded performance/debugging.

Ric

Jerry Harrow wrote:

Sorry for the re-send, but appearently the mail system truncated the previous message after the first paragraph. Hopefully the whole thing comes through this time.

-Jer
=============================================

I've had some on & off discussions with Ulrich about adding detailed
event tracing support to NPTL for quite a while.  While favoring the
idea in general, Ulrich isn't convinced that people (i.e. you all) have
much of a desire for it.  When I last posted a proposal in this regard
(https://listman.redhat.com/archives/phil-list/2002-December/msg00043.html)
there was only one reply -- obviously not an overwhelming
endorsement.  I've done some updates on the original proposal and
published it to
ftp://ftp.compaq.com/pub/products/visualthreads/Linux/NPTLTracingV2.html


Is there any significant interest in having support for this kind of tracing/debug in NPTL?



Let me summarize the concepts:


* Add call-outs at most significant state changes in NPTL. This would
include attempt to lock a mutex, blocking while locking a mutex,
acquiring a mutex, thread creation, thread termination, etc.


* The call-outs are directed to a replaceable shared library which
allows activity to be communicated, or analyzed in whatever manner is
desired.  This is basically an extension of the hooks libthread_db.so
uses to support GDB, but it can invoke locally loaded code to handle the
event.

* A simple text log writer would be provided with NPTL to make it useful
out-of-the-box and as an example of how to use the interfaces.  Another
relatively simple tracing engine that could be built is a shared-memory
ring buffer.  So by just defining an environment variable, you could get
a text file describing all the threading activity of an application.

Potential Concerns some people may have:

* Won't having all these event points in the code cause reduced
performance?


When we built similar capability into the Tru64 UNIX pthreads library it
was done by having a bit test before each event, and it didn't result in
any significant impact -- although in that case our compilers knew how
to move the event code to "cold" sections of the library through
profile-directed optimization.  When HP-UX added similar tracing to its
thread library, it supplied a separate version of libpthread.so which
exactly matches the default library except with tracing support.  In
that case there is no runtime overhead unless LD_LIBRARY_PATH is used to
redirect to the tracing library.

* Is this really useful without writing a bunch of code?

My suggestion is to provide a simple tracing engine with NPTL that
writes out events to a file, so that could be edited/searched with
common tools like sed/awk/perl etc.  Using this it is very easy to find
pthread functions that are failing for unexpected reasons (how many of
you always check the status of pthread_mutex_lock?).  To write a tracing
engine it only takes 4 routines (basically init, fini, re-init on fork,
and event write) so it is very easy to build special-purpose event
handling to help debug a particular problem in your application
(especially when building from an example).   For more details and an
example see the updated proposal.


So if there is interest in this type of facility, I'd be glad to help
put it in place; either in the proposed form or perhaps a simplified
version. The existing proposal is very detailed and almost completely
isolates the trace engine from the libpthread.so implementation details.
Though it may seem complex, the NPTL version of these changes is the
addition of about 81 lines of code plus one module to load the trace
engine, and format the events.


-Jer

=====================================================

Jerry J. Harrow, Jr. Hewlett-Packard Company
110 Spit Brook Rd Nashua, NH 03062-2698 USA
Jerry.Harrow at hp dot com
Telephone: 603.884.2193









[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]