A trace mechanism for NPTL

[Tony Reix]

Hi Ulrich,

Since it appears that several people (Java JVMs, ...) would like to
have a trace mechanism for NPTL, we propose to work on such a trace
mechanism to be added to the NPTL library.
It will be a free software development and all code will be part of
glibc/nptl and other free software tools.
We have some basic ideas about the features to be provided and how it
could be done, but we need your help for understanding the rules to
follow and for defining the features, requirements and how to build
that trace, in order to make necessary changes acceptable and widely
usable.

Here are the features we are thinking of:

- Users of a NPTL trace:
------------------------
 a) end-users who want to understand why their multi-threaded program
       do not do what they are expecting. (there is a bug in their code!).
       They only need a high-level view of what NPTL is doing (a set
       of NPTL events must be defined).
        
 b) end-users who have checked that their program works perfectly well
       in some other environment (other operating system, or other
       Linux POSIX thread environment, or another NPTL version) and
       need to understand why it behaves differently with NPTL (there
       is a bug in their code that appears only within special
       circumstancies).
        
 c) end-users who are suspecting a complex problem in NPTL and need
       some data for analysing the problem and discuss with people
       maintaining NPTL. They are interested in having more details
       about what NPTL internals are doing and probably which calls
       to the kernel were done.

- NPTL trace integrated with NPTL:
----------------------------------
  - Users of type a) and b) need to be able to quickly use the NPTL
       trace without been required to get and install patches and
       recompile NPTL/glibc and maybe the kernel too.
  - Users of type c) may have skills and time for using a more complex
        tool.
  
- Very light impact to NPTL performance:
----------------------------------------
  - We see 2 solutions:
    - NPTL trace is incorporated into NPTL library and can be switched
      on easily by means of some variable to be tested:
        if(trace) ....
    - If the cost of all these "if(trace)" tests appears to be too
      heavy, then 2 NPTL libpthread.so.0 libs could be delivered:
                one with no trace
                one with trace
  - In both cases, each tracing code added to NPTL source code
        (C and asm) will appear as:
                #ifdef NPTL_TRACING
                if(trace) ....
                #endif
  - Tracing is done in memory and a tool is in charge of saving trace
       data into a file.
    This should reduce the performance impact of NPTL tracing: the
       chance to alter the behaviour of a multi-threaded application
       would be reduced.

- Events to be traced:
----------------------
  - List of Events to be defined:
  	Mutex Events
  	Thread Event
  	Barrier Events
  	Spinlock Events
  	Semaphore Events
  	Condition Variable Events
  	Read-Write Lock Events
  - Each event log should contain:
        date    ThreadId        EventId         Additional Event data

- Visualization of multi-threaded trace:
----------------------------------------
  - A tool will enable the end-user to see what each thread has done.
  - If kernel events have been recorded (with ltt) it is possible to
        see not only which NPTL routines have been called but also
        which kernel calls have been launched by NPTL.
  - We see 2 solutions :
    - Deliver a NPTL-specific trace tool delivered with NPTL.
    - Use a Linux standard trace tool.
       Like: strace, ltrace, or LTT (Linux Trace Toolkit, which
         collects both NPTL and kernel events info).
  - In order to be able to switch to one of the 2 solutions, the
        modification done into NPTL will be independent on the trace
        toolkit been used.


What is your opinion about this proposal ?


Amicalement/Regards,

Tony

                            Carpe Diem