Notes 20081204 utrace meeting

[Jim Keniston]

On Thu, 2008-12-04 at 22:50 -0500, Elena Zannoni wrote:
> All,
> we had a phone meeting about utrace and plans for utrace.
> Here are my notes, from the meeting. They are a bit scattered, but I
> think they can serve as a thread for Roland to prepare a more
> comprehensive email with all the details. This wa just a "let's regroup"
> type of discussion, real technical details will be in email in the next
> couple of weeks.
> 
> Jim will send out his notes as well.

Elena's notes are pretty thorough.  I've added a few things below.
Jim

> 
> Feel free to correct, I am sure I missed a few things, with
> the static noises and all.
> 
> elena
> 
> 
> -----------------
> 
> 
> 
> Notes: utrace meeting 20081204
> 
> development code, bleeding edge, still has some bugs, not in any
> fedora builds, maybe its in rawhide.
> 
> instability, race conditions, we don't know if those are still around
> now
> 
> there is only one place where the fatc code path is affected by the
> issue of data struct being a separate pointer or not only issue id the
> size of the structure, but not a big deal. it's a change that somebody
> could make, there is a private data structure that should move to
> header files. Could be done by anybody. Not a priority for Roland
> himself to do.
> 
> git tree status
> 
> 2.6.28-rc7 patches. Updated December 2. See the git tree. git tree ==
> linus' latest tree + utrace patches.
> 
> 2.6.27 there are 2 patches for
> ia64-> do not use, not tested
> i386-> ok to use
> 
> makefile patch, one word change (?)
> 
> 
> upstream:
> 
> architecture support is upstream now, maintenance is with the
> architecture maintainers.
> 
> new api: not upstream yet, kernel folks want a user to be integrated
> as well.
> 
> We need something on top of utrace.
> maybe something with ftrace, or froggy.
> 
> 
> Roland: write up technical details about utrace.
> technical roadmap
> goal: posted by the end of next week.
> regular commitment for a discussion from a utrace component POV,
> instead of stap.
> 
> 
> systemtap
> task finding
> get all threads
> get al children and keep trac of those
> what you exec
> 
> modular vision, bottom up
> 
> trace processes that exec foo()
> trace all the children of this process
> and the like

In other words, Roland envisions (among other things) a more generally
available API that subsumes what stap's task_finder currently does.
> 
> should be in components that can be reused by stap, gdb and any
> userlevel or kernel interfaces
> 
> collection of utrace engines

(for related tasks, such as all threads in a process)

> : general framework built around the async
> model of utrace extend idea of event notification to have rule sets
> for identifying, filtering and reacting to even sources and also

s/even sources/event sources/

> treating groups of threads alike, having a set of rules that is
> associated with a group. How a group treats all of the threads.
> 
> canonical rules set, that trace all the childresn, that traces all the
> threads, that filters by exec name, etc.
> 
> Something as a framework that just has idea of threads gorups and rule
> sets. It should be extensible with the kind of filetering rules and
> actions triggered that you can have.
> 
> All the basic group management stuff and the basic utrace calls and
> then everything else could be built up from the basic stuff.
> 
> Idea of synthetic kinds of events: new tracing sources can generate
> new types of events.
> 
> hw bp, watchpoint, branch tracing stuff, perfmon, etc

... i.e., looking beyond just existing utrace and uprobes for the types
of events we can respond to.

> 
> have this framework be dynamically extensible via kernel modules
> registering new kinds of rules and events and maybe there will be
> specific hardware events built into the kernel
> 
> then on that i would build the bp mechanism, for different kind of
> uses, not all the pieces want to be in the kernel as opposed to user
> space
> 
> ubp is more in the manner that roland has in mind but he would like it
> to be sliced differently
> 
> do you have always some kind of kernel storage for bp or
> not. userlevel debugging: you need people to insert a lot of bps while
> not consuming any special kernel resources. So piece to be sliced out
> and get an interface is a purely instruction analysys and step(?)
> piece, architecture specific.

ubp could already be characterized as instruction analysis +
single-stepping.  I think Roland envisions slicing that thinner, so that
instruction analysis is its own layer (service?).  Something with a
ubp-like API could be a client of the instruction-analysis service.

(BTW, I believe there's nothing in the current ubp implementation that
prevents the "user-breakpoint" objects (struct ubp or equivalent info)
from being stored and maintained exclusively in user space.  That was a
design goal.)

> could be reused by kprobes, and
> breakpoints and reused outside of the kernel
> 
> hardware single step,
> instruction copying + trap
> ....
> 
> all the different way different hardware does stepping/breakpoints/whatnot

This was in answer to the question as to whether single-stepping out of
line (SSOL) should be the focus for instruction analysis.  Some CPU
architecture have no single-step capability.  E.g., you can get the
effect of single-stepping (after a breakpoint trap) by pointing the IP
at  instruction sequence consisting of a copy of the probed instruction
followed by a trap instruction.  In other cases, there are instruction
sequences where you can't single-step individual instructions and still
get the desired effect; you might have to execute the whole basic block
as a unit.

> 
> gdb, perfmon2/3, kvm, ftrace, djprobes -> all have analysis of instructions

For now, start with what's needed for SSOL, and add other analyses as we
go.

> 
> let's have just one instance of this code, so that not every tool
> has to reinvent the wheel, and maintain all the complexity. -> main idea
> 
> How the code will be sliced, and where it will physically be, is
> different matter.
> 

Is the utrace change history available in the utrace git?  Not the way
Roland currently uses git.  But essentially all the changes now are in
kernel/utrace.c.  So something like
	git diff <commit-hex1> <commit-hex2> kernel/utrace.c
should provide that.

Jim