[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

RE: Poor thread performance on Linux vs. Solaris

> From: Bill Soudan [mailto:bsoudan brass com]

> > It'd be interesting to verify if there is still a lot of contention in
> > the locking.
> oprofile results on 2.6.0-0.test4.1.32 with Boris' futex_q_lock patch:
> ...
> c011e270 4046     3.25066     kunmap_atomic
> c0147bf0 4628     3.71825     scan_poisoned_obj
> c011e1e0 4955     3.98097     kmap_atomic
> c0138100 24115    19.3746     do_futex
> c0138bcd 46827    37.622      .text.lock.futex
> > A question about your app: are there many different locks used at the
> > same time or the number is actually low?
> Many.  I'd estimate at least 15 on the typical code path (which I'm
> exercising with the performance test).  However, I'm not the original
> architect so I'm not certain, but it definitely isn't  a low number like
> 2 or 3.

I wanted to get a feel on how could be the distribution on the hash
buckets. If many of them happen to be on the same bucket [I guess
it would be kind of difficult--as in winning the $140M lottery], the
thing would not be solved, but as Ingo pointed out, boosting up hashbits
would get rid that using Boris' patch.

Still I think the coulprit are vcache_lock and current->mm->page_table_lock.

> > No matter how we do it, there is still a bunch of lock acquisition in
> > the futex code that would serialize access [specially if they are locks
> > within the same thread, like current->mm->page_table_lock].
> >
> > And then we also have vcache_lock--this one is also serializing
> > everything, as it is taken unconditionally whenever we call
> > lock_futex_mm().
> Oh! Hits in .text.lock.futex represent spinning for any of the locks
> acquired in futex.c, yes?  Originally I thought it represented time spent
> spinning while waiting for the 'futex' lock in 'futex.c'.  But's that's
> obviously gone after Boris' patch.

Thinking about it twice...I think it need not be necessarily--I just did
some silly tests and I get something very different.

If you apply this patch [warning, breaks compilation]

--- include/linux/spinlock.h	10 Jul 2003 19:27:31 -0000
+++ include/linux/spinlock.h	8 Sep 2003 21:15:38 -0000
@@ -20,7 +20,7 @@
  * Must define these before including other files, inline functions need them
 #define LOCK_SECTION_NAME			\
-	".text.lock." __stringify(KBUILD_BASENAME)
+	".text.lock." __stringify(KBUILD_BASENAME) "." __FILE__ "." __stringify(__LINE__)
 #define LOCK_SECTION_START(extra)		\
 	".subsection 1\n\t"			\

and build with "make kernel/futex.s"

You see that the actual two lock sections come from the semaphores for
the mm semaphore (mm->mmap_sem) in __pin_page(). Maybe it is also my
.config, but finally, even when static, the spin lock calls in the
asm code get transformed into a call to __preempt_spin_lock(), than
then would show with a different text section (under .text.lock.sched
I think).

> Now - how to distinguish between these locks in the oprofile results?  Can
> I add a few helper files that just contain functions to acquire/release
> one lock (assuming one .text.lock section is created per file)?  For
> instance, futex_vcache_lock.c, futex_page_table_lock.c, etc?

That is what I was trying to do with that patch; unfortunately, the __FILE__
breaks it beyond repair because the assembler does not accept the slash for
a section name [not that I blame it :)]--anyone knows some CPP voodoo for
fixing that? 

But still, I am kind of sure that those are the ones; if you take away the
__FILE__ thing, the line number stays, and it maps pretty well to the one
who defined the lock. Can you generate your futex.s and send it to me?

Still, if it is the __pin_page() one, I don't know what can be done. It is
strange though, that it would have to walk that much to get a page, after
a while they should stay in... or maybe not.

A lot of VM used in the app [like with a lot of trashin?]

Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own (and my fault)

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]