condvar performance in .59 vs .60

[Boris Kolpackov <boris kolpackov net>]

Good day,

I was running some customary tests with different threading 
libraries and noticed significant performance drop once 
switched from nptl v.59 to v.60. Thought you might be interested.

Test consists of three threads and two queues. One thread is 
consumer  and is extracting messages from both queues as fast as 
it can. The other two threads are suppliers (thread per queue) and 
insert messages into the queue as fast as they can.

I have two sub-configurations of this test:

The first is using one pthread_mutex/pthread_cond to protect/signal 
both queues.

The second is using two pthread_mutex'es one for each queue plus 
one twomutex_cond conditional variable which is capable of handling 
two mutexes instead of one (the pseudo-code is provided at the end 
of the posting).

Each supplier is inserting 1M messages (simply int) into a queue. 
Each queue is preallocated before the threads are run.

The test is executed on dual P4 Xeon 2Ghz 1GB running 2.6.0-test5. 

And here is the result (nptl-0.59 vs nptl-0.60 vs LinuxThreads):

Legend: rate is in messages per wakeup (consumer)
        time is in sec:mlsec (so 1:8 means 1sec 800mlsec)
_____________________________________________
         |                 |                 |
         |  pthread_cond   |  twomutex_cond  |
library  |-----------------|-----------------|
         |  rate  |  time  |  rate  |  time  |
_________|________|________|________|________|
         |        |        |        |        |
nptl-59  |  3.5   |  1:8   |  6.2   |  1:8   |
_________|________|________|________|________|
         |        |        |        |        |
nptl-60  |  1.5   | 19:48  | 13.2   |  2:2   |
_________|________|________|________|________|
         |        |        |        |        |
lt       |  1.6   | 18:9   | 13.6   |  2:2   |
_________|________|________|________|________|


First variation of the test runs about 10 times longer on 60 
compared to 59. At the same time decrease in rate is less than
3 times.

hth,
-boris


#include <pthread.h>

typedef struct
{
  unsigned long long int seq_;
  pthread_mutex_t mutex_;
  pthread_cond_t  cond_;
  
} twomutex_cond_t;

int
twomutex_cond_wait (twomutex_cond_t *c,
                    pthread_mutex_t *m1,
                    pthread_mutex_t *m2)
{
  unsigned long long int before, after;

  /* get sequence number before we start unlocking */
  pthread_mutex_lock (&c->mutex_);
  before = c->seq_;
  pthread_mutex_unlock (&c->mutex_);

  pthread_mutex_unlock (m2);

  /* get sequence number after we unlocked all mutexes except the last */
  pthread_mutex_lock (&c->mutex_);
  after = c->seq_;
  pthread_mutex_unlock (&c->mutex_);

  /* check if somebody has signalled while we were busy unlocking */
  if (after > before) return 0;

  /* if not then go to bed */
  pthread_cond_wait (&c->cond_, m1);

  /* note, that m1 is automatically locked by cond_wait */
  pthread_mutex_unlock (m2);

  return 0;
}

/*
  twomutex_cond_signal () assumes that the caller has one of
  the m1 or m2 locked.
  
 */

int
twomutex_cond_signal (twomutex_cond_t *c)
{
  /* advance sequence number */
  pthread_mutex_lock (&c->mutex_);
  ++c->seq_;
  pthread_mutex_unlock (&c->mutex_);
  
  pthread_cond_signal (&c->cond_);
  
  return 0;
}