[Spacewalk-list] Massive problems with slow updates on rhnServerAction

[Patrick Hurrelmann]

Hi List,

since some weeks my SW 1.7 on CentOS 6.3 is grinding to halt regularly
and it is getting worse from day to day. Right now I have to restart it
several times a day. The db connections to PostgreSQL fail with "FATAL:
 sorry, too many clients already". The max connections were already
bumped several times and is set to 300 currently. But thats not the real
problem, it seems.

I tried to track it down and and stumbled over frequent updates on the
table rhnServerAction that take ages (several hours for a single update
statement) to complete. The client seems to run into a timeout and
reissues the statements (I have some update statement several times in
logs) while the old ones are still running until all connections are in
use and SW grinds to halt.
E.g.:
2012-09-18 11:24:31 CEST [3284]: [118-1] LOG:  duration: 8250548.876 ms
 statement:
	                update rhnServerAction
	                    set status = 1,
	                        pickup_time = current_timestamp,
	                        remaining_tries = 3 - 1
	                where action_id = 6233
	                  and server_id = 1000010014

2012-09-18 11:24:31 CEST [3119]: [295-1] LOG:  duration: 8248422.890 ms
 statement:
	                update rhnServerAction
	                    set status = 1,
	                        pickup_time = current_timestamp,
	                        remaining_tries = 3 - 1
	                where action_id = 6252
	                  and server_id = 1000010007
	
For each update on rhnServerAction the trigger
rhn_server_action_mod_trig_fun() is fired, but I still can't see why the
update should take so long. Manually analyzing the updates does not show
anything suspicious.

My SW installation is not that big (35 clients, with osad and
configuration management). Total database size is 2,3GB. The table
rhnServerAction itself only has 4600 rows.


Can anybody please help in this regard or shed some light on this?

Regards
Patrick

-- 
Lobster LOGsuite GmbH, Münchner Straße 15a, D-82319 Starnberg

HRB 178831, Amtsgericht München
Geschäftsführer: Dr. Martin Fischer, Rolf Henrich