[Spacewalk-list] CPU bound postgresql process ? Why and how to stop it?

[Boyd, Robert]

I'm running spacewalk 2.0 on RHEL 6.4 with database local postgresql 8.4 all patched current.  In recent days I'm seeing one or more very busy processes like this one:

15809 postgres  20   0 1161m 256m 225m R 99.3  6.7  64:35.98 postgres: spaceuser spaceschema 127.0.0.1(43711) SELECT

The server has been CPU bound since about 11/21/2013.   I've tried stopping/restarting spacewalk and postgresql, but as soon as both get fully started one postgresql process goes CPU bound.

Here's a CPU usage graph from a couple of days before through a couple of days after the problem began.

[cid:image005.jpg at 01CEEF71.C8E9EE00]

How can I begin to find out what is going on with spacewalk and the database that is keeping it busy like this?

I tried looking in the /var/log files for something at the system level but there just isn't anything noteworthy in the time interval where this problem began.

In the pgsql logs (e.g. /var/lib/pgsql/data/pg_log/postgresql-Sun.log) I see a lot of messages like these:

LOG:  unexpected EOF on client connection
LOG:  unexpected EOF on client connection
LOG:  automatic analyze of table "spaceschema.public.qrtz_triggers" system usage: CPU 0.00s/0.00u sec elapsed 0.18 sec
LOG:  unexpected EOF on client connection
LOG:  automatic vacuum of table "spaceschema.public.qrtz_triggers": index scans: 0
        pages: 0 removed, 1421 remain
        tuples: 0 removed, 30797 remain
        system usage: CPU 0.17s/0.03u sec elapsed 2.33 sec
LOG:  unexpected EOF on client connection
LOG:  automatic analyze of table "spaceschema.public.qrtz_cron_triggers" system usage: CPU 0.00s/0.00u sec elapsed 0.08 sec
LOG:  automatic vacuum of table "spaceschema.public.qrtz_cron_triggers": index scans: 0
        pages: 0 removed, 786 remain
        tuples: 0 removed, 30856 remain
        system usage: CPU 0.00s/0.00u sec elapsed 0.15 sec
LOG:  unexpected EOF on client connection
LOG:  unexpected EOF on client connection
ERROR:  -20243 : (package_arch_not_found) - Package architecture could not be found
CONTEXT:  SQL statement "SELECT  rhn_exception.raise_exception('package_arch_not_found')"
        PL/pgSQL function "lookup_package_arch" line 14 at PERFORM
STATEMENT:
                    insert into rhnServerPackage
                    (server_id, name_id, evr_id, package_arch_id, installtime)
                    values (1000010465, LOOKUP_PACKAGE_NAME(E'keyutils-libs'), LOOKUP_EVR(NULL, E'1.0', E'2'),
                        LOOKUP_PACKAGE_ARCH(E''), TO_TIMESTAMP(NULL, 'YYYY-MM-DD HH24:MI:SS')
                    )

LOG:  unexpected EOF on client connection
LOG:  unexpected EOF on client connection
LOG:  unexpected EOF on client connection
LOG:  unexpected EOF on client connection
LOG:  unexpected EOF on client connection
LOG:  unexpected EOF on client connection
LOG:  unexpected EOF on client connection
LOG:  unexpected EOF on client connection
LOG:  unexpected EOF on client connection
LOG:  unexpected EOF on client connection
LOG:  unexpected EOF on client connection
LOG:  unexpected EOF on client connection
LOG:  unexpected EOF on client connection
LOG:  unexpected EOF on client connection

Robert Boyd
Senior Systems Engineer | Peoplefluent
p. 919-645-2972 | c. 919-306-4681
e. Robert.Boyd at peoplefluent.com<mailto:michael.dileonardo at peoplefluent.com>
Visit: www.peoplefluent.com<http://www.peoplefluent.com/> | Read: Peoplefluent Blog<http://peoplefluent.com/resources/peoplefluent-blog>
Follow: @peoplefluent<http://twitter.com/peoplefluent> | Download: iPad App<http://itunes.apple.com/us/app/peoplefluent/id474251804?mt=8>
[cid:image001.png at 01CEEF6D.47C8AA90]
[cid:image002.png at 01CEEF6D.47C8AA90]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20131202/6c942eb1/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 8604 bytes
Desc: image001.png
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20131202/6c942eb1/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 22864 bytes
Desc: image002.png
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20131202/6c942eb1/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.jpg
Type: image/jpeg
Size: 40787 bytes
Desc: image005.jpg
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20131202/6c942eb1/attachment.jpg>