[Spacewalk-list] rhntaskorun issues

[Andy Ingham]

Hello list --

I haven't gotten any responses to my request from last week (below).

I'm also now noticing THESE errors in my Tomcat logs.  Any idea why?

Oct 29, 2012 1:46:22 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: A web application registered the JBDC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Oct 29, 2012 1:46:22 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: A web application appears to have started a thread named [RHN Message Dispatcher] but has failed to stop it. This is very likely to create a memory leak.
Oct 29, 2012 1:46:22 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type [null] (value [com.redhat.rhn.common.hibernate.ConnectionManager$1 at 13742ad8]) and a value of type [null] (value [null]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed.


Thanks,
Andy

Andy Ingham
IT Infrastructure
Fuqua School of Business
Duke University



From: Andy Ingham <andy.ingham at duke.edu<mailto:andy.ingham at duke.edu>>
Date: Wednesday, October 24, 2012 12:16 PM
To: "spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>" <spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>>
Subject: Re: [Spacewalk-list] rhntaskorun issues

Folks --

(Beth and I work in the same office)

We believe that we traced the pgsql problem (below) to some iSCSI connection issues.  Unfortunately, since we did not have a usable database dump, we ended up having to do the following:

from the spaceschema database:

truncate table rhntaskorun;

(this was done *without* the cascade option, so only that table was truncated.)

The good news is that we can now do SELECTS from all tables in the database without spectacular postgres crashes.

The bad news is that certain spacewalk functions are now broken (not surprisingly, due to the deletion of records above).

Specifically:

* going to https://our.spacewalk.server.edu/rhn/admin/SatSchedules.do results in "The scheduling service appears down. Please contact your Satellite administrator."

*  osa-dispatcher startup results in:  Starting osa-dispatcher: RHN 2349 2012/10/24 11:14:23 -04:00: ('Error caught:',)
RHN 2349 2012/10/24 11:14:23 -04:00: ('Traceback (most recent call last):\n  File "/usr/share/rhn/osad/jabber_lib.py", line 117, in main\n    self.setup_config(config)\n  File "/usr/share/rhn/osad/osa_dispatcher.py", line 112, in setup_config\n    rhnSQL.initDB()\n  File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/__init__.py", line 124, in initDB\n    __init__DB(backend, host, port, username, password, database)\n  File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/__init__.py", line 55, in __init__DB\n    __DB.connect()\n  File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py", line 171, in connect\n    return self.connect(reconnect=0)\n  File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py", line 160, in connect\n    password=self.password)\nSQLConnectError: (None, None, \'spaceschema\', \'Attempting Re-Connect to the database failed\')\n',)

* the rhn_taskomatic_daemon.log shows, among other things:  INFO   | jvm 1    | 2012/10/24 11:46:13 | WARNING: java.lang.NoClassDefFoundError: Could not initialize class net.sf.cglib.proxy.Enhancer

* the rhn_server_xmlrpc.log shows, among other things:  2012/10/24 11:53:23 -04:00 6184 10.148.42.65: server/rhnChannel.guess_channels_for_server('ERROR', 'No available channels for (server, org)', (1000010031, 1), '6.3', 'x86_64-redhat-linux')


My questions:
(1)  Any chance that a "stock" set of records might be available somewhere for re-populating the rhntaskorun table?

(2) Any advice on the other errors noted above? (Assuming they aren't just red herrings).

Many thanks in advance.

Andy

Andy Ingham
IT Infrastructure
Fuqua School of Business
Duke University



From: Elizabeth Good <elizabeth.good at duke.edu<mailto:elizabeth.good at duke.edu>>
Reply-To: "spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>" <spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>>
Date: Tuesday, October 23, 2012 9:57 AM
To: "spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>" <spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>>
Subject: [Spacewalk-list] rhntaskorun issues

Hello,

I’m reposting this in the hopes that someone has run across a similar issue with the rhntaskorun table.  Spacewalk is down when I’ve performed the following actions.   As far as we can tell it is not a problem with our disks.

We are on spacewalk 1.7, postgresql 8.4.13.  We have not been able to get a good pg_dump, as the postgresql backup stops at the rhntaskorun table:

pg_dump: Dumping the contents of table "rhntaskorun" failed: PQgetCopyData() failed.
pg_dump: Error message from server: pg_dump: The command was: COPY public.rhntaskorun (id, template_id, schedule_id, org_id, start_time, end_time, std_output_path, std_error_path, status, created, modified) TO stdout;
pg_dump: *** aborted because of error


In addition,  selecting from the rhntaskorun also causes issues.

All the postgresql processes go down when I do the select or run a pg_dump, with the exception of the postmaster and logger processes.



Thanks,

Beth

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20121029/7dfaf57e/attachment.htm>