[Spacewalk-list] after installation of 3rd party SSL certificate, now getting (intermittent) errors!

Tue Apr 29 14:38:19 UTC 2014

Helpful spacewalk list --

I'm still looking for an answer to this.  Any ideas?

The problem:
* I can no longer successfully get configuration comparisons for my 72 servers (either "manually" via the GUI or scheduled daily through taskomatic) [ever since I installed new third-party certificates and changed various settings to use https instead of http]

The symptoms:
* The following type of error is spewed to rhn_server_xmlrpc.log (starting about 3 minutes after the job "starts"):
2014/04/28 23:43:49 -04:00 13820 XYZ.XYZ.XYZ.XYZ: rhnSQL/driver_postgresql.check_connection('ERROR', "DATABASE CONNECTION TO 'XXXXXXXXXX' LOST", "Exception information: Database instance has no attribute 'dbh'")

* A "few" (up to 10ish) systems SUCCEED each time; the rest (~62) fail.  Sometimes I get a 500 server error from the GUI.

The questions:
* Are there postgres tuning steps that are needed? (and why did this happen all of a sudden after years of no issues?)
* What other logs should I be looking at?

I'm running spacewalk 2.0 (NOT 2.1) with, obviously, a postgres backend

TIA!
Andy

Andy Ingham
IT Infrastructure
Fuqua School of Business
Duke University

From: Andy Ingham <andy.ingham at duke.edu<mailto:andy.ingham at duke.edu>>
Reply-To: "spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>" <spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>>
Date: Friday, April 25, 2014 9:30 AM
To: "spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>" <spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>>
Subject: Re: [Spacewalk-list] after installation of 3rd party SSL certificate, now getting (intermittent) errors!

Thanks, Paul, for your message.

I did update the CA cert on the hosts (and edited their configurations to point to it).  The package events work GREAT across the servers!

The real trouble is with, for example, the "compare-configs-default" task (via the built-in Task Engine).

Anyone have any ideas why this is now wedged?  (Some configuration file that needs a new pointer to the new cert, or where a password is wrong, for example?)

TIA,
Andy

PS, I'm not sure I agree with your assessment of the heartbleed vulnerability, but that's a discussion for another time and/or place.  ;)

From: Paul Robert Marino <prmarino1 at gmail.com<mailto:prmarino1 at gmail.com>>
Reply-To: "spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>" <spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>>
Date: Thursday, April 24, 2014 7:16 PM
To: "spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>" <spacewalk-list at redhat.com<mailto:spacewalk-list at redhat.com>>
Subject: Re: [Spacewalk-list] after installation of 3rd party SSL certificate, now getting (intermittent) errors!

Did you deploy the CA cert from the third party signer to the hosts as in place of the one initially deployed by spacewalk or update the /etc/sysconfig/rhn/up2date file to point to the stock copy form the 3rd party vendor deployed by the rpm included in the distro.

Also heartbleed effected a smaller number of certs than most people think essentially if the cert was generated prior to the poisonous patch its safe. Its only new certs generated by versions after the poisonous patch and or systems that haven't updated the openssl libraries since which are vulnerable.
The vast majority of production systems are reasonably safe. Further more I wouldn't worry about it that much unless your traffic goes over a public network. Essentially if someone has the access to utilize it in your internal network you have a much bigger problem.

To be clear if you are vulnerable its a huge problem especially if you deal with ecommerce, web based financial information, or data that can be used for identity theaft. But you have really look at and understand the problem before you should panic and replace all your certs.

-- Sent from my HP Pre3

________________________________
On Apr 24, 2014 14:19, Andy Ingham <andy.ingham at duke.edu<mailto:andy.ingham at duke.edu>> wrote:

Ever since we switched from a self-signed to a third-party SSL certificate
two days ago (thank you, Heartbleed!), I've seen intermittent issues.

The most obvious error: the daily 11:00 PM "compare-configs-default" task
(which has always run successfully for all 70+ servers), now is failing
for 98% (but not ALL!) hosts. The event message when it fails is: "This
action has been picked up multiple times without a successful transaction;
this action is now failed for this system."

The osa-dispatcher.log and rhn_server_xmlrpc.log show LOTS of entries like
rhnSQL/driver_postgresql.check_connection('ERROR', "DATABASE CONNECTION
TO 'spaceschema' LOST", "Exception information: Database instance has no
attribute 'dbh'")
clustered around the time of the scheduled run

I've done full restarts of postgres and spacewalk (and even a full reboot
for good measure).

Is there some reason that a cert change would lead to problems with
postgres connections, or is the error above a red herring?

Any ideas of where to look next?

TIA,
Andy

Andy Ingham
IT Infrastructure
Fuqua School of Business
Duke University

_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com<mailto:Spacewalk-list at redhat.com>
https://www.redhat.com/mailman/listinfo/spacewalk-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20140429/462ed9f4/attachment.htm>