[Spacewalk-list] Spacewalk 1.7 w/ postgresql crashed

[Wolfgang Neudorfer]

Hello Paul,

nobody was logged in and the host is only reachable from a very small network range. I think I can say that nobody did "anything naughty".

I cannot outrule that there was a memory issue and oomkiller started it's madness - but I don't see anything related to this in /var/log/messages.

Any other ideas?

Regards,

Wolfgang

----- Original Message -----
From: "Paul Robert Marino" <prmarino1 at gmail.com>
To: spacewalk-list at redhat.com
Sent: Monday, 19 November, 2012 3:35:56 PM
Subject: Re: [Spacewalk-list] Spacewalk 1.7 w/ postgresql crashed




Postgresql was killed with a -9 which means some one hard killed the process then restarted it. Looks like some one was doing something naughty on your box. 
This is not a spacewalk problem this is a sysadmin who made a mistake then didn't fess to it. 
On Nov 19, 2012 4:18 AM, "Wolfgang Neudorfer" < mlist at woifi.at > wrote: 


Hi, 

starting Saturday 17/11/2012 01:46, our Spacewalk server started to send out multiple mails per minute (probably on each connection attempt of a client?) like this: 

------------------------------------------------- 
RHN TRACEBACK from spacewalk1: 

Exception reported from spacewalk1 
Time: Sat Nov 17 01:45:30 2012 
Exception type <class 'spacewalk.server.rhnSQL.sql_base.SQLConnectError'> 
Request object information: 
URI: /XMLRPC 
Remote Host: 192.168.254.xxx 
Server Name: spacewalk1:443 
Headers passed in: 
Accept-Encoding: identity 
CONTENT_LENGTH: 2325 
CONTENT_TYPE: text/xml 
DOCUMENT_ROOT: /var/www/html 
GATEWAY_INTERFACE: CGI/1.1 
HTTPS: 1 
HTTP_ACCEPT_ENCODING: identity 
HTTP_HOST: spacewalk1 
HTTP_USER_AGENT: rhn.rpclib.py/$Revision$ 
HTTP_X_CLIENT_VERSION: 1 
HTTP_X_INFO: RPC Processor (C) Red Hat, Inc (version $Revision$) 
HTTP_X_RHN_TRANSPORT_CAPABILITY: follow-redirects=3 
HTTP_X_TRANSPORT_INFO: Extended Capabilities Transport (C) Red Hat, Inc (version $Revision$) 
Host: tsasecspacewalk1.sec 
PATH_INFO: 
QUERY_STRING: 
REMOTE_ADDR: 192.168.254.xxx 
REMOTE_PORT: 59649 
REQUEST_METHOD: POST 
REQUEST_URI: /XMLRPC 
SCRIPT_FILENAME: /usr/share/rhn/wsgi/xmlrpc.py 
SCRIPT_NAME: /XMLRPC 
SCRIPT_URI: https://tsasecspacewalk1.sec/XMLRPC 
SCRIPT_URL: /XMLRPC 
SERVER_ADDR: 192.168.254.xxx 
SERVER_ADMIN: root at localhost 
SERVER_NAME: spacewalk1 
SERVER_PORT: 443 
SERVER_PROTOCOL: HTTP/1.1 
SERVER_SIGNATURE: <address>Apache Server at spacewalk1 Port 443</address> 

SERVER_SOFTWARE: Apache 
User-Agent: rhn.rpclib.py/$Revision$ 
X-Client-Version: 1 
X-Info: RPC Processor (C) Red Hat, Inc (version $Revision$) 
X-RHN-Transport-Capability: follow-redirects=3 
X-Transport-Info: Extended Capabilities Transport (C) Red Hat, Inc (version $Revision$) 
mod_wsgi.application_group: tsasecspacewalk1.sec|/xmlrpc 
mod_wsgi.callable_object: application 
mod_wsgi.handler_script: 
mod_wsgi.input_chunked: 0 
mod_wsgi.listener_host: 
mod_wsgi.listener_port: 443 
mod_wsgi.process_group: 
mod_wsgi.request_handler: wsgi-script 
mod_wsgi.script_reloading: 1 
mod_wsgi.version: (3, 2) 
wsgi.errors: <mod_wsgi.Log object at 0x7f8e4a83d370> 
wsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f8e4a83c300> 
wsgi.input: <mod_wsgi.Input object at 0x7f8e4a83d330> 
wsgi.multiprocess: True 
wsgi.multithread: False 
wsgi.run_once: False 
wsgi.url_scheme: https 
wsgi.version: (1, 1) 
------------------------------------------------- 

Apparently, something happend to the postgres server. In the log I see: 

------------------------------------------------- 
LOG: server process (PID 31999) was terminated by signal 9: Killed 
LOG: terminating any other active server processes 
WARNING: terminating connection because of crash of another server process 
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 

... (the last 2 lines appear multiple times) 

FATAL: the database system is in recovery mode 
FATAL: the database system is in recovery mode 
FATAL: the database system is in recovery mode 
FATAL: the database system is in recovery mode 

... (this line apprears multiple times) 
------------------------------------------------- 

The harddisk was not full, also RAM was ok. I restarted the host and Spacewalk seems to be fine. I can login an all hosts are there. 

Any hints? I am running Spacewalk 1.7 on CentOS x64 6.3 with PostgresSQL 8.4.13. 

Thanks, 

Wolfgang 

_______________________________________________ 
Spacewalk-list mailing list 
Spacewalk-list at redhat.com 
https://www.redhat.com/mailman/listinfo/spacewalk-list 

_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list