[Spacewalk-list] Postgresql Fails to Start After Datastore Failure

Wed Nov 26 15:06:56 UTC 2014

I celebrated too early… although the service starts up now and I can access spacewalk with all of the information intact, if I try to add any new items or make changes to a lot of existing stuff, I end up receiving internal errors on subsequent attempts to access those items.  For instance, adding a new machine to SW will generate an internal server error, deleting a machine will not remove the machine from SW and I will see internal server errors when trying to click on it from the interface.  

I tried to upgrade to 2.2 but I get errors on the schema upgrade step.  I think at this point I will just work on building a replacement and then use the old server to verify all of my settings, unless anyone has some last ditch attempt ideas?   

From: spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-bounces at redhat.com] On Behalf Of Paul Robert Marino
Sent: Monday, November 24, 2014 5:57 PM
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] Postgresql Fails to Start After Datastore Failure

Wow that's a very unusual problem. PostgreSQL usually recovers from that kind of thing easily. The only scenarios where I can picture this happening is if there was a power loss and a battery in a RAID got depleted or the same scenario with consumer grade SATA drives because they often lie about the cache state when an fsync is called.

-- Sent from my HP Pre3

  _____  

On Nov 24, 2014 12:58 PM, Glennie, Jonathan - 0443 - MITLL <jrglenni at ll.mit.edu> wrote: 

Phew… blowing away the pgsql commit log allowed it to start up successfully.  There may have been items in there that were lost, but I’ll take that any day over having to recreate the entire server from scratch.  Thanks for your help!   

From: spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-bounces at redhat.com] On Behalf Of Glennie, Jonathan - 0443 - MITLL
Sent: Monday, November 24, 2014 11:32 AM
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] Postgresql Fails to Start After Datastore Failure

I found the log file… looks like it is complaining that it cannot complete the autorecovery and the database needs to be restores from backup.  Unfortunately all of my VM snapshots got hosed and that was my backup…. 

I’m still hoping there is a way I can attempt to repair the schema while preserving the data… in the process of trying to re-run the spacewalk-setup.  

From: spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-bounces at redhat.com] On Behalf Of Glennie, Jonathan - 0443 - MITLL
Sent: Monday, November 24, 2014 10:13 AM
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] Postgresql Fails to Start After Datastore Failure

I had received the lock error message before so I did start by removing that file.  It removed that error message but it was still saying Failed whenever I tried to start the service.  

Are there any commands that I can use to stat the DB manually in a more verbose way so I can get some additional info on what’s failing?   Or is there any “repair” command I can use from spacewalk to re-install the service but not overwrite the database?   

From: spacewalk-list-bounces at redhat.com [mailto:spacewalk-list-bounces at redhat.com] On Behalf Of Waldirio Manhães Pinheiro
Sent: Friday, November 21, 2014 8:05 PM
To: spacewalk-list at redhat.com
Subject: Re: [Spacewalk-list] Postgresql Fails to Start After Datastore Failure

Hello Jon,

You can check in /var/lib/pgsql/data, check if you have a file named postmaster.pid, if yes, you can remove (rm postmaster.pid) and restart or start the postgresql daemon.

[root at spacewalk ~]# service postgresql start

if you check, will be recreated the file

[root at spacewalk data]# ll postmaster.pid

-rw-------. 1 postgres postgres 45 Nov 21 23:03 postmaster.pid

[root at spacewalk data]#

The problem normally is, if your db are running and for any reason you lost the connection with the directory, the file will still there and when you try to restart, the file will not be rewrite and the daemon will not start.

Try it and tell me if fix your problem.

Take Care and have a great weekend.

______________
Atenciosamente
Waldirio
msn: waldirio at gmail.com
Skype: waldirio
Site: www.waldirio.com.br
Blog: blog.waldirio.com.br

LinkedIn: http://br.linkedin.com/pub/waldirio-pinheiro/22/b21/646
PGP: www.waldirio.com.br/public.html

On Fri, Nov 21, 2014 at 6:36 PM, Glennie, Jonathan - 0443 - MITLL <jrglenni at ll.mit.edu> wrote:

Hello-

Our spacewalk server failed badly the other day when the underlying datastore went offline.  The system boots but when spacewalk tries to start up, the postgresql service fails to start.  I have checked /var/log/messages for any indication as to why, but there is nothing I can find.  I’ve also checked in /var/lib/pqsql/pgstatrtup.log but the entries in there seem old and not relevant to the current situation.  

Does anyone know where or how I might go about starting the service in some kind of debug mode to try and get a better idea of what’s going on, or if there are logs that get saved somewhere that I haven’t found yet?  I’m definitely not very well versed in postgrest DB maintenance or troubleshooting, so any help would be greatly appreciated.  Thanks in advance.

-Jon

_______________________________________________
Spacewalk-list mailing list
Spacewalk-list at redhat.com
https://www.redhat.com/mailman/listinfo/spacewalk-list

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20141126/80b0290d/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5613 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20141126/80b0290d/attachment.p7s>