- Forgot about ulimit, not sure why... I deal with Oracle almost daily. I've adjusted my ulimit nofile setting (both soft and hard): <div style="margin-left:40px"># ulimit -a | grep files open files (-n) 4096 # ulimit -aH | grep files open files (-n) 65000 </div>- There is a /var/lib/pgsql/data/base/pgsql_tmp/ folder, but it is empty at the moment; I will keep an eye on it. - I have the keepalive changed at the sysctl level as every time I do so on the postgresql side, I cannot do so much as cancel a scheduled task without the app going 503 on me. Neither sysctl or postgresql.conf changes appear to help in this case, regardless of what I set them to. Thanks again for your time and assistance with this frustrating issue o' mine, Jonathan <div class="gmail_quote">On Thu, Sep 20, 2012 at 2:09 PM, Paul Robert Marino <<a href="mailto:prmarino1@gmail.com" target="_blank">prmarino1@gmail.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">well thats one part of it but also ulimit -n which can be set persistently with a nofile entry in /etc/security/limits.conf or equivalent file in /etc/security/limits.d/ there is also one more thing do you have a /var/lib/pgsql/data/base/pgsql_tmp/ directory and does it contain files. the existence of that directory indicates there was a query that required more memory than it was allowed to use. I suspect you may have a query thats timing out and the space walk kills the connection uncleanly and leaves behind artifact connections. or possibly you may be reaching a file handle limit on one of the spacewalk processes and keep in mind an network socket is counted as a file. also in the postgresql.conf look at the section about TCP keepalive " # - TCP Keepalives - # see "man 7 tcp" for details #tcp_keepalives_idle = 0 # TCP_KEEPIDLE, in seconds; # 0 selects the system default #tcp_keepalives_interval = 0 # TCP_KEEPINTVL, in seconds; # 0 selects the system default #tcp_keepalives_count = 0 # TCP_KEEPCNT; # 0 selects the system default " note all of these have have a value greater than 1 setting these flags should cull any connections from clients that are no longer there. <div class="HOEnZb"><div class="h5"> On Thu, Sep 20, 2012 at 1:42 PM, Jonathan Scott <<a href="mailto:lists@xistenz.org">lists@xistenz.org</a>> wrote: > By this, do you mean the "fs.file-max" kernel parameter? If so, no I have > not; I am still at the default. > > - Jonathan > > > On Thu, Sep 20, 2012 at 1:12 PM, Paul Robert Marino <<a href="mailto:prmarino1@gmail.com">prmarino1@gmail.com</a>> > wrote: >> >> Also did either of you tune the max open file limit >> >> On Sep 20, 2012 1:08 PM, "Paul Robert Marino" <<a href="mailto:prmarino1@gmail.com">prmarino1@gmail.com</a>> wrote: >>> >>> I think I may have some idea on what may be causing this but I haven't >>> had time to look. At it yet. Did eitherof you tune the sort memory or >>> working memory in your postgres.conf >>> >>> On Sep 20, 2012 10:20 AM, "Patrick Hurrelmann" >>> <<a href="mailto:patrick.hurrelmann@lobster.de">patrick.hurrelmann@lobster.de</a>> wrote: >>>> >>>> On 20.09.2012 16:04, Jonathan Scott wrote: >>>> > Paul, >>>> > >>>> > This is reading like the exact same issue you and I discussed on-list >>>> > a >>>> > few weeks ago. I had closed out that thread as resolved, but the issue >>>> > has since creeped its way back up. Patrick breaks it down well, I too >>>> > just get a pile up of "idle in transaction" db connections which do >>>> > not >>>> > clear with any configuration change I have made (tcp timeout, idle >>>> > timeout and connection limit adjustments in postgresql.conf); a >>>> > restart >>>> > of all associated services gives me about 3-5 days before the app >>>> > becomes unresponsive. >>>> > >>>> > Patrick, may I ask how are you loading your errata? >>>> > >>>> > - Jonathan >>>> >>>> As a short update: I missed one node that had osad still running. Osad >>>> was disabled there as well. I no longer have any update queries waiting. >>>> There are still some idle transactions, but the number is way lower now. >>>> I have the strong feeling that this is all connected osad and push to >>>> clients. Anyone else? >>>> >>>> But back to your question. I'm running David Nutter's centos-errata.py >>>> on a nightly basis directly after a spacewalk restart. >>>> >>>> Regards >>>> Patrick >>>> >>>> -- >>>> Lobster LOGsuite GmbH, Münchner Straße 15a, D-82319 Starnberg >>>> >>>> HRB 178831, Amtsgericht München >>>> Geschäftsführer: Dr. Martin Fischer, Rolf Henrich >>>> >>>> _______________________________________________ >>>> Spacewalk-list mailing list >>>> <a href="mailto:Spacewalk-list@redhat.com">Spacewalk-list@redhat.com</a> >>>> <a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a> >> >> >> _______________________________________________ >> Spacewalk-list mailing list >> <a href="mailto:Spacewalk-list@redhat.com">Spacewalk-list@redhat.com</a> >> <a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a> > > > > _______________________________________________ > Spacewalk-list mailing list > <a href="mailto:Spacewalk-list@redhat.com">Spacewalk-list@redhat.com</a> > <a href="https://www.redhat.com/mailman/listinfo/spacewalk-list" target="_blank">https://www.redhat.com/mailman/listinfo/spacewalk-list</a> </div></div></blockquote></div>