[Spacewalk-list] Massive problems with slow updates on rhnServerAction

[Jonathan Scott]

- Forgot about ulimit, not sure why... I deal with Oracle almost daily.
I've adjusted my ulimit nofile setting (both soft and hard):

# ulimit -a | grep files
open files                      (-n) 4096

# ulimit -aH | grep files
open files                      (-n) 65000

- There is a /var/lib/pgsql/data/base/pgsql_tmp/ folder, but it is empty at
the moment; I will keep an eye on it.

- I have the keepalive changed at the sysctl level as every time I do so on
the postgresql side, I cannot do so much as cancel a scheduled task without
the app going 503 on me. Neither sysctl or postgresql.conf changes appear
to help in this case, regardless of what I set them to.

Thanks again for your time and assistance with this frustrating issue o'
mine,
Jonathan

On Thu, Sep 20, 2012 at 2:09 PM, Paul Robert Marino <prmarino1 at gmail.com>wrote:

> well thats one part of it but also
> ulimit -n
> which can be set persistently with a nofile entry in
> /etc/security/limits.conf or equivalent file in
> /etc/security/limits.d/
>
> there is also one more thing do you have a
> /var/lib/pgsql/data/base/pgsql_tmp/ directory and does it contain
> files. the existence of that directory indicates there was a query
> that required more memory than it was allowed to use.
>
>
> I suspect you may have a query thats timing out and the space walk
> kills the connection uncleanly and leaves behind artifact connections.
> or possibly you may be reaching a file handle limit on one of the
> spacewalk processes and keep in mind an network socket is counted as a
> file.
>
>
>
> also in the postgresql.conf look at the section about TCP keepalive
>
>
> "
> # - TCP Keepalives -
> # see "man 7 tcp" for details
>
> #tcp_keepalives_idle = 0                # TCP_KEEPIDLE, in seconds;
>                                         # 0 selects the system default
> #tcp_keepalives_interval = 0            # TCP_KEEPINTVL, in seconds;
>                                         # 0 selects the system default
> #tcp_keepalives_count = 0               # TCP_KEEPCNT;
>                                         # 0 selects the system default
> "
>
> note all of these have have a value greater than 1
> setting these flags should cull any connections from clients that are
> no longer there.
>
> On Thu, Sep 20, 2012 at 1:42 PM, Jonathan Scott <lists at xistenz.org> wrote:
> > By this, do you mean the "fs.file-max" kernel parameter? If so, no I have
> > not; I am still at the default.
> >
> > - Jonathan
> >
> >
> > On Thu, Sep 20, 2012 at 1:12 PM, Paul Robert Marino <prmarino1 at gmail.com
> >
> > wrote:
> >>
> >> Also did either of you tune the max open file limit
> >>
> >> On Sep 20, 2012 1:08 PM, "Paul Robert Marino" <prmarino1 at gmail.com>
> wrote:
> >>>
> >>> I think I may have some idea on what may be causing this but I haven't
> >>> had time to look. At it yet. Did eitherof you tune the sort memory or
> >>> working memory in your postgres.conf
> >>>
> >>> On Sep 20, 2012 10:20 AM, "Patrick Hurrelmann"
> >>> <patrick.hurrelmann at lobster.de> wrote:
> >>>>
> >>>> On 20.09.2012 16:04, Jonathan Scott wrote:
> >>>> > Paul,
> >>>> >
> >>>> > This is reading like the exact same issue you and I discussed
> on-list
> >>>> > a
> >>>> > few weeks ago. I had closed out that thread as resolved, but the
> issue
> >>>> > has since creeped its way back up. Patrick breaks it down well, I
> too
> >>>> > just get a pile up of "idle in transaction" db connections which do
> >>>> > not
> >>>> > clear with any configuration change I have made (tcp timeout, idle
> >>>> > timeout and connection limit adjustments in postgresql.conf); a
> >>>> > restart
> >>>> > of all associated services gives me about 3-5 days before the app
> >>>> > becomes unresponsive.
> >>>> >
> >>>> > Patrick, may I ask how are you loading your errata?
> >>>> >
> >>>> > - Jonathan
> >>>>
> >>>> As a short update: I missed one node that had osad still running. Osad
> >>>> was disabled there as well. I no longer have any update queries
> waiting.
> >>>> There are still some idle transactions, but the number is way lower
> now.
> >>>> I have the strong feeling that this is all connected osad and push to
> >>>> clients. Anyone else?
> >>>>
> >>>> But back to your question. I'm running David Nutter's centos-errata.py
> >>>> on a nightly basis directly after a spacewalk restart.
> >>>>
> >>>> Regards
> >>>> Patrick
> >>>>
> >>>> --
> >>>> Lobster LOGsuite GmbH, Münchner Straße 15a, D-82319 Starnberg
> >>>>
> >>>> HRB 178831, Amtsgericht München
> >>>> Geschäftsführer: Dr. Martin Fischer, Rolf Henrich
> >>>>
> >>>> _______________________________________________
> >>>> Spacewalk-list mailing list
> >>>> Spacewalk-list at redhat.com
> >>>> https://www.redhat.com/mailman/listinfo/spacewalk-list
> >>
> >>
> >> _______________________________________________
> >> Spacewalk-list mailing list
> >> Spacewalk-list at redhat.com
> >> https://www.redhat.com/mailman/listinfo/spacewalk-list
> >
> >
> >
> > _______________________________________________
> > Spacewalk-list mailing list
> > Spacewalk-list at redhat.com
> > https://www.redhat.com/mailman/listinfo/spacewalk-list
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/spacewalk-list/attachments/20120920/062001d2/attachment.htm>