[Freeipa-users] SLAPD stops answering

Mon Jan 9 13:54:40 UTC 2017

Hi,

there seem to be to issues here, maybe related: a hanging slapd process 
and the retro CL errors.

If the slapd process is not responding can we get a pstack or gdb 
backtrace (http://www.port389.org/docs/389ds/FAQ/faq.html#debug_crashes) 
of the process ?
About the Retro CL messages, is it always the same changenumber which 
is  reported ?

On 01/09/2017 02:06 PM, Troels Hansen wrote:
> Hi, we have a IPA installation, which obviously needs upgrading.
> Its a single server running RHEL7.1 running IPA 4.1
>
> However, it have been running smooth untill now:
>
> Rebooting makes everything running again, but only for a few days.
>
> It looks like everything fails around 0:17:47 and comes up again just 
> before 8, when the server is rebooted.
>
> Jan  6 00:19:46 fbbidm01 winbindd[2965]: failed to bind to server 
> ldapi://%2fvar%2frun%2fslapd-DOMAIN.LAN.socket with dn="[Anonymous 
> bind]" Error: Local error
> Jan  6 00:19:46 fbbidm01 winbindd[2965]: (unknown)
> Jan  6 00:20:29 fbbidm01 winbindd[2965]: [2017/01/06 00:20:29.758332,  
> 0] ipa_sam.c:4128(bind_callback_cleanup)
>
> Looking at the SLAPD logs also reveals it stopped answering:
>
> [06/Jan/2017:00:17:47 +0100] conn=40702 op=62 SRCH 
> base="cn=radius_aura_admin,cn=groups,cn=accounts,dc=domain,dc=lan" 
> scope=0 filter="(objectClass=*)" attrs="cn"
> [06/Jan/2017:00:17:47 +0100] conn=40702 op=62 RESULT err=0 tag=101 
> nentries=1 etime=0
> [06/Jan/2017:00:17:47 +0100] conn=40702 op=63 SRCH 
> base="cn=radius_users,cn=groups,cn=accounts,dc=domain,dc=lan" scope=0 
> filter="(objectClass=*)" attrs="cn"
> [06/Jan/2017:00:17:47 +0100] conn=40702 op=63 RESULT err=0 tag=101 
> nentries=1 etime=0
> [06/Jan/2017:00:17:47 +0100] conn=40702 op=64 SRCH 
> base="cn=system_radius_users,cn=groups,cn=accounts,dc=domain,dc=lan" 
> scope=0 filter="(objectClass=*)" attrs="cn"
> [06/Jan/2017:00:17:47 +0100] conn=40702 op=64 RESULT err=0 tag=101 
> nentries=1 etime=0
> [06/Jan/2017:00:17:48 +0100] conn=40702 op=65 SRCH 
> base="cn=accounts,dc=domain,dc=lan" scope=2 
> filter="(uid=sys_prov_aura)" attrs=ALL
> [06/Jan/2017:00:17:48 +0100] conn=40702 op=65 RESULT err=0 tag=101 
> nentries=1 etime=0
> [06/Jan/2017:00:17:48 +0100] conn=40702 op=66 BIND 
> dn="uid=sys_prov_aura,cn=users,cn=accounts,dc=domain,dc=lan" 
> method=128 version=3
> [06/Jan/2017:00:17:48 +0100] conn=40702 op=66 RESULT err=0 tag=97 
> nentries=0 etime=0 
> dn="uid=sys_prov_aura,cn=users,cn=accounts,dc=domain,dc=lan"
> [06/Jan/2017:00:17:51 +0100] conn=40703 fd=158 slot=158 connection 
> from 10.250.8.66 to 10.250.8.58
> [06/Jan/2017:00:17:53 +0100] conn=40704 fd=159 slot=159 SSL connection 
> from 10.250.8.37 to 10.250.8.58
> [06/Jan/2017:00:18:02 +0100] conn=40705 fd=160 slot=160 SSL connection 
> from 10.250.8.57 to 10.250.8.58
> [06/Jan/2017:00:18:02 +0100] conn=40706 fd=161 slot=161 SSL connection 
> from 10.250.20.102 to 10.250.8.58
> [06/Jan/2017:00:18:03 +0100] conn=40707 fd=162 slot=162 SSL connection 
> from 10.250.20.102 to 10.250.8.58
> [06/Jan/2017:00:18:58 +0100] conn=40708 fd=163 slot=163 connection 
> from 10.250.8.66 to 10.250.8.58
> [06/Jan/2017:00:19:03 +0100] conn=40709 fd=164 slot=164 connection 
> from local to /var/run/slapd-DOMAIN.LAN.socket
> [06/Jan/2017:00:19:35 +0100] conn=40710 fd=165 slot=165 connection 
> from 10.250.8.58 to 10.250.8.58
> [06/Jan/2017:00:19:35 +0100] conn=40711 fd=166 slot=166 connection 
> from 10.150.27.7 to 10.250.8.58
> [06/Jan/2017:00:19:43 +0100] conn=40712 fd=167 slot=167 SSL connection 
> from 10.250.20.102 to 10.250.8.58
> [06/Jan/2017:00:19:46 +0100] conn=40713 fd=168 slot=168 connection 
> from local to /var/run/slapd-DOMAIN.LAN.socket
>
> It looks like it just stops answering at 00:17:48
>
> The slapd error log reveals nothing:
>
> [06/Jan/2017:00:17:34 +0100] DSRetroclPlugin - replog: an error 
> occured while adding change number 3875312, dn = 
> changenumber=3875312,cn=changelog: Already exists.
> [06/Jan/2017:00:17:34 +0100] retrocl-plugin - retrocl_postob: 
> operation failure [68]
> [06/Jan/2017:07:57:21 +0100] - slapd shutting down - signaling 
> operation threads - op stack size 0 max work q size 735 max work q 
> stack size 23
> [06/Jan/2017:07:57:21 +0100] - slapd shutting down - waiting for 30 
> threads to terminate
> [06/Jan/2017:07:58:02 +0100] SSL Initialization - Configured SSL 
> version range: min: TLS1.0, max: TLS1.2
>
> However, see a gazillion of these lines in the error log:
>
> DSRetroclPlugin - replog: an error occured while adding change number 
> 3875312, dn = changenumber=3875312,cn=changelog: Already exists.
>
> Anyone with some thoughts about this, other that "Just upgrade".
>
> -- 
>
> Med venlig hilsen
>
> *Troels Hansen*
>
> Systemkonsulent
>
> Casalogic A/S
>
> T  (+45) 70 20 10 63
>
> M (+45) 22 43 71 57
>
> <http://www.casalogic.dk/signatur/th.vcf> 
> <http://www.linkedin.com/company/67524> <http://twitter.com/casalogic>
> Red Hat, SUSE, VMware, Citrix, Novell, Yellowfin BI, EnterpriseDB, 
> Sophos og meget mere.
>
>

-- 
Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/freeipa-users/attachments/20170109/130a3bae/attachment.htm>