<div dir="ltr"><div>As it sounds like I might have hit a bug during the certificate update process. Thinking about alternative solutions.</div>Is there an option to export the data, rebuild the server and re-import to fully recover the IPA system incl. established members?</div><div class="gmail_extra"> <div class="gmail_quote">2017-01-16 3:57 GMT-06:00 Florence Blanc-Renaud <<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a>>: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 01/16/2017 01:47 AM, Daniel Schimpfoessl wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Anything else I should look for? </blockquote> Hi Daniel, did you see this mail thread [1]? They had the same issue and found a temporary workaround to enable dogtag to connect to LDAP. If the workaround works, it definitely means that the issue comes from the secured communications between Dogtag and LDAP, and the following could be checked: - LDAPs port 636 is enabled and answering - The server certificate used by the LDAP server is valid (nickname 'Server-Cert' in /etc/dirsrv/slapd-DOMAIN) - The Server certificate used by the LDAP server has been delivered by a CA trusted by Dogtag (CA cert must be in /etc/pki/pki-tomcat/alias) - The certificate used by Dogtag to authenticate to LDAP (nickname subsystemCert cert-pki-ca in /etc/pki/pki-tomcat/alias) is valid and stored in a corresponding user entry in LDAP (uid=pkidbuser,ou=people,o=ipaca). - The certificates must match the ones in /etc/pki/pki-tomcat/ca/CS.cfg (line ca.signing.cert=... must match the CA cert and ca.subsystem.cert=... must match subsystemCert cert-pki-ca). If the system is configured with SE linux mode = enforcing, it may explain the renewal issues (see BZ 1365188 [2] and 1366915 [3]). Flo. [1] <a href="https://www.redhat.com/archives/freeipa-users/2017-January/msg00215.html" rel="noreferrer" target="_blank">https://www.redhat.com/archives/freeipa-users/2017-January/msg00215.html</a> [2] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1365188" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1365188</a> [3] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1366915" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1366915</a> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> 2017-01-11 22:33 GMT-06:00 Daniel Schimpfoessl <<a href="mailto:daniel@schimpfoessl.com" target="_blank">daniel@schimpfoessl.com</a> <mailto:<a href="mailto:daniel@schimpfoessl.com" target="_blank">daniel@schimpfoessl.com</a>>>: Flo, these are all the errors found: grep 'RESULT err=' access | perl -pe 's/.*(RESULT\s+err=\d+).*/$1/g' | sort -n | uniq -c | sort -n 2 RESULT err=6 95 RESULT err=32 200 RESULT err=14 2105 RESULT err=0 2017-01-05 8:10 GMT-06:00 Florence Blanc-Renaud <<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a> <mailto:<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a>>>:<div><div class="h5"> On 01/04/2017 07:24 PM, Daniel Schimpfoessl wrote: From the logs: /var/log/dirsrv/slapd-DOMAIN-COM/errors ... a few warnings about cache size, NSACLPLugin and schema-compat-plugin [04/Jan/2017:12:14:21.392642021 -0600] slapd started. Listening on All Interfaces port 389 for LDAP requests /var/log/dirsrv/slapd-DOMAIN-COM/access ... lots of entries, not sure what to look for some lines contain RESULT with err!=0 [04/Jan/2017:12:18:01.753400307 -0600] conn=5 op=243 RESULT err=32 tag=101 nentries=0 etime=0 [04/Jan/2017:12:18:01.786928085 -0600] conn=44 op=1 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress Hi Daniel, are there any RESULT err=48 that could correspond to the error seen on pki logs? Flo /var/log/dirsrv/slapd-DOMAIN-COM/errors [04/Jan/2017:12:19:25.566022098 -0600] slapd shutting down - signaling operation threads - op stack size 5 max work q size 2 max work q stack size 2 [04/Jan/2017:12:19:25.572566622 -0600] slapd shutting down - closing down internal subsystems and plugins 2017-01-04 8:38 GMT-06:00 Daniel Schimpfoessl <<a href="mailto:daniel@schimpfoessl.com" target="_blank">daniel@schimpfoessl.com</a> <mailto:<a href="mailto:daniel@schimpfoessl.com" target="_blank">daniel@schimpfoessl.com</a>> </div></div> <mailto:<a href="mailto:daniel@schimpfoessl.com" target="_blank">daniel@schimpfoessl.com</a> <mailto:<a href="mailto:daniel@schimpfoessl.com" target="_blank">daniel@schimpfoessl.com</a>>>>: Do you have a list of all log files involved in IPA? Would be good to consolidate them into ELK for analysis. 2017-01-04 2:48 GMT-06:00 Florence Blanc-Renaud <<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a> <mailto:<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a>> <mailto:<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a> <mailto:<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a>>>>:<div><div class="h5"> On 01/02/2017 07:24 PM, Daniel Schimpfoessl wrote: Thanks for your reply. This was the initial error I asked for help a while ago and did not get resolved. Further digging showed the recent errors. The service was running (using ipactl start --force) and only after a restart I am getting a stack trace for two primary messages: Could not connect to LDAP server host <a href="http://wwgwho01.webwim.com" rel="noreferrer" target="_blank">wwgwho01.webwim.com</a> <<a href="http://wwgwho01.webwim.com" rel="noreferrer" target="_blank">http://wwgwho01.webwim.com</a>> <<a href="http://wwgwho01.webwim.com" rel="noreferrer" target="_blank">http://wwgwho01.webwim.com</a>> <<a href="http://wwgwho01.webwim.com" rel="noreferrer" target="_blank">http://wwgwho01.webwim.com</a>> port 636 Error netscape.ldap.LDAPException: Authentication failed (48) ... Internal Database Error encountered: Could not connect to LDAP server host <a href="http://wwgwho01.webwim.com" rel="noreferrer" target="_blank">wwgwho01.webwim.com</a> <<a href="http://wwgwho01.webwim.com" rel="noreferrer" target="_blank">http://wwgwho01.webwim.com</a>> <<a href="http://wwgwho01.webwim.com" rel="noreferrer" target="_blank">http://wwgwho01.webwim.com</a>> <<a href="http://wwgwho01.webwim.com" rel="noreferrer" target="_blank">http://wwgwho01.webwim.com</a>> port 636 Error netscape.ldap.LDAPException: Authentication failed (48) ... and finally: [02/Jan/2017:12:20:34][localhost-startStop-1]: CMSEngine.shutdown() 2017-01-02 3:45 GMT-06:00 Florence Blanc-Renaud <<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a> <mailto:<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a>> <mailto:<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a> <mailto:<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a>>> <mailto:<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a> <mailto:<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a>> <mailto:<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a> <mailto:<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a>>>>>: systemctl start pki-tomcatd@pki-tomcat.service Hi Daniel, the next step would be to understand the root cause of this "Authentication failed (48)" error. Note the exact time of this log and look for a corresponding log in the LDAP server logs (/var/log/dirsrv/slapd-DOMAIN-COM/access), probably a failing BIND with err=48. This may help diagnose the issue (if we can see which certificate is used for the bind or if there is a specific error message). For the record, a successful bind over SSL would produce this type of log where we can see the certificate subject and the user mapped to this certificate: [...] conn=47 fd=84 slot=84 SSL connection from 10.34.58.150 to 10.34.58.150 [...] conn=47 TLS1.2 128-bit AES; client CN=CA Subsystem,O=<a href="http://DOMAIN.COM" rel="noreferrer" target="_blank">DOMAIN.COM</a> <<a href="http://DOMAIN.COM" rel="noreferrer" target="_blank">http://DOMAIN.COM</a>> <<a href="http://DOMAIN.COM" rel="noreferrer" target="_blank">http://DOMAIN.COM</a>>; issuer CN=Certificate Authority,O=<a href="http://DOMAIN.COM" rel="noreferrer" target="_blank">DOMAIN.COM</a> </div></div> <<a href="http://DOMAIN.COM" rel="noreferrer" target="_blank">http://DOMAIN.COM</a>> <<a href="http://DOMAIN.COM" rel="noreferrer" target="_blank">http://DOMAIN.COM</a>> [...] conn=47 TLS1.2 client bound as uid=pkidbuser,ou=people,o=ipaca [...] conn=47 op=0 BIND dn="" method=sasl version=3 mech=EXTERNAL [...] conn=47 op=0 RESULT err=0 tag=97 nentries=0 etime=0 dn="uid=pkidbuser,ou=people,o=ipaca" Flo </blockquote> </blockquote></div> </div>