<div dir="ltr"><br><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br></div><div dir="ltr"><div dir="ltr" style="font-size:12.8px"><br></div><div dir="ltr"><b style="font-size:12.8px"><font size="2">Mike Plemmons | Senior DevOps Engineer | CROSSCHX<br></font></b><div>614.427.2411</div><div><a href="mailto:mike.plemmons@crosschx.com" style="font-size:12.8px" target="_blank">mike.plemmons@crosschx.com</a><br></div><div style="font-size:12.8px"><a href="http://www.crosschx.com/" target="_blank">www.crosschx.com</a></div></div></div></div></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Thu, May 18, 2017 at 8:02 AM, Florence Blanc-Renaud <span dir="ltr"><<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 05/15/2017 08:33 PM, Michael Plemmons wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I have done more searching in my logs and I see the following errors.<br>
<br>
This is in the localhost log file /var/lib/pki/pki-tomcat/logs<br>
<br>
May 15, 2017 3:08:08 PM org.apache.catalina.core.Appli<wbr>cationContext log<br>
SEVERE: StandardWrapper.Throwable<br>
java.lang.NullPointerException<br>
<br>
May 15, 2017 3:08:08 PM org.apache.catalina.core.Stand<wbr>ardContext<br>
loadOnStartup<br>
SEVERE: Servlet [castart] in web application [/ca] threw load() exception<br>
java.lang.NullPointerException<br>
<br>
May 15, 2017 3:08:09 PM org.apache.catalina.core.Stand<wbr>ardHostValve invoke<br>
SEVERE: Exception Processing /ca/admin/ca/getStatus<br>
<a href="http://javax.ws.rs" rel="noreferrer" target="_blank">javax.ws.rs</a> <<a href="http://javax.ws.rs" rel="noreferrer" target="_blank">http://javax.ws.rs</a>>.ServiceUn<wbr>availableException: Subsystem<br>
unavailable<br>
<br>
<br>
Looking at the debug log it says Authentication failed for port 636.<br>
<br>
[15/May/2017:17:39:25][localho<wbr>st-startStop-1]: LdapAuthInfo: init()<br>
[15/May/2017:17:39:25][localho<wbr>st-startStop-1]: LdapAuthInfo: init begins<br>
[15/May/2017:17:39:25][localho<wbr>st-startStop-1]: LdapAuthInfo: init ends<br>
[15/May/2017:17:39:25][localho<wbr>st-startStop-1]: init: before<br>
makeConnection errorIfDown is true<br>
[15/May/2017:17:39:25][localho<wbr>st-startStop-1]: makeConnection:<br>
errorIfDown true<br>
[15/May/2017:17:39:25][localho<wbr>st-startStop-1]:<br>
SSLClientCertificateSelectionC<wbr>B: Setting desired cert nickname to:<br>
subsystemCert cert-pki-ca<br>
[15/May/2017:17:39:25][localho<wbr>st-startStop-1]: LdapJssSSLSocket: set<br>
client auth cert nickname subsystemCert cert-pki-ca<br>
[15/May/2017:17:39:25][localho<wbr>st-startStop-1]:<br>
SSLClientCertificatSelectionCB<wbr>: Entering!<br>
[15/May/2017:17:39:25][localho<wbr>st-startStop-1]:<br>
SSLClientCertificateSelectionC<wbr>B: returning: null<br>
[15/May/2017:17:39:25][localho<wbr>st-startStop-1]: SSL handshake happened<br>
Could not connect to LDAP server host <a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">ipa12.mgmt.crosschx.com</a><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>> port 636 Error<br>
netscape.ldap.LDAPException: Authentication failed (48)<br>
at<br>
com.netscape.cmscore.ldapconn.<wbr>LdapBoundConnFactory.makeConne<wbr>ction(LdapBoundConnFactory.<wbr>java:205)<br>
<br>
<br>
I looked at the validity of the cert it mentions and it is fine.<br>
<br>
(root)>getcert status -v -d /etc/pki/pki-tomcat/alias -n 'subsystemCert<br>
cert-pki-ca'<br>
State MONITORING, stuck: no.<br>
<br>
<br>
I then looked at the ldap errors around the time of this failure and I<br>
am seeing this log entry.<br>
<br>
<br>
[15/May/2017:17:38:42.06308075<wbr>8 +0000] set_krb5_creds - Could not get<br>
initial credentials for principal<br>
[ldap/<a href="mailto:ipa12.mgmt.crosschx.com@MGMT.CROSSCHX.COM" target="_blank">ipa12.mgmt.crosschx.com@<wbr>MGMT.CROSSCHX.COM</a><br>
<mailto:<a href="mailto:ipa12.mgmt.crosschx.com@MGMT.CROSSCHX.COM" target="_blank">ipa12.mgmt.crosschx.co<wbr>m@MGMT.CROSSCHX.COM</a>>] in keytab<br>
[FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for<br>
requested realm)<br>
<br>
When I perform a klist against that keytab nothing appears out of the<br>
ordinary compared to working IPA servers.<br>
<br>
I am not sure what to look at next.<br>
<br>
</blockquote>
<br>
Hi,<br>
<br>
you can try the following to manually replay the connection established by Dogtag to LDAP server:<br>
<br>
root$ export LDAPTLS_CACERTDIR=/etc/pki/pki<wbr>-tomcat/alias<br>
root$ export LDAPTLS_CERT='subsystemCert cert-pki-ca'<br>
<br>
The above commands specify the NSSDB containing the user certificate and its name for SASL-EXTERNAL authentication.<br>
<br>
Then note the value obtained below as it will be used for the next step as the password to access the private key in the NSSDB:<br>
root$ grep internal /etc/pki/pki-tomcat/password.c<wbr>onf<br>
internal=<some value><br>
<br>
root$ ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL -Q -LLL dn namingcontexts<br>
Please enter pin, password, or pass phrase for security token 'ldap(0)': <<<< here supply the value found above<br>
dn:<br>
namingcontexts: cn=changelog<br>
namingcontexts: dc=ipadomain,dc=com<br>
namingcontexts: o=ipaca<br>
<br></blockquote><div><br></div><div><br></div><div>So I guess I found my problem.</div><div><br></div><div><div>(root)>ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL -Q -LLL dn namingcontexts</div><div>Please enter pin, password, or pass phrase for security token 'ldap(0)':</div><div>ldap_sasl_interactive_bind_s: Can't contact LDAP server (-1)</div><div> additional info: TLS error -12195:Peer does not recognize and trust the CA that issued your certificate.</div></div><div><br></div><div><br></div><div>I looked at our certs in /etc/dirsrv/slapd-IPADOMAIN-COM and found the following.</div><div><br></div><div>IPA12 - problem server</div><div><div>(root)>certutil -L -d /etc/dirsrv/slapd-IPADOMAIN-COM</div><div><br></div><div>Certificate Nickname Trust Attributes</div><div> SSL,S/MIME,JAR/XPI</div><div><br></div><div>Server-Cert u,u,u</div><div>IPADOMAIN-COM IPA CA C,,</div><div><br></div><div><br></div><div><br></div><div>IPA11/IPA13 - 11 was the master and 13 is the new master</div><div>(root)>certutil -L -d /etc/dirsrv/slapd-IPADOMAIN-COM</div><div><br></div><div>Certificate Nickname Trust Attributes</div><div> SSL,S/MIME,JAR/XPI</div><div><br></div><div>Server-Cert u,u,u</div><div>IPADOMAIN-COM IPA CA CT,C,C</div></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
In the LDAP server access log (in /etc/dirsrv/slapd-IPADOMAIN.CO<wbr>M/access), you should see the corresponding connection:<br>
<br>
[18/May/2017:13:35:14.82209041<wbr>7 +0200] conn=297 fd=150 slot=150 SSL connection from xxx to yyy<br>
[18/May/2017:13:35:15.78941401<wbr>7 +0200] conn=297 TLS1.2 128-bit AES-GCM; client CN=CA Subsystem,O=<a href="http://IPADOMAIN.COM" rel="noreferrer" target="_blank">IPADOMAIN.COM</a>; issuer CN=Certificate Authority,O=<a href="http://IPADOMAIN.COM" rel="noreferrer" target="_blank">IPADOMAIN.COM</a><br>
[18/May/2017:13:35:15.79310850<wbr>9 +0200] conn=297 TLS1.2 client bound as uid=pkidbuser,ou=people,o=ipac<wbr>a<br>
[18/May/2017:13:35:15.79810150<wbr>5 +0200] conn=297 op=0 BIND dn="" method=sasl version=3 mech=EXTERNAL<br>
[18/May/2017:13:35:15.80032207<wbr>6 +0200] conn=297 op=0 RESULT err=0 tag=97 nentries=0 etime=0 dn="uid=pkidbuser,ou=people,o=<wbr>ipaca"<br>
<br>
HTH,<br>
Flo.<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
<br>
<br>
*Mike Plemmons | Senior DevOps Engineer | CROSSCHX<br>
*<br>
614.427.2411<br>
<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx.com</a> <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">www.crosschx.com</a> <<a href="http://www.crosschx.com/" rel="noreferrer" target="_blank">http://www.crosschx.com/</a>><br>
<br>
On Wed, May 10, 2017 at 3:35 PM, Michael Plemmons<br>
<<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@crosschx.com</a> <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>>><br>
wrote:<br>
<br>
The PKI service came up successfully but only when it uses BasicAuth<br>
rather than SSL auth. I am not sure about what I need to do in<br>
order to get the auth working over SSL again.<br>
<br>
None of the certs are expired when I run getcert list and<br>
ipa-getcert list.<br>
<br>
Since the failure is with attempts to login to LDAP over 636. I<br>
have been attempting to auth to LDAP via port 636 and the ldapsearch<br>
is not completing. When looking at packet captures, I see some the<br>
TCP handshake and what appears to be the start of a SSL process and<br>
then everything hangs.<br>
<br>
What is the proper method to test performing a ldapsearch over 636?<br>
Also, the CS.cfg shows it wants to auth as cn=Directory Manager. I<br>
can successfully auth with cn=Directory Manager over 389 but I think<br>
I am not performing ldapsearch over 636 correctly.<br>
<br>
<br>
<br>
*Mike Plemmons | Senior DevOps Engineer | CROSSCHX<br>
*<br>
614.427.2411<br>
<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx.com</a> <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">www.crosschx.com</a> <<a href="http://www.crosschx.com/" rel="noreferrer" target="_blank">http://www.crosschx.com/</a>><br>
<br>
On Fri, May 5, 2017 at 3:33 PM, Michael Plemmons<br>
<<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@crosschx.com</a><br>
<mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>>> wrote:<br>
<br>
I think I found the email thread. Asking for help with crashed<br>
freeIPA istance. That email pointed to this<br>
link, <a href="https://www.redhat.com/archives/freeipa-users/2017-January/msg00215.html" rel="noreferrer" target="_blank">https://www.redhat.com/archive<wbr>s/freeipa-users/2017-January/<wbr>msg00215.html</a><br>
<<a href="https://www.redhat.com/archives/freeipa-users/2017-January/msg00215.html" rel="noreferrer" target="_blank">https://www.redhat.com/archiv<wbr>es/freeipa-users/2017-January/<wbr>msg00215.html</a>>.<br>
That link talked about changing the CS.cfg file to use port 389<br>
for PKI to auth to LDAP. I made the necessary changes and PKI<br>
came up successfully.<br>
<br>
<br>
<br>
*Mike Plemmons | Senior DevOps Engineer | CROSSCHX<br>
*<br>
614.427.2411<br>
<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx.com</a> <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">www.crosschx.com</a> <<a href="http://www.crosschx.com/" rel="noreferrer" target="_blank">http://www.crosschx.com/</a>><br>
<br>
On Fri, May 5, 2017 at 3:19 PM, Michael Plemmons<br>
<<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@crosschx.com</a><br>
<mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>>> wrote:<br>
<br>
<br>
<br>
<br>
<br>
*Mike Plemmons | Senior DevOps Engineer | CROSSCHX<br>
*<br>
614.427.2411<br>
<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx.com</a> <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">www.crosschx.com</a> <<a href="http://www.crosschx.com/" rel="noreferrer" target="_blank">http://www.crosschx.com/</a>><br>
<br>
On Fri, May 5, 2017 at 3:15 PM, Rob Crittenden<br>
<<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a> <mailto:<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a>>> wrote:<br>
<br>
Michael Plemmons wrote:<br>
> I just realized that I sent the reply directly to Rob<br>
and not to the<br>
> list. My response is inline<br>
<br>
Ok, this is actually good news.<br>
<br>
I made a similar proposal in another case and I was<br>
completely wrong.<br>
Flo had the user do something and it totally fixed their<br>
auth error, I<br>
just can't remember what it was or find the e-mail<br>
thread. I'm pretty<br>
sure it was this calendar year though.<br>
<br>
rob<br>
<br>
<br>
Do you or Flo know what I could search for in the past<br>
emails to find the answer to the problem?<br>
<br>
<br>
<br>
><br>
><br>
><br>
> *Mike Plemmons | Senior DevOps Engineer | CROSSCHX<br>
> *<br>
> 614.427.2411<br>
> <a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx.com</a><br>
<mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
<mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a><br>
<mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>>><br>
> <a href="http://www.crosschx.com" rel="noreferrer" target="_blank">www.crosschx.com</a> <<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">http://www.crosschx.com</a>><br>
<<a href="http://www.crosschx.com/" rel="noreferrer" target="_blank">http://www.crosschx.com/</a>><br>
><br>
> On Thu, May 4, 2017 at 9:39 AM, Michael Plemmons<br>
> <<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@crosschx.com</a><br>
<mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>><br>
<mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a><br>
<mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>>>><br>
> wrote:<br>
><br>
><br>
><br>
><br>
><br>
> *Mike Plemmons | Senior DevOps Engineer | CROSSCHX<br>
> *<br>
> 614.427.2411<br>
> <a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx.com</a><br>
<mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
<mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a><br>
<mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>>><br>
> <a href="http://www.crosschx.com" rel="noreferrer" target="_blank">www.crosschx.com</a> <<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">http://www.crosschx.com</a>><br>
<<a href="http://www.crosschx.com/" rel="noreferrer" target="_blank">http://www.crosschx.com/</a>><br>
><br>
> On Thu, May 4, 2017 at 9:24 AM, Rob Crittenden<br>
<<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a> <mailto:<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a>><br>
> <mailto:<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a><br>
<mailto:<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a>>>> wrote:<br>
><br>
> Michael Plemmons wrote:<br>
> > I realized that I was not very clear in my<br>
statement about<br>
> testing with<br>
> > ldapsearch. I had initially run it without<br>
logging in with a<br>
> DN. I was<br>
> > just running the local ldapsearch -x<br>
command. I then tested on<br>
> > ipa12.mgmt and ipa11.mgmt logging in with a<br>
full DN for the<br>
> admin and<br>
> > "cn=Directory Manager" from ipa12.mgmt<br>
(broken server) and<br>
> ipa11.mgmt<br>
> > and both ldapsearch command succeeded.<br>
> ><br>
> > I ran the following from ipa12.mgmt and<br>
ipa11.mgmt as a non<br>
> root user.<br>
> > I also ran the command showing a line count<br>
for the output and<br>
> the line<br>
> > counts for each were the same when run from<br>
ipa12.mgmt and<br>
> ipa11.mgmt.<br>
> ><br>
> > ldapsearch -LLL -h <a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">ipa12.mgmt.crosschx.com</a><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
> <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.c<wbr>om</a><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>><br>
> > <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
> <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.c<wbr>om</a><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>>> -D "DN" -w PASSWORD -b<br>
> ><br>
"cn=users,cn=accounts,dc=mgmt,<wbr>dc=crosschx,dc=com" dn<br>
> ><br>
> > ldapsearch -LLL -h <a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">ipa12.mgmt.crosschx.com</a><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
> <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.c<wbr>om</a><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>><br>
> > <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
> <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.c<wbr>om</a><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>>> -D "cn=directory<br>
manager" -w<br>
> PASSWORD dn<br>
><br>
> The CA has its own suffix and replication<br>
agreements. Given the auth<br>
> error and recent (5 months) renewal of CA<br>
credentials I'd check<br>
> that the<br>
> CA agent authentication entries are correct.<br>
><br>
> Against each master with a CA run:<br>
><br>
> $ ldapsearch -LLL -x -D 'cn=directory manager'<br>
-W -b<br>
> uid=ipara,ou=people,o=ipaca description<br>
><br>
> The format is 2;serial#,subject,issuer<br>
><br>
> Then on each run:<br>
><br>
> # certutil -L -d /etc/httpd/alias -n ipaCert<br>
|grep Serial<br>
><br>
> The serial # should match that in the<br>
description everywhere.<br>
><br>
> rob<br>
><br>
><br>
><br>
> On the CA (IPA13.MGMT) I ran the ldapsearch<br>
command and see that the<br>
> serial number is 7. I then ran the certutil<br>
command on all three<br>
> servers and the serial number is 7 as well.<br>
><br>
><br>
> I also ran the ldapsearch command against the<br>
other two servers and<br>
> they also showed a serial number of 7.<br>
><br>
><br>
><br>
><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > *Mike Plemmons | Senior DevOps Engineer |<br>
CROSSCHX<br>
> > *<br>
> > 614.427.2411<br>
> > <a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx.com</a><br>
<mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
<mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a><br>
<mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>>><br>
> <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crossch<wbr>x.com</a><br>
<mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
> <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crossch<wbr>x.com</a><br>
<mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>>>><br>
> > <a href="http://www.crosschx.com" rel="noreferrer" target="_blank">www.crosschx.com</a> <<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">http://www.crosschx.com</a>><br>
<<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">http://www.crosschx.com</a>><br>
> <<a href="http://www.crosschx.com/" rel="noreferrer" target="_blank">http://www.crosschx.com/</a>><br>
> ><br>
> > On Wed, May 3, 2017 at 5:28 PM, Michael Plemmons<br>
> > <<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@crosschx.com</a><br>
<mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>><br>
> <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cros<wbr>schx.com</a><br>
<mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>>><br>
> <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cros<wbr>schx.com</a><br>
<mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>><br>
> <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cros<wbr>schx.com</a><br>
<mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>>>>><br>
> > wrote:<br>
> ><br>
> > I have a three node IPA cluster.<br>
> ><br>
> > ipa11.mgmt - was a master over 6 months ago<br>
> > ipa13.mgmt - current master<br>
> > ipa12.mgmt<br>
> ><br>
> > ipa13 has agreements with ipa11 and<br>
ipa12. ipa11 and<br>
> ipa12 do not<br>
> > have agreements between each other.<br>
> ><br>
> > It appears that either ipa12.mgmt lost<br>
some level of its<br>
> replication<br>
> > agreement with ipa13. I saw some level<br>
because users /<br>
> hosts were<br>
> > replicated between all systems but we<br>
started seeing DNS<br>
> was not<br>
> > resolving properly from ipa12. I do not<br>
know when this<br>
> started.<br>
> ><br>
> > When looking at replication agreements<br>
on ipa12 I did not<br>
> see any<br>
> > agreement with ipa13.<br>
> ><br>
> > When I run ipa-replica-manage list all<br>
three hosts show<br>
> has master.<br>
> ><br>
> > When I run ipa-replica-manage ipa11.mgmt<br>
I see ipa13.mgmt<br>
> is a replica.<br>
> ><br>
> > When I run ipa-replica-manage ipa12.mgmt<br>
nothing returned.<br>
> ><br>
> > I ran ipa-replica-manage connect<br>
--cacert=/etc/ipa/ca.crt<br>
> > <a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">ipa12.mgmt.crosschx.com</a><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>><br>
> <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.c<wbr>om</a><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>>><br>
> > <a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">ipa13.mgmt.crosschx.com</a><br>
<<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>><br>
<<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a><br>
<<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>>><br>
> <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.c<wbr>om</a><br>
<<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>><br>
> <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.c<wbr>om</a><br>
<<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>>>> on ipa12.mgmt<br>
> ><br>
> > I then ran the following<br>
> ><br>
> > ipa-replica-manage force-sync --from<br>
> <a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">ipa13.mgmt.crosschx.com</a><br>
<<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>><br>
<<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a><br>
<<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>>><br>
> > <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.c<wbr>om</a><br>
<<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>><br>
> <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.c<wbr>om</a><br>
<<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>>>><br>
> ><br>
> > ipa-replica-manage re-initialize --from<br>
> <a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">ipa13.mgmt.crosschx.com</a><br>
<<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>><br>
<<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a><br>
<<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>>><br>
> > <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.c<wbr>om</a><br>
<<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>><br>
> <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.c<wbr>om</a><br>
<<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>>>><br>
> ><br>
> > I was still seeing bad DNS returns when<br>
dig'ing against<br>
> ipa12.mgmt.<br>
> > I was able to create user and DNS<br>
records and see the<br>
> information<br>
> > replicated properly across all three nodes.<br>
> ><br>
> > I then ran ipactl stop on ipa12.mgmt and<br>
then ipactl start on<br>
> > ipa12.mgmt because I wanted to make sure<br>
everything was<br>
> running<br>
> > fresh after the changes above. While<br>
IPA was staring up (DNS<br>
> > started) we were able to see valid DNS<br>
queries returned but<br>
> > pki-tomcat would not start.<br>
> ><br>
> > I am not sure what I need to do in order<br>
to get this<br>
> working. I<br>
> > have included the output of certutil and<br>
getcert below<br>
> from all<br>
> > three servers as well as the debug<br>
output for pki.<br>
> ><br>
> ><br>
> > While the IPA system is coming up I am<br>
able to<br>
> successfully run<br>
> > ldapsearch -x as the root user and see<br>
results. I am also<br>
> able to<br>
> > login with the "cn=Directory Manager"<br>
account and see results.<br>
> ><br>
> ><br>
> > The debug log shows the following error.<br>
> ><br>
> ><br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]:<br>
> > =============================<wbr>===============<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: ===== DEBUG<br>
> > SUBSYSTEM INITIALIZED =======<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]:<br>
> > =============================<wbr>===============<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> restart at<br>
> > autoShutdown? false<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> > autoShutdown crumb file path?<br>
> ><br>
/var/lib/pki/pki-tomcat/logs/<wbr>autoShutdown.crumb<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> about to<br>
> > look for cert for auto-shutdown<br>
support:auditSigningCert<br>
> cert-pki-ca<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> found<br>
> > cert:auditSigningCert cert-pki-ca<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> done init<br>
> > id=debug<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> > initialized debug<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> > initSubsystem id=log<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> ready to<br>
> > init id=log<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: Creating<br>
> ><br>
><br>
RollingLogFile(/var/lib/pki/pk<wbr>i-tomcat/logs/ca/signedAudit/<wbr>ca_audit)<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: Creating<br>
> ><br>
RollingLogFile(/var/lib/pki/p<wbr>ki-tomcat/logs/ca/system)<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: Creating<br>
> ><br>
RollingLogFile(/var/lib/pki/p<wbr>ki-tomcat/logs/ca/transactions<wbr>)<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> restart at<br>
> > autoShutdown? false<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> > autoShutdown crumb file path?<br>
> ><br>
/var/lib/pki/pki-tomcat/logs/<wbr>autoShutdown.crumb<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> about to<br>
> > look for cert for auto-shutdown<br>
support:auditSigningCert<br>
> cert-pki-ca<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> found<br>
> > cert:auditSigningCert cert-pki-ca<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> done init<br>
> > id=log<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> > initialized log<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> > initSubsystem id=jss<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> ready to<br>
> > init id=jss<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> restart at<br>
> > autoShutdown? false<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> > autoShutdown crumb file path?<br>
> ><br>
/var/lib/pki/pki-tomcat/logs/<wbr>autoShutdown.crumb<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> about to<br>
> > look for cert for auto-shutdown<br>
support:auditSigningCert<br>
> cert-pki-ca<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> found<br>
> > cert:auditSigningCert cert-pki-ca<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> done init<br>
> > id=jss<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> > initialized jss<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> > initSubsystem id=dbs<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: CMSEngine:<br>
> ready to<br>
> > init id=dbs<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]:<br>
> DBSubsystem: init()<br>
> > mEnableSerialMgmt=true<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: Creating<br>
> > LdapBoundConnFactor(DBSubsyst<wbr>em)<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]:<br>
> LdapBoundConnFactory:<br>
> > init<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]:<br>
> > LdapBoundConnFactory:<wbr>doCloning true<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]:<br>
> LdapAuthInfo: init()<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]:<br>
> LdapAuthInfo: init begins<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]:<br>
> LdapAuthInfo: init ends<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]: init: before<br>
> > makeConnection errorIfDown is true<br>
> ><br>
[03/May/2017:21:22:01][localh<wbr>ost-startStop-1]:<br>
makeConnection:<br>
> > errorIfDown true<br>
> ><br>
[03/May/2017:21:22:02][localh<wbr>ost-startStop-1]:<br>
> > <wbr>SSLClientCertificateSelectionC<wbr>B: Setting<br>
desired cert<br>
> nickname to:<br>
> > subsystemCert cert-pki-ca<br>
> ><br>
[03/May/2017:21:22:02][localh<wbr>ost-startStop-1]:<br>
> LdapJssSSLSocket: set<br>
> > client auth cert nickname subsystemCert<br>
cert-pki-ca<br>
> ><br>
[03/May/2017:21:22:02][localh<wbr>ost-startStop-1]:<br>
> > <wbr>SSLClientCertificatSelectionCB<wbr>: Entering!<br>
> ><br>
[03/May/2017:21:22:02][localh<wbr>ost-startStop-1]:<br>
> > <wbr>SSLClientCertificateSelectionC<wbr>B:<br>
returning: null<br>
> ><br>
[03/May/2017:21:22:02][localh<wbr>ost-startStop-1]: SSL<br>
> handshake happened<br>
> > Could not connect to LDAP server host<br>
> <a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">ipa12.mgmt.crosschx.com</a><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>><br>
> > <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.c<wbr>om</a><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
> <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.c<wbr>om</a><br>
<<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>>> port 636 Error<br>
> > netscape.ldap.LDAPException:<br>
Authentication failed (48)<br>
> > at<br>
> ><br>
><br>
com.netscape.cmscore.ldapconn.<wbr>LdapBoundConnFactory.makeConne<wbr>ction(LdapBoundConnFactory.<wbr>java:205)<br>
> > at<br>
> ><br>
><br>
com.netscape.cmscore.ldapconn.<wbr>LdapBoundConnFactory.init(Ldap<wbr>BoundConnFactory.java:166)<br>
> > at<br>
> ><br>
><br>
com.netscape.cmscore.ldapconn.<wbr>LdapBoundConnFactory.init(Ldap<wbr>BoundConnFactory.java:130)<br>
> > at<br>
><br>
com.netscape.cmscore.dbs.DBSu<wbr>bsystem.init(DBSubsystem.java:<wbr>654)<br>
> > at<br>
> ><br>
><br>
com.netscape.cmscore.apps.CMSE<wbr>ngine.initSubsystem(CMSEngine.<wbr>java:1169)<br>
> > at<br>
> ><br>
><br>
com.netscape.cmscore.apps.CMSE<wbr>ngine.initSubsystems(CMSEngine<wbr>.java:1075)<br>
> > at<br>
><br>
com.netscape.cmscore.apps.CMS<wbr>Engine.init(CMSEngine.java:571<wbr>)<br>
> > at<br>
com.netscape.certsrv.apps.CMS.<wbr>init(CMS.java:187)<br>
> > at<br>
com.netscape.certsrv.apps.CMS.<wbr>start(CMS.java:1616)<br>
> > at<br>
> ><br>
><br>
com.netscape.cms.servlet.base.<wbr>CMSStartServlet.init(CMSStartS<wbr>ervlet.java:114)<br>
> > at<br>
><br>
javax.servlet.GenericServlet.<wbr>init(GenericServlet.java:158)<br>
> > at<br>
sun.reflect.NativeMethodAccess<wbr>orImpl.invoke0(Native<br>
> Method)<br>
> > at<br>
> ><br>
><br>
sun.reflect.NativeMethodAccess<wbr>orImpl.invoke(NativeMethodAcce<wbr>ssorImpl.java:62)<br>
> > at<br>
> ><br>
><br>
sun.reflect.DelegatingMethodAc<wbr>cessorImpl.invoke(DelegatingMe<wbr>thodAccessorImpl.java:43)<br>
> > at<br>
java.lang.reflect.Method.invok<wbr>e(Method.java:498)<br>
> > at<br>
> ><br>
><br>
org.apache.catalina.security.S<wbr>ecurityUtil$1.run(SecurityUtil<wbr>.java:288)<br>
> > at<br>
> ><br>
><br>
org.apache.catalina.security.S<wbr>ecurityUtil$1.run(SecurityUtil<wbr>.java:285)<br>
> > at<br>
java.security.AccessController<wbr>.doPrivileged(Native<br>
> Method)<br>
> > at <a href="http://javax.security.auth.Subject.do" rel="noreferrer" target="_blank">javax.security.auth.Subject.do</a><br>
<<a href="http://javax.security.auth.Subject.do" rel="noreferrer" target="_blank">http://javax.security.auth.Su<wbr>bject.do</a>><br>
> <<a href="http://javax.security.auth.Subject.do" rel="noreferrer" target="_blank">http://javax.security.auth.S<wbr>ubject.do</a><br>
<<a href="http://javax.security.auth.Subject.do" rel="noreferrer" target="_blank">http://javax.security.auth.Su<wbr>bject.do</a>>>AsPrivileged(Subject<wbr>.java:549)<br>
> > at<br>
> ><br>
><br>
org.apache.catalina.security.S<wbr>ecurityUtil.execute(SecurityUt<wbr>il.java:320)<br>
> > at<br>
> ><br>
><br>
org.apache.catalina.security.S<wbr>ecurityUtil.doAsPrivilege(Secu<wbr>rityUtil.java:175)<br>
> > at<br>
> ><br>
><br>
org.apache.catalina.security.S<wbr>ecurityUtil.doAsPrivilege(Secu<wbr>rityUtil.java:124)<br>
> > at<br>
> ><br>
><br>
org.apache.catalina.core.Stand<wbr>ardWrapper.initServlet(Standar<wbr>dWrapper.java:1270)<br>
> > at<br>
> ><br>
><br>
org.apache.catalina.core.Stand<wbr>ardWrapper.loadServlet(Standar<wbr>dWrapper.java:1195)<br>
> > at<br>
> ><br>
><br>
org.apache.catalina.core.Stand<wbr>ardWrapper.load(StandardWrappe<wbr>r.java:1085)<br>
> > at<br>
> ><br>
><br>
org.apache.catalina.core.Stand<wbr>ardContext.loadOnStartup(Stand<wbr>ardContext.java:5318)<br>
> > at<br>
> ><br>
><br>
org.apache.catalina.core.Stand<wbr>ardContext.startInternal(Stand<wbr>ardContext.java:5610)<br>
> > at<br>
> ><br>
><br>
org.apache.catalina.util.Lifec<wbr>ycleBase.start(LifecycleBase.<wbr>java:147)<br>
> > at<br>
> ><br>
><br>
org.apache.catalina.core.Conta<wbr>inerBase.addChildInternal(Cont<wbr>ainerBase.java:899)<br>
> > at<br>
> ><br>
><br>
org.apache.catalina.core.Conta<wbr>inerBase.access$000(ContainerB<wbr>ase.java:133)<br>
> > at<br>
> ><br>
><br>
org.apache.catalina.core.Conta<wbr>inerBase$PrivilegedAddChild.<wbr>run(ContainerBase.java:156)<br>
> > at<br>
> ><br>
><br>
org.apache.catalina.core.Conta<wbr>inerBase$PrivilegedAddChild.<wbr>run(ContainerBase.java:145)<br>
> > at<br>
java.security.AccessController<wbr>.doPrivileged(Native</blockquote>
</blockquote></div><br></div></div>