<div dir="ltr">SOLVED!<div><br></div><div>Thank you Flo!  That did the trick.  Once I made the change to the certificate and restarted the IPA services everything came back up like it was supposed to.</div><div><br></div><div>High five!</div><div><div class="gmail_extra"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr" style="font-size:12.8px"><br></div><div dir="ltr"><b style="font-size:12.8px"><font size="2">Mike Plemmons | Senior DevOps Engineer | CROSSCHX<br></font></b><div>614.427.2411</div><div><a href="mailto:mike.plemmons@crosschx.com" style="font-size:12.8px" target="_blank">mike.plemmons@crosschx.com</a><br></div><div style="font-size:12.8px"><a href="http://www.crosschx.com/" target="_blank">www.crosschx.com</a></div></div></div></div></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Thu, May 18, 2017 at 10:28 AM, Florence Blanc-Renaud <span dir="ltr"><<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 05/18/2017 03:49 PM, Michael Plemmons wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
<br>
<br>
*Mike Plemmons | Senior DevOps Engineer | CROSSCHX<br>
*<br>
614.427.2411<br>
<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx.com</a> <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">www.crosschx.com</a> <<a href="http://www.crosschx.com/" rel="noreferrer" target="_blank">http://www.crosschx.com/</a>><br>
<br>
On Thu, May 18, 2017 at 8:02 AM, Florence Blanc-Renaud <<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a><br>
<mailto:<a href="mailto:flo@redhat.com" target="_blank">flo@redhat.com</a>>> wrote:<br>
<br>
    On 05/15/2017 08:33 PM, Michael Plemmons wrote:<br>
<br>
        I have done more searching in my logs and I see the following<br>
        errors.<br>
<br>
        This is in the localhost log file /var/lib/pki/pki-tomcat/logs<br>
<br>
        May 15, 2017 3:08:08 PM<br>
        org.apache.catalina.core.Appli<wbr>cationContext log<br>
        SEVERE: StandardWrapper.Throwable<br>
        java.lang.NullPointerException<br>
<br>
        May 15, 2017 3:08:08 PM org.apache.catalina.core.Stand<wbr>ardContext<br>
        loadOnStartup<br>
        SEVERE: Servlet [castart] in web application [/ca] threw load()<br>
        exception<br>
        java.lang.NullPointerException<br>
<br>
        May 15, 2017 3:08:09 PM<br>
        org.apache.catalina.core.Stand<wbr>ardHostValve invoke<br>
        SEVERE: Exception Processing /ca/admin/ca/getStatus<br>
        <a href="http://javax.ws.rs" rel="noreferrer" target="_blank">javax.ws.rs</a> <<a href="http://javax.ws.rs" rel="noreferrer" target="_blank">http://javax.ws.rs</a>><br>
        <<a href="http://javax.ws.rs" rel="noreferrer" target="_blank">http://javax.ws.rs</a>>.ServiceUn<wbr>availableException: Subsystem<br>
        unavailable<br>
<br>
<br>
        Looking at the debug log it says Authentication failed for port 636.<br>
<br>
        [15/May/2017:17:39:25][localho<wbr>st-startStop-1]: LdapAuthInfo: init()<br>
        [15/May/2017:17:39:25][localho<wbr>st-startStop-1]: LdapAuthInfo:<br>
        init begins<br>
        [15/May/2017:17:39:25][localho<wbr>st-startStop-1]: LdapAuthInfo:<br>
        init ends<br>
        [15/May/2017:17:39:25][localho<wbr>st-startStop-1]: init: before<br>
        makeConnection errorIfDown is true<br>
        [15/May/2017:17:39:25][localho<wbr>st-startStop-1]: makeConnection:<br>
        errorIfDown true<br>
        [15/May/2017:17:39:25][localho<wbr>st-startStop-1]:<br>
        SSLClientCertificateSelectionC<wbr>B: Setting desired cert nickname to:<br>
        subsystemCert cert-pki-ca<br>
        [15/May/2017:17:39:25][localho<wbr>st-startStop-1]: LdapJssSSLSocket: set<br>
        client auth cert nickname subsystemCert cert-pki-ca<br>
        [15/May/2017:17:39:25][localho<wbr>st-startStop-1]:<br>
        SSLClientCertificatSelectionCB<wbr>: Entering!<br>
        [15/May/2017:17:39:25][localho<wbr>st-startStop-1]:<br>
        SSLClientCertificateSelectionC<wbr>B: returning: null<br>
        [15/May/2017:17:39:25][localho<wbr>st-startStop-1]: SSL handshake<br>
        happened<br>
        Could not connect to LDAP server host <a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">ipa12.mgmt.crosschx.com</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>> port 636 Error<br>
        netscape.ldap.LDAPException: Authentication failed (48)<br>
                at<br>
        com.netscape.cmscore.ldapconn.<wbr>LdapBoundConnFactory.makeConne<wbr>ction(LdapBoundConnFactory.<wbr>java:205)<br>
<br>
<br>
        I looked at the validity of the cert it mentions and it is fine.<br>
<br>
        (root)>getcert status -v -d /etc/pki/pki-tomcat/alias -n<br>
        'subsystemCert<br>
        cert-pki-ca'<br>
        State MONITORING, stuck: no.<br>
<br>
<br>
        I then looked at the ldap errors around the time of this failure<br>
        and I<br>
        am seeing this log entry.<br>
<br>
<br>
        [15/May/2017:17:38:42.06308075<wbr>8 +0000] set_krb5_creds - Could<br>
        not get<br>
        initial credentials for principal<br>
        [ldap/<a href="mailto:ipa12.mgmt.crosschx.com@MGMT.CROSSCHX.COM" target="_blank">ipa12.mgmt.crosschx.com@<wbr>MGMT.CROSSCHX.COM</a><br>
        <mailto:<a href="mailto:ipa12.mgmt.crosschx.com@MGMT.CROSSCHX.COM" target="_blank">ipa12.mgmt.crosschx.co<wbr>m@MGMT.CROSSCHX.COM</a>><br>
        <mailto:<a href="mailto:ipa12.mgmt.crosschx.com@MGMT.CROSSCHX.COM" target="_blank">ipa12.mgmt.crosschx.co<wbr>m@MGMT.CROSSCHX.COM</a><br>
        <mailto:<a href="mailto:ipa12.mgmt.crosschx.com@MGMT.CROSSCHX.COM" target="_blank">ipa12.mgmt.crosschx.co<wbr>m@MGMT.CROSSCHX.COM</a>>>] in keytab<br>
        [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any<br>
        KDC for<br>
        requested realm)<br>
<br>
        When I perform a klist against that keytab nothing appears out<br>
        of the<br>
        ordinary compared to working IPA servers.<br>
<br>
        I am not sure what to look at next.<br>
<br>
<br>
    Hi,<br>
<br>
    you can try the following to manually replay the connection<br>
    established by Dogtag to LDAP server:<br>
<br>
    root$ export LDAPTLS_CACERTDIR=/etc/pki/pki<wbr>-tomcat/alias<br>
    root$ export LDAPTLS_CERT='subsystemCert cert-pki-ca'<br>
<br>
    The above commands specify the NSSDB containing the user certificate<br>
    and its name for SASL-EXTERNAL authentication.<br>
<br>
    Then note the value obtained below as it will be used for the next<br>
    step as the password to access the private key in the NSSDB:<br>
    root$ grep internal /etc/pki/pki-tomcat/password.c<wbr>onf<br>
    internal=<some value><br>
<br>
    root$ ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL<br>
    -Q -LLL dn namingcontexts<br>
    Please enter pin, password, or pass phrase for security token<br>
    'ldap(0)':                        <<<< here supply the value found above<br>
    dn:<br>
    namingcontexts: cn=changelog<br>
    namingcontexts: dc=ipadomain,dc=com<br>
    namingcontexts: o=ipaca<br>
<br>
<br>
<br>
So I guess I found my problem.<br>
<br>
(root)>ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL -Q<br>
-LLL dn namingcontexts<br>
Please enter pin, password, or pass phrase for security token 'ldap(0)':<br>
ldap_sasl_interactive_bind_s: Can't contact LDAP server (-1)<br>
  additional info: TLS error -12195:Peer does not recognize and trust<br>
the CA that issued your certificate.<br>
<br>
<br>
I looked at our certs in /etc/dirsrv/slapd-IPADOMAIN-CO<wbr>M and found the<br>
following.<br>
<br>
IPA12 - problem server<br>
(root)>certutil -L -d /etc/dirsrv/slapd-IPADOMAIN-CO<wbr>M<br>
<br>
Certificate Nickname                                         Trust<br>
Attributes<br>
<br>
 SSL,S/MIME,JAR/XPI<br>
<br>
Server-Cert                                                  u,u,u<br>
IPADOMAIN-COM IPA CA                                     C,,<br>
<br>
<br>
<br>
IPA11/IPA13 - 11 was the master and 13 is the new master<br>
(root)>certutil -L -d /etc/dirsrv/slapd-IPADOMAIN-CO<wbr>M<br>
<br>
Certificate Nickname                                         Trust<br>
Attributes<br>
<br>
 SSL,S/MIME,JAR/XPI<br>
<br>
Server-Cert                                                  u,u,u<br>
IPADOMAIN-COM IPA CA                                     CT,C,C<br>
<br>
<br>
<br>
</blockquote>
Good news! In this case the fix is trivial:<br>
root$ certutil -M -d /etc/dirsrv/slapd-IPADOMAIN-CO<wbr>M -n 'IPADOMAIN-COM IPA CA' -t CT,C,C<br>
<br>
Flo.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
    In the LDAP server access log (in<br>
    /etc/dirsrv/slapd-IPADOMAIN.CO<wbr>M/access), you should see the<br>
    corresponding connection:<br>
<br>
    [18/May/2017:13:35:14.82209041<wbr>7 +0200] conn=297 fd=150 slot=150 SSL<br>
    connection from xxx to yyy<br>
    [18/May/2017:13:35:15.78941401<wbr>7 +0200] conn=297 TLS1.2 128-bit<br>
    AES-GCM; client CN=CA Subsystem,O=<a href="http://IPADOMAIN.COM" rel="noreferrer" target="_blank">IPADOMAIN.COM</a><br>
    <<a href="http://IPADOMAIN.COM" rel="noreferrer" target="_blank">http://IPADOMAIN.COM</a>>; issuer CN=Certificate<br>
    Authority,O=<a href="http://IPADOMAIN.COM" rel="noreferrer" target="_blank">IPADOMAIN.COM</a> <<a href="http://IPADOMAIN.COM" rel="noreferrer" target="_blank">http://IPADOMAIN.COM</a>><br>
    [18/May/2017:13:35:15.79310850<wbr>9 +0200] conn=297 TLS1.2 client bound<br>
    as uid=pkidbuser,ou=people,o=ipac<wbr>a<br>
    [18/May/2017:13:35:15.79810150<wbr>5 +0200] conn=297 op=0 BIND dn=""<br>
    method=sasl version=3 mech=EXTERNAL<br>
    [18/May/2017:13:35:15.80032207<wbr>6 +0200] conn=297 op=0 RESULT err=0<br>
    tag=97 nentries=0 etime=0 dn="uid=pkidbuser,ou=people,o=<wbr>ipaca"<br>
<br>
    HTH,<br>
    Flo.<br>
<br>
<br>
<br>
<br>
        *Mike Plemmons | Senior DevOps Engineer | CROSSCHX<br>
        *<br>
        614.427.2411<br>
        <a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx.com</a> <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>>><br>
        <a href="http://www.crosschx.com" rel="noreferrer" target="_blank">www.crosschx.com</a> <<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">http://www.crosschx.com</a>><br>
        <<a href="http://www.crosschx.com/" rel="noreferrer" target="_blank">http://www.crosschx.com/</a>><br>
<br>
        On Wed, May 10, 2017 at 3:35 PM, Michael Plemmons<br>
        <<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@crosschx.com</a><br>
        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>><br>
        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a><br>
        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>>>><br>
        wrote:<br>
<br>
            The PKI service came up successfully but only when it uses<br>
        BasicAuth<br>
            rather than SSL auth.  I am not sure about what I need to do in<br>
            order to get the auth working over SSL again.<br>
<br>
            None of the certs are expired when I run getcert list and<br>
            ipa-getcert list.<br>
<br>
            Since the failure is with attempts to login to LDAP over 636.  I<br>
            have been attempting to auth to LDAP via port 636 and the<br>
        ldapsearch<br>
            is not completing.  When looking at packet captures, I see<br>
        some the<br>
            TCP handshake and what appears to be the start of a SSL<br>
        process and<br>
            then everything hangs.<br>
<br>
            What is the proper method to test performing a ldapsearch<br>
        over 636?<br>
            Also, the CS.cfg shows it wants to auth as cn=Directory<br>
        Manager.  I<br>
            can successfully auth with cn=Directory Manager over 389 but<br>
        I think<br>
            I am not performing ldapsearch over 636 correctly.<br>
<br>
<br>
<br>
            *Mike Plemmons | Senior DevOps Engineer | CROSSCHX<br>
            *<br>
            614.427.2411<br>
            <a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>>><br>
            <a href="http://www.crosschx.com" rel="noreferrer" target="_blank">www.crosschx.com</a> <<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">http://www.crosschx.com</a>><br>
        <<a href="http://www.crosschx.com/" rel="noreferrer" target="_blank">http://www.crosschx.com/</a>><br>
<br>
            On Fri, May 5, 2017 at 3:33 PM, Michael Plemmons<br>
            <<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@crosschx.com</a><br>
        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>><br>
            <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a><br>
        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>>>> wrote:<br>
<br>
                I think I found the email thread.  Asking for help with<br>
        crashed<br>
                freeIPA istance.  That email pointed to this<br>
                link,<br>
        <a href="https://www.redhat.com/archives/freeipa-users/2017-January/msg00215.html" rel="noreferrer" target="_blank">https://www.redhat.com/archive<wbr>s/freeipa-users/2017-January/<wbr>msg00215.html</a><br>
        <<a href="https://www.redhat.com/archives/freeipa-users/2017-January/msg00215.html" rel="noreferrer" target="_blank">https://www.redhat.com/archiv<wbr>es/freeipa-users/2017-January/<wbr>msg00215.html</a>><br>
<br>
        <<a href="https://www.redhat.com/archives/freeipa-users/2017-January/msg00215.html" rel="noreferrer" target="_blank">https://www.redhat.com/archiv<wbr>es/freeipa-users/2017-January/<wbr>msg00215.html</a><br>
        <<a href="https://www.redhat.com/archives/freeipa-users/2017-January/msg00215.html" rel="noreferrer" target="_blank">https://www.redhat.com/archiv<wbr>es/freeipa-users/2017-January/<wbr>msg00215.html</a>>>.<br>
                That link talked about changing the CS.cfg file to use<br>
        port 389<br>
                for PKI to auth to LDAP.  I made the necessary changes<br>
        and PKI<br>
                came up successfully.<br>
<br>
<br>
<br>
                *Mike Plemmons | Senior DevOps Engineer | CROSSCHX<br>
                *<br>
                614.427.2411<br>
                <a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>>><br>
                <a href="http://www.crosschx.com" rel="noreferrer" target="_blank">www.crosschx.com</a> <<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">http://www.crosschx.com</a>><br>
        <<a href="http://www.crosschx.com/" rel="noreferrer" target="_blank">http://www.crosschx.com/</a>><br>
<br>
                On Fri, May 5, 2017 at 3:19 PM, Michael Plemmons<br>
                <<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@crosschx.com</a><br>
        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>><br>
                <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a><br>
        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>>>> wrote:<br>
<br>
<br>
<br>
<br>
<br>
                    *Mike Plemmons | Senior DevOps Engineer | CROSSCHX<br>
                    *<br>
                    614.427.2411<br>
                    <a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>>><br>
                    <a href="http://www.crosschx.com" rel="noreferrer" target="_blank">www.crosschx.com</a> <<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">http://www.crosschx.com</a>><br>
        <<a href="http://www.crosschx.com/" rel="noreferrer" target="_blank">http://www.crosschx.com/</a>><br>
<br>
                    On Fri, May 5, 2017 at 3:15 PM, Rob Crittenden<br>
                    <<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a> <mailto:<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a>><br>
        <mailto:<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a> <mailto:<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a>>>> wrote:<br>
<br>
                        Michael Plemmons wrote:<br>
                        > I just realized that I sent the reply directly<br>
        to Rob<br>
                        and not to the<br>
                        > list.  My response is inline<br>
<br>
                        Ok, this is actually good news.<br>
<br>
                        I made a similar proposal in another case and I was<br>
                        completely wrong.<br>
                        Flo had the user do something and it totally<br>
        fixed their<br>
                        auth error, I<br>
                        just can't remember what it was or find the e-mail<br>
                        thread. I'm pretty<br>
                        sure it was this calendar year though.<br>
<br>
                        rob<br>
<br>
<br>
                    Do you or Flo know what I could search for in the past<br>
                    emails to find the answer to the problem?<br>
<br>
<br>
<br>
                        ><br>
                        ><br>
                        ><br>
                        > *Mike Plemmons | Senior DevOps Engineer | CROSSCHX<br>
                        > *<br>
                        > 614.427.2411<br>
                        > <a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
                        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>>><br>
                        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
                        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>>>><br>
                        > <a href="http://www.crosschx.com" rel="noreferrer" target="_blank">www.crosschx.com</a> <<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">http://www.crosschx.com</a>><br>
        <<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">http://www.crosschx.com</a>><br>
                        <<a href="http://www.crosschx.com/" rel="noreferrer" target="_blank">http://www.crosschx.com/</a>><br>
                        ><br>
                        > On Thu, May 4, 2017 at 9:39 AM, Michael Plemmons<br>
                        > <<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@crosschx.com</a><br>
        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>><br>
                        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a><br>
        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>>><br>
                        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a><br>
        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>><br>
                        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a><br>
        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>>>>><br>
                        > wrote:<br>
                        ><br>
                        ><br>
                        ><br>
                        ><br>
                        ><br>
                        >     *Mike Plemmons | Senior DevOps Engineer |<br>
        CROSSCHX<br>
                        >     *<br>
                        >     614.427.2411<br>
                        >     <a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
                        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>>><br>
                        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
                        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>>>><br>
                        >     <a href="http://www.crosschx.com" rel="noreferrer" target="_blank">www.crosschx.com</a> <<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">http://www.crosschx.com</a>><br>
        <<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">http://www.crosschx.com</a>><br>
                        <<a href="http://www.crosschx.com/" rel="noreferrer" target="_blank">http://www.crosschx.com/</a>><br>
                        ><br>
                        >     On Thu, May 4, 2017 at 9:24 AM, Rob Crittenden<br>
                        <<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a><br>
        <mailto:<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a>> <mailto:<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a><br>
        <mailto:<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a>>><br>
                        >     <mailto:<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a><br>
        <mailto:<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a>><br>
                        <mailto:<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a><br>
        <mailto:<a href="mailto:rcritten@redhat.com" target="_blank">rcritten@redhat.com</a>>>><wbr>> wrote:<br>
                        ><br>
                        >         Michael Plemmons wrote:<br>
                        >         > I realized that I was not very clear<br>
        in my<br>
                        statement about<br>
                        >         testing with<br>
                        >         > ldapsearch.  I had initially run it<br>
        without<br>
                        logging in with a<br>
                        >         DN.  I was<br>
                        >         > just running the local ldapsearch -x<br>
                        command.  I then tested on<br>
                        >         > ipa12.mgmt and ipa11.mgmt logging in<br>
        with a<br>
                        full DN for the<br>
                        >         admin and<br>
                        >         > "cn=Directory Manager" from ipa12.mgmt<br>
                        (broken server) and<br>
                        >         ipa11.mgmt<br>
                        >         > and both ldapsearch command succeeded.<br>
                        >         ><br>
                        >         > I ran the following from ipa12.mgmt and<br>
                        ipa11.mgmt as a non<br>
                        >         root user.<br>
                        >         > I also ran the command showing a<br>
        line count<br>
                        for the output and<br>
                        >         the line<br>
                        >         > counts for each were the same when<br>
        run from<br>
                        ipa12.mgmt and<br>
                        >         ipa11.mgmt.<br>
                        >         ><br>
                        >         > ldapsearch -LLL -h<br>
        <a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">ipa12.mgmt.crosschx.com</a> <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>><br>
                        >         <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.c<wbr>om</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>>><br>
                        >         > <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>><br>
                        >         <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.c<wbr>om</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>>>> -D "DN" -w PASSWORD -b<br>
                        >         ><br>
                        "cn=users,cn=accounts,dc=mgmt,<wbr>dc=crosschx,dc=com" dn<br>
                        >         ><br>
                        >         > ldapsearch -LLL -h<br>
        <a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">ipa12.mgmt.crosschx.com</a> <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>><br>
                        >         <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.c<wbr>om</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>>><br>
                        >         > <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>><br>
                        >         <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.c<wbr>om</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>>>> -D "cn=directory<br>
                        manager" -w<br>
                        >         PASSWORD dn<br>
                        ><br>
                        >         The CA has its own suffix and replication<br>
                        agreements. Given the auth<br>
                        >         error and recent (5 months) renewal of CA<br>
                        credentials I'd check<br>
                        >         that the<br>
                        >         CA agent authentication entries are<br>
        correct.<br>
                        ><br>
                        >         Against each master with a CA run:<br>
                        ><br>
                        >         $ ldapsearch -LLL -x -D 'cn=directory<br>
        manager'<br>
                        -W -b<br>
                        >         uid=ipara,ou=people,o=ipaca description<br>
                        ><br>
                        >         The format is 2;serial#,subject,issuer<br>
                        ><br>
                        >         Then on each run:<br>
                        ><br>
                        >         # certutil -L -d /etc/httpd/alias -n<br>
        ipaCert<br>
                        |grep Serial<br>
                        ><br>
                        >         The serial # should match that in the<br>
                        description everywhere.<br>
                        ><br>
                        >         rob<br>
                        ><br>
                        ><br>
                        ><br>
                        >     On the CA (IPA13.MGMT) I ran the ldapsearch<br>
                        command and see that the<br>
                        >     serial number is 7.  I then ran the certutil<br>
                        command on all three<br>
                        >     servers and the serial number is 7 as well.<br>
                        ><br>
                        ><br>
                        >     I also ran the ldapsearch command against the<br>
                        other two servers and<br>
                        >     they also showed a serial number of 7.<br>
                        ><br>
                        ><br>
                        ><br>
                        ><br>
                        >         ><br>
                        >         ><br>
                        >         ><br>
                        >         ><br>
                        >         ><br>
                        >         > *Mike Plemmons | Senior DevOps<br>
        Engineer |<br>
                        CROSSCHX<br>
                        >         > *<br>
                        >         > 614.427.2411<br>
                        >         > <a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
                        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>>><br>
                        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
                        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>>>><br>
                        >         <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crossch<wbr>x.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
                        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>>><br>
                        >         <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crossch<wbr>x.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>><br>
                        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a><br>
        <mailto:<a href="mailto:mike.plemmons@crosschx.com" target="_blank">mike.plemmons@crosschx<wbr>.com</a>>>>><br>
                        >         > <a href="http://www.crosschx.com" rel="noreferrer" target="_blank">www.crosschx.com</a><br>
        <<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">http://www.crosschx.com</a>> <<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">http://www.crosschx.com</a>><br>
                        <<a href="http://www.crosschx.com" rel="noreferrer" target="_blank">http://www.crosschx.com</a>><br>
                        >         <<a href="http://www.crosschx.com/" rel="noreferrer" target="_blank">http://www.crosschx.com/</a>><br>
                        >         ><br>
                        >         > On Wed, May 3, 2017 at 5:28 PM,<br>
        Michael Plemmons<br>
                        >         > <<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@crosschx.com</a><br>
        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>><br>
                        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a><br>
        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>>><br>
                        >         <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cros<wbr>schx.com</a><br>
        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>><br>
                        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a><br>
        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>>>><br>
                        >         <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cros<wbr>schx.com</a><br>
        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>><br>
                        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a><br>
        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>>><br>
                        >         <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cros<wbr>schx.com</a><br>
        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>><br>
                        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a><br>
        <mailto:<a href="mailto:michael.plemmons@crosschx.com" target="_blank">michael.plemmons@cross<wbr>chx.com</a>>>>>><br>
                        >         > wrote:<br>
                        >         ><br>
                        >         >     I have a three node IPA cluster.<br>
                        >         ><br>
                        >         >     ipa11.mgmt - was a master over 6<br>
        months ago<br>
                        >         >     ipa13.mgmt - current master<br>
                        >         >     ipa12.mgmt<br>
                        >         ><br>
                        >         >     ipa13 has agreements with ipa11 and<br>
                        ipa12.  ipa11 and<br>
                        >         ipa12 do not<br>
                        >         >     have agreements between each other.<br>
                        >         ><br>
                        >         >     It appears that either<br>
        ipa12.mgmt lost<br>
                        some level of its<br>
                        >         replication<br>
                        >         >     agreement with ipa13.  I saw<br>
        some level<br>
                        because users /<br>
                        >         hosts were<br>
                        >         >     replicated between all systems<br>
        but we<br>
                        started seeing DNS<br>
                        >         was not<br>
                        >         >     resolving properly from ipa12.<br>
        I do not<br>
                        know when this<br>
                        >         started.<br>
                        >         ><br>
                        >         >     When looking at replication<br>
        agreements<br>
                        on ipa12 I did not<br>
                        >         see any<br>
                        >         >     agreement with ipa13.<br>
                        >         ><br>
                        >         >     When I run ipa-replica-manage<br>
        list all<br>
                        three hosts show<br>
                        >         has master.<br>
                        >         ><br>
                        >         >     When I run ipa-replica-manage<br>
        ipa11.mgmt<br>
                        I see ipa13.mgmt<br>
                        >         is a replica.<br>
                        >         ><br>
                        >         >     When I run ipa-replica-manage<br>
        ipa12.mgmt<br>
                        nothing returned.<br>
                        >         ><br>
                        >         >     I ran ipa-replica-manage connect<br>
                        --cacert=/etc/ipa/ca.crt<br>
                        >         >     <a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">ipa12.mgmt.crosschx.com</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>><br>
                        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>>><br>
                        >         <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.c<wbr>om</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>><br>
                        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa12.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa12.mgmt.crosschx.co<wbr>m</a>>>>><br>
                        >         >     <a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">ipa13.mgmt.crosschx.com</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>>><br>
                        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>>>><br>
                        >         <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.c<wbr>om</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>>><br>
                        >         <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.c<wbr>om</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>>>>> on ipa12.mgmt<br>
                        >         ><br>
                        >         >     I then ran the following<br>
                        >         ><br>
                        >         >     ipa-replica-manage force-sync --from<br>
                        >         <a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">ipa13.mgmt.crosschx.com</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>>><br>
                        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>>>><br>
                        >         >     <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.c<wbr>om</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>>><br>
                        >         <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.c<wbr>om</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>>>>><br>
                        >         ><br>
                        >         >     ipa-replica-manage re-initialize<br>
        --from<br>
                        >         <a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">ipa13.mgmt.crosschx.com</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>>><br>
                        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>>>><br>
                        >         >     <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.c<wbr>om</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>>><br>
                        >         <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.c<wbr>om</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>><br>
                        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a><br>
        <<a href="http://ipa13.mgmt.crosschx.com" rel="noreferrer" target="_blank">http://ipa13.mgmt.crosschx.co<wbr>m</a>>>>><br>
                        >         ><br>
                        >         >     I was still seeing bad DNS<br>
        returns when<br>
                        dig'ing against<br>
                        >         ipa12.mgmt.<br>
                        >         >     I was able to create user and DNS<br>
                        records and see the<br>
                        >         information<br>
                        >         >     replicated properly across all<br>
        three nodes.<br>
                        >         ><br>
                        >         >     I then ran ipactl stop on<br>
        ipa12.mgmt and<br>
                        then ipactl start on</blockquote>
</blockquote></div><br></div></div></div>