<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">just now getting back to this, have been truncating httpd logs via cron since then to keep from filing up my disk.<div class=""><br class=""></div><div class="">So, does this ring any bells :)</div><div class=""><br class=""></div><div class=""><div class="">[Wed Dec 14 00:38:39 2016] [error] ipa: DEBUG: IPA: virtual verify retrieve certificate</div><div class="">[Wed Dec 14 00:38:39 2016] [error] ipa: DEBUG: Not granted by ACI to retrieve certificate, looking at principal</div><div class="">[Wed Dec 14 00:38:39 2016] [error] ipa: INFO: <a href="mailto:host/phoenix-168.nym1.placeiq.net@placeiq.net" class="">host/phoenix-168.nym1.placeiq.net@PLACEIQ.NET</a>: cert_request(u'MIID4zCCAssCAQAwPTEUMBIGA1UEChMLUExBQ0VJUS5ORVQxJTAjBgNVBAMTHHBob2VuaXgtMTY4Lm55bTEucGxhY2VpcS5uZXQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDboyDDyTaP4+LxYThfy1w7C67TN2qBp8JjA7Y55gnthyUp/PUUXmm3FUQ4yG4cBqDSxZcUWzl+eA33/ceSIp0HtVl34ZkkUitY1hmUvu2nh16ReR65YC+qRqkAIypOilLdPzXIWZ7u5LbM/Xpj3/Npzm18UupAAznNXkZnojuPmAQ+ulPGypyefLnhr5my6hIaR1atTm1ZvHTG3raMJOJhFe4jMeI/tgPVdNCUfOUdEKhmmCm5KivxhTtHMEcH+8obuQnbaSokJscxlzHBLiyQGqhAn5gznXg0mlVCC0H4mwdB1g2g9cG+SxSua/7XCm+AnGXlc75MZAt594QBTc3fAgMBAAGgggFfMHsGCSqGSIb3DQEJFDFuHmwASQBQAEEAIABNAGEAYwBoAGkAbgBlACAAQwBlAHIAdABpAGYAaQBjAGEAdABlACAALQAgAHAAaABvAGUAbgBpAHgALQAxADYAOAAuAG4AeQBtADEALgBwAGwAYQBjAGUAaQBxAC4AbgBlAHQwgd8GCSqGSIb3DQEJDjGB0TCBzjCBmwYDVR0RAQEABIGQMIGNoD0GCisGAQQBgjcUAgOgLwwtaG9zdC9waG9lbml4LTE2OC5ueW0xLnBsYWNlaXEubmV0QFBMQUNFSVEuTkVUoEwGBisGAQUCAqBCMECgDRsLUExBQ0VJUS5ORVShLzAtoAMCAQGhJjAkGwRob3N0GxxwaG9lbml4LTE2OC5ueW0xLnBsYWNlaXEubmV0MAwGA1UdEwEB/wQCMAAwIAYDVR0OAQEABBYEFKDhj0rZ6mZwJfrJx3pCI12dct5WMA0GCSqGSIb3DQEBCwUAA4IBAQC/zeFtiiiJXi9bws2NWx2u/vB7aTVRci2yb9wb70dUzcpeJ3qRTqsE5I+D3MHxLYnixrG4iMEaWgEyuJy4SIWqW1YVSrHwhJZufyRnxy7luNXON+TBBBI0Gro5gICy9XiDKbA/q6clYzEbwDLe6Es/U5/h4Bl/ziV61KYCJN0P1bMWI21A73iP3EQHx2lefYxI8BJN68hJLQiK+E0IWGCqfvipTHA0bHYSQy6WFmmS96v0wr93jy783iromycZeXWzFJyx+LruVPmPOewmUOJ2Y2NJNPJv35pPYUw5Dt2hlLgWZ18po8C+XYqvMbzxM2DFlxMX3Ff+ZiwCRYSGknFI', <a href="mailto:principal=u'host/phoenix-168.nym1.placeiq.net@placeiq.net" class="">principal=u'host/phoenix-168.nym1.placeiq.net@PLACEIQ.NET</a>', add=True): ACIError</div><div class="">[Wed Dec 14 00:38:39 2016] [error] ipa: DEBUG: response: ACIError: Insufficient access: Gettext('not allowed to perform this command', domain='ipa', localedir=None)</div><div class="">[Wed Dec 14 00:38:39 2016] [error] ipa: DEBUG: no session id in request, generating empty session data with id=9beb89831ebfca453453ad48feaaa4d0</div><div class="">[Wed Dec 14 00:38:39 2016] [error] ipa: DEBUG: store session: session_id=9beb89831ebfca453453ad48feaaa4d0 start_timestamp=2016-12-14T00:38:39 access_timestamp=2016-12-14T00:38:39 expiration_timestamp=1970-01-01T00:00:00</div><div class="">[Wed Dec 14 00:38:39 2016] [error] ipa: DEBUG: finalize_kerberos_acquisition: xmlserver ccache_name="FILE:/tmp/krb5cc_apache_SQg9kf" session_id="9beb89831ebfca453453ad48feaaa4d0"</div><div class="">[Wed Dec 14 00:38:39 2016] [error] ipa: DEBUG: reading ccache data from file "/tmp/krb5cc_apache_SQg9kf"</div><div class="">[Wed Dec 14 00:38:39 2016] [error] ipa: DEBUG: get_credential_times: <a href="mailto:principal=krbtgt/PLACEIQ.NET@placeiq.net" class="">principal=krbtgt/PLACEIQ.NET@PLACEIQ.NET</a>, authtime=12/14/16 00:38:36, starttime=12/14/16 00:38:37, endtime=12/15/16 00:38:36, renew_till=01/01/70 00:00:00</div><div class="">[Wed Dec 14 00:38:39 2016] [error] ipa: DEBUG: KRB5_CCache FILE:/tmp/krb5cc_apache_SQg9kf endtime=1481762316 (12/15/16 00:38:36)</div><div class="">[Wed Dec 14 00:38:39 2016] [error] ipa: DEBUG: set_session_expiration_time: duration_type=inactivity_timeout duration=1200 max_age=1481762016 expiration=1481677119.46 (2016-12-14T00:58:39)</div><div class="">[Wed Dec 14 00:38:39 2016] [error] ipa: DEBUG: store session: session_id=9beb89831ebfca453453ad48feaaa4d0 start_timestamp=2016-12-14T00:38:39 access_timestamp=2016-12-14T00:38:39 expiration_timestamp=2016-12-14T00:58:39</div><div class="">[Wed Dec 14 00:38:39 2016] [error] ipa: DEBUG: Destroyed connection context.ldap2</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""> <div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><table width="550px" bgcolor="#ffffff" border="0" cellpadding="0" cellspacing="0" style="color: rgb(51, 51, 51); font-size: 13.3333px; orphans: 2; widows: 2; font-family: 'Times New Roman';" class=""><tbody class=""><tr height="10" class=""></tr><tr border="0" cellspacing="0" cellpadding="0" class=""><td style="font-family: arial, sans-serif; margin: 0px; padding: 6px 0px 0px; color: rgb(136, 136, 136); width: 550px; border-top-width: 8px; border-top-style: solid; border-top-color: rgb(103, 89, 163);" class=""><table width="100%" border="0" cellspacing="0" cellpadding="0" class=""><tbody class=""><tr class=""><th rowspan="3" style="border-right-width: 1px; border-right-style: solid; border-right-color: rgb(210, 210, 210); padding-right: 1px; width: 90px;" class=""><a href="http://www.placeiq.com/" target="_blank" style="color: rgb(17, 85, 204);" class=""></a><a href="http://www.placeiq.com/" target="_blank" style="color: rgb(17, 85, 204);" class=""></a><a href="http://www.placeiq.com/" target="_blank" style="color: rgb(17, 85, 204);" class=""><img src="https://ci3.googleusercontent.com/proxy/tFn1I-GEOnccUtv8DHHEc49-6g3x3CbuQKzbfl2Z1BObEy0Qz6QebJimpP96TK3Za5MXwXTuwBZaobKp22nYAG3NdxAC0Q=s0-d-e1-ft#https://marketing.placeiq.net/images/placeiq.png" alt="" style="width: 80px;" class=""></a></th><td align="left" style="font-family: sans-serif; margin: 0px; color: rgb(136, 136, 136); line-height: 10px; padding-left: 10px; padding-top: 5px;" class=""><span style="color: rgb(94, 95, 94); font-family: Trebuchet, sans-serif; font-size: 16px; font-weight: bold;" class="">Jim Richard</span></td><th rowspan="3" style="padding-right: 1px; width: 40px; padding-left: 5px;" class=""><a href="https://twitter.com/placeiq" target="_blank" style="color: rgb(17, 85, 204);" class=""></a><a href="https://twitter.com/placeiq" target="_blank" style="color: rgb(17, 85, 204);" class=""></a><a href="https://twitter.com/placeiq" target="_blank" style="color: rgb(17, 85, 204);" class=""><img src="https://ci4.googleusercontent.com/proxy/490PXYv9O6OiIp_DL4vuabJqVn53fMon5xNYZdftCVea9ySR2LcFDHe6Cdntb2G68uDAuA6FgLny8wKWLFWpsrPAt_FtLaE=s0-d-e1-ft#https://marketing.placeiq.net/images/twitter1.png" alt="" style="width: 35px;" class=""></a></th><th rowspan="3" style="padding-right: 1px; width: 40px;" class=""><a href="https://www.facebook.com/PlaceIQ" target="_blank" style="color: rgb(17, 85, 204);" class=""></a><a href="https://www.facebook.com/PlaceIQ" target="_blank" style="color: rgb(17, 85, 204);" class=""><img src="https://ci3.googleusercontent.com/proxy/fztHf1lRKLQYcAxebqfp2PYXCwVap3GobHVIbyp0j3NcuJOY16bUAZBibVOFf-fd1GsiuhrOfYy6dSwhlCwWU8ZUlw9OX5I=s0-d-e1-ft#https://marketing.placeiq.net/images/facebook.png" alt="" style="width: 35px;" class=""></a></th><th rowspan="3" style="padding-right: 1px; width: 40px;" class=""><a href="https://www.linkedin.com/company/placeiq" target="_blank" style="color: rgb(17, 85, 204);" class=""></a><a href="https://www.linkedin.com/company/placeiq" target="_blank" style="color: rgb(17, 85, 204);" class=""><img src="https://ci5.googleusercontent.com/proxy/H26ThD7R6DOqxoLTgzi6k5SMrHoF2Tj44xI_7XlD9KfOIiGwe1WIMc5iQBxUBA9EuIyJMdaRXrhZTOrnkrn8O9Rf1FP9UQU=s0-d-e1-ft#https://marketing.placeiq.net/images/linkedin.png" alt="" style="width: 35px;" class=""></a></th></tr><tr class=""><td align="left" style="font-family: Trebuchet, sans-serif; margin: 0px; font-size: 9px; text-transform: uppercase; font-weight: bold; color: rgb(136, 136, 136); line-height: 10px; padding-left: 10px; padding-top: 7px;" class=""><span rowspan="1" class="">SYSTEM ADMINISTRATOR III</span></td></tr><tr class=""><td align="left" style="font-family: sans-serif; margin: 0px; color: rgb(136, 136, 136); line-height: 10px; padding-left: 10px; padding-top: 3px;" class=""><font face="Georgia, sans-serif" class=""><span style="font-size: 10px;" class=""><i class="">(646) 338-8905 </i></span></font> </td></tr></tbody></table></td></tr></tbody></table><a href="http://placeiq.com/2016/10/26/the-making-of-a-location-data-industry-milestone/" class=""><br style="color: rgb(51, 51, 51); font-family: 'Open Sans', sans-serif; font-size: 13.3333px; font-variant-ligatures: normal; line-height: normal; orphans: 2; widows: 2; background-color: rgb(255, 255, 255);" class=""><img src="https://marketing.placeiq.net/images/Alibaba.png" alt="PlaceIQ:Alibaba" style="font-family: 'open sans', sans-serif; font-size: 13px;" class=""></a></div></div><br class="Apple-interchange-newline"></div><br class="Apple-interchange-newline"><br class="Apple-interchange-newline"> </div> <br class=""><div><blockquote type="cite" class=""><div class="">On Dec 2, 2016, at 5:29 PM, Rob Crittenden <<a href="mailto:rcritten@redhat.com" class="">rcritten@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Jim Richard wrote:<br class=""><blockquote type="cite" class="">Hmm ya. So before I rebuilt anything I thought maybe it was my DNS<br class="">records but it looks like that’s not it.<br class=""><br class="">More background, I used to have sso-109 and sso-110, both CA’s. I<br class="">rebuilt sso-110 without CA.<br class=""><br class="">My DNS is external, BIND on another host.<br class=""><br class="">Using the following (at the end of the message) host/key issue as an<br class="">example. On this host, in sssd.conf, ipa_server and krb5_server values<br class="">are both _srv_ so that means they’ll discover from DNS right?<br class=""><br class="">But in my krb5.conf I have:<br class=""><br class="">[realms]<br class=""> <a href="http://placeiq.net" class="">PLACEIQ.NET</a> <<a href="http://placeiq.net" class="">http://placeiq.net</a>> = {<br class=""> kdc = <a href="http://sso-110.nym1.placeiq.net" class="">sso-110.nym1.placeiq.net</a> <<a href="http://sso-110.nym1.placeiq.net" class="">http://sso-110.nym1.placeiq.net</a>>:88<br class=""> master_kdc = <a href="http://sso-110.nym1.placeiq.net" class="">sso-110.nym1.placeiq.net</a><br class=""><<a href="http://sso-110.nym1.placeiq.net" class="">http://sso-110.nym1.placeiq.net</a>>:88<br class=""> admin_server = <a href="http://sso-110.nym1.placeiq.net" class="">sso-110.nym1.placeiq.net</a><br class=""><<a href="http://sso-110.nym1.placeiq.net" class="">http://sso-110.nym1.placeiq.net</a>>:749<br class=""> default_domain = <a href="http://placeiq.net" class="">placeiq.net</a> <<a href="http://placeiq.net" class="">http://placeiq.net</a>><br class=""> pkinit_anchors = FILE:/etc/ipa/ca.crt<br class=""> }<br class=""><br class=""><br class="">Is there any other IPA related config file that might reference a host name?<br class=""><br class="">I’ll include my DNS records at the end here, do they look correct for a<br class="">two server setup, one with a CA (sso-109) and the other no CA (sso-110)?<br class=""><br class="">I never have been sure about the “kerberos-master” entries, what makes<br class="">an IPA host a “kerberos master” and is this related to the CA in any way?<br class=""><br class="">; ldap servers<br class="">_ldap._tcp IN SRV 0 100 389 <a href="http://sso-109.nym1.placeiq.net" class="">sso-109.nym1.placeiq.net</a><br class=""><<a href="http://sso-109.nym1.placeiq.net" class="">http://sso-109.nym1.placeiq.net</a>>.<br class="">_ldap._tcp IN SRV 0 100 389 <a href="http://sso-110.nym1.placeiq.net" class="">sso-110.nym1.placeiq.net</a><br class=""><<a href="http://sso-110.nym1.placeiq.net" class="">http://sso-110.nym1.placeiq.net</a>>.<br class=""><br class="">;kerberos realm<br class="">_kerberos IN TXT <a href="http://placeiq.net" class="">PLACEIQ.NET</a> <<a href="http://placeiq.net" class="">http://placeiq.net</a>><br class=""><br class="">; kerberos servers<br class="">_kerberos._tcp IN SRV 0 100 88 <a href="http://sso-109.nym1.placeiq.net" class="">sso-109.nym1.placeiq.net</a><br class=""><<a href="http://sso-109.nym1.placeiq.net" class="">http://sso-109.nym1.placeiq.net</a>>.<br class="">_kerberos._tcp IN SRV 0 100 88 <a href="http://sso-110.nym1.placeiq.net" class="">sso-110.nym1.placeiq.net</a><br class=""><<a href="http://sso-110.nym1.placeiq.net" class="">http://sso-110.nym1.placeiq.net</a>>.<br class=""><br class="">_kerberos._udp IN SRV 0 100 88 <a href="http://sso-109.nym1.placeiq.net" class="">sso-109.nym1.placeiq.net</a><br class=""><<a href="http://sso-109.nym1.placeiq.net" class="">http://sso-109.nym1.placeiq.net</a>>.<br class="">_kerberos._udp IN SRV 0 100 88 <a href="http://sso-110.nym1.placeiq.net" class="">sso-110.nym1.placeiq.net</a><br class=""><<a href="http://sso-110.nym1.placeiq.net" class="">http://sso-110.nym1.placeiq.net</a>>.<br class=""><br class="">_kerberos-master._tcp IN SRV 0 100 88 <a href="http://sso-109.nym1.placeiq.net" class="">sso-109.nym1.placeiq.net</a><br class=""><<a href="http://sso-109.nym1.placeiq.net" class="">http://sso-109.nym1.placeiq.net</a>>.<br class="">_kerberos-master._udp IN SRV 0 100 88 <a href="http://sso-109.nym1.placeiq.net" class="">sso-109.nym1.placeiq.net</a><br class=""><<a href="http://sso-109.nym1.placeiq.net" class="">http://sso-109.nym1.placeiq.net</a>>.<br class="">_kerberos-adm._tcp IN SRV 0 100 749 <a href="http://sso-109.nym1.placeiq.net" class="">sso-109.nym1.placeiq.net</a><br class=""><<a href="http://sso-109.nym1.placeiq.net" class="">http://sso-109.nym1.placeiq.net</a>>.<br class="">_kerberos-adm._udp IN SRV 0 100 749 <a href="http://sso-109.nym1.placeiq.net" class="">sso-109.nym1.placeiq.net</a><br class=""><<a href="http://sso-109.nym1.placeiq.net" class="">http://sso-109.nym1.placeiq.net</a>>.<br class=""><br class="">_kpasswd._tcp IN SRV 0 100 464 <a href="http://sso-109.nym1.placeiq.net" class="">sso-109.nym1.placeiq.net</a><br class=""><<a href="http://sso-109.nym1.placeiq.net" class="">http://sso-109.nym1.placeiq.net</a>>.<br class="">_kpasswd._tcp IN SRV 0 100 464 <a href="http://sso-110.nym1.placeiq.net" class="">sso-110.nym1.placeiq.net</a><br class=""><<a href="http://sso-110.nym1.placeiq.net" class="">http://sso-110.nym1.placeiq.net</a>>.<br class=""><br class="">_kpasswd._udp IN SRV 0 100 464 <a href="http://sso-109.nym1.placeiq.net" class="">sso-109.nym1.placeiq.net</a><br class=""><<a href="http://sso-109.nym1.placeiq.net" class="">http://sso-109.nym1.placeiq.net</a>>.<br class="">_kpasswd._udp IN SRV 0 100 464 <a href="http://sso-110.nym1.placeiq.net" class="">sso-110.nym1.placeiq.net</a><br class=""><<a href="http://sso-110.nym1.placeiq.net" class="">http://sso-110.nym1.placeiq.net</a>>.<br class=""><br class="">; CNAME for IPA CA replicas (used for CRL, OCSP)<br class="">ipa-ca IN A 10.1.41.109<br class=""><br class=""><br class=""><br class="">Number of certificates and requests being tracked: 1.<br class="">Request ID '20141110221330':<br class=""> status: MONITORING<br class=""> ca-error: Server at <a href="https://sso-110.nym1.placeiq.net/ipa/xml" class="">https://sso-110.nym1.placeiq.net/ipa/xml</a><br class="">denied our request, giving up: 2100 (RPC failed at server. Insufficient<br class="">access: not allowed to perform this command).<br class=""> stuck: no<br class=""> key pair storage:<br class="">type=NSSDB,location='/etc/pki/nssdb',nickname='IPA Machine Certificate -<br class=""><a href="http://phoenix-142.nym1.placeiq.net" class="">phoenix-142.nym1.placeiq.net</a><br class=""><http://phoenix-142.nym1.placeiq.net>',token='NSS Certificate DB'<br class=""> certificate: type=NSSDB,location='/etc/pki/nssdb',nickname='IPA<br class="">Machine Certificate - phoenix-142.nym1.placeiq.net<br class=""><http://phoenix-142.nym1.placeiq.net>',token='NSS Certificate DB'<br class=""> CA: IPA<br class=""> issuer: CN=Certificate Authority,O=PLACEIQ.NET <http://placeiq.net><br class=""> subject: CN=phoenix-142.nym1.placeiq.net<br class=""><http://phoenix-142.nym1.placeiq.net>,O=PLACEIQ.NET <http://placeiq.net><br class=""> expires: 2016-11-10 22:13:31 UTC<br class=""> key usage:<br class="">digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment<br class=""> eku: id-kp-serverAuth,id-kp-clientAuth<br class=""> pre-save command:<br class=""> post-save command:<br class=""> track: yes<br class=""> auto-renew: yes<br class=""><br class=""><br class=""><br class="">We are moving to latest version on RHEL so we’ll have paid support but<br class="">before than, gaining this understanding is massively valuable :)<br class=""></blockquote><br class="">I'm pretty certain this has nothing to do with servers being removed.<br class="">IPA isn't saying it can't find something, it's saying you aren't allowed<br class="">to do something.<br class=""><br class="">Why that is the case I don't know. A way to maybe find out would involve<br class="">enabling debugging on the server. You can do this by creating<br class="">/etc/ipa/server.conf with these contents:<br class=""><br class="">[global]<br class="">debug=True<br class=""><br class="">Restart httpd and watch. I'd leave it on just long enough to see the<br class="">problem, then turn it off again given you are already having disk space<br class="">issues.<br class=""><br class="">There is no way to dynamically do this w/o restarting the service.<br class=""><br class="">rob<br class=""><br class=""></div></div></blockquote></div><br class=""></div></body></html>