<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> <style type="text/css" style="display:none;"></style> </head> <body dir="ltr"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;"> <p><span style="font-size: 10pt;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> Hi,</span><br> </p> <div style="color: rgb(0, 0, 0);"><font size="2"><span style="font-size:10pt;"> <div class="PlainText"><span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> <br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> I am experiencing slow logins and sudo authentication for servers joined to my FreeIPA domain. I have been following the other recent thread on slow logins and believe my issue is different.<br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> <br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> I have replication setup with 2 FreeIPA servers at each of 3 sites. The replication is working well and I am able to login correctly on client servers with correct sudo permissions etc. Logins seem to take a long time however. There seems to be some kind of DNS/connection timeout issues, see the example below where the client times out on the auth01 server, then retries and connects. I have also seen it switch to an alternate IPA server on timeout. Total delay in this example is about 10 seconds however it can take longer (approx 30 seconds). It is worth mentioning that client servers in each site cannot connect to IPA servers is a different site - however in the example below the auth01 IPA server is in the same site as the client server. I'm not sure if there is any way to make the IPA clients site aware so they prefer to log in to a local server?<br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> <br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> <br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> On the IPA servers themselves there is no noticeable delay and once I have authenticated with sudo once, subsequent attempts in the same login are also near instant. I have not been able to find any reason for this delay in any logs (which probably just means I'm not looking in the right place).<br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> <br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> <br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> DNS servers are running on each IPA server and responding well whenever I have tested.<br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> <br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> <br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> IPA Servers: CentOS 7.2.1511 running IPA 4.2.0 (from standard CentOS repo)<br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> <br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> Client servers: Ubuntu 14.04 running IPA 3.3.4 (From standard Ubuntu repo)<br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> <br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> <br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> Any comments or suggestions greatly appreciated.<br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> <br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> <br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> Thanks,<br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> <br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> Neal.<br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> <br> > <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;"> > </span><br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> Example sssd log for a "sudo -l" attempt.<br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> <br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> (Mon Aug 1 14:39:59 2016) [sssd[be[fqdn.com]]] [krb5_child_timeout]<br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> (0x0040): Timeout for child [7430] reached. In case KDC is distant or<br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> network is slow you may consider increasing value of krb5_auth_timeout.<br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> (Mon Aug 1 14:39:59 2016) [sssd[be[fqdn.com]]] [krb5_auth_done] (0x0020):<br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>> child timed out!<br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span><br> <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 13.3333px;">> </span>These debug messages seem to be telling you what the problem is. Have<br> > you tried how long does it take to kinit (preferably with<br> > KRB5_TRACE=/dev/stderr prepended) ?<br> <br> Hi Jakub,</div> <div class="PlainText"><br> </div> <div class="PlainText">Thanks for your response and sorry for my delay in replying. kinit takes between 2 and 25 seconds to complete - the KRB5_TRACE option shows it trying a random auth server, timing out and trying another random server until it picks a local server which then completes almost immediately. This seems to confirm that the problem is simply the server tries to authenticate against a FreeIPA server that is unreachable and times out causing the randomly slow logins. Given 6 auth servers with only 2 on each site there is a ~ 10% chance of hitting 3 bad servers in a row before login succeeds - if each takes 20 seconds that would explain the random login times of a few sec - 1 minute.</div> <div class="PlainText"><br> </div> <div class="PlainText">If I enter the local kdc servers manually in the realm section of krb5.conf then ssh logins always happen in < 2sec - however I would prefer to avoid the manual step of configuring and updating this (planning to expand out to a few hundred servers over 4-5 sites). Manually setting these is likely to lead to mistakes and it just feels inelegant compared to DNS SRV records.</div> <div class="PlainText"><br> </div> <div class="PlainText">I have seen <a href="https://www.freeipa.org/page/V4/DNS_Location_Mechanism" class="OWAAutoLink">https://www.freeipa.org/page/V4/DNS_Location_Mechanism</a> which looks good but is a proposal from 2013 with no indications that it has actually been developed. I was also very interested by <a href="https://www.freeipa.org/page/Howto/IPA_locations" class="OWAAutoLink">https://www.freeipa.org/page/Howto/IPA_locations</a> which would be perfect - except the "ipa location-add" commands do not seem to be recognised by my FreeIPA installs.</div> <div class="PlainText"><br> </div> <div class="PlainText">Am I missing a better way to handle the case of multiple locations with clients in Location A being unable to authenticate against FreeIPA servers at location B?</div> <div class="PlainText"><br> </div> <div class="PlainText">Any suggestions greatly appreciated.</div> <div class="PlainText"><br> </div> <div class="PlainText">Thanks,</div> <div class="PlainText">Neal.<br> <br> </div> </span></font></div> </div> </body> </html>