[Freeipa-users] Slow logins on FreeIPA 4.1.2 (F21)

[Dan Mossor]

On 04/05/2015 12:51 PM, Dmitri Pal wrote:
> On 04/05/2015 12:10 AM, Dan Mossor wrote:
>> I've recently deployed a new domain based on 4.1.2 in F21. We've
>> noticed an issue and can't quite seem to nail it down. The problem is
>> that logins are taking an inordinate amount of time to complete - the
>> fastest logon we can get using LDAP credentials is 8 seconds. During
>> our testing, even logons to the IPA server itself took over 30 seconds
>> to complete.
>>
>> I've narrowed this down to sssd, but that is as far as I can get. When
>> cranking up debugging for sshd and PAM, I see a minimum 2 second delay
>> between ssh handing off the authentication request to sssd and the
>> reply back. The only troubleshooting I've done is with ssh, but the
>> area that causes the most grief is Apache logins. We configured Apache
>> to use PAM for auth through IPA, vice directly calling IPA itself.
>> Logging in to our Redmine site takes users a minimum of 34 seconds to
>> complete. Following this, a simple webpage containing two hyperlinks
>> and two small thumbnail images takes over a minute to load on a
>> gigabit network.
>>
>> The *only* thing changed in this environment was the IPA server. We
>> moved the Redmine from our old network that was using IPA 3.x (F20
>> branch) to the new one. My initial reaction was that it was the VM
>> that was hosting Redmine, but we've run these tests against bare metal
>> machines in the same network and have the same issue. It appears that
>> sssd is taking a very, very long time to talk to FreeIPA - even on the
>> IPA server itself.
>>
>> However, Kerberos logins into the IPA web GUI are near instantaneous,
>> while Username/Password logins take more than a few seconds.
>>
>> I need to get this solved. My developers don't appreciate the glory
>> days of XP taking 5 minutes to log into an IIS 2.1 web server on the
>> local network. I don't have the budget to keep them at the coffee pot
>> waiting on the network. So, what further information do you need from
>> me to track this one down?
>>
>> Dan
>>
> Several tips.
> Please check your DNS configuration.
> Such delay is usually caused by the DNS lookups timing out. That means
> that the servers probably trying to resolve names against an old DNS
> server that is not around. Look at resolve.conf and make sure only valid
> DNS servers are there and they are in the proper order.
>
> If this does not help please turn on SSSD debug_level to 10, sanitize
> and send the SSSD domain logs and sssd.conf to the list.
> More hints can be found here:
> https://fedorahosted.org/sssd/wiki/Troubleshooting
>
DNS lookups are good - 'dig' and 'dig -x' return instantaneous forward 
and reverse lookups on the IPA server, the target server, and the 
client. The only DNS server configured is the IPA server.

I did catch some sssd logs. I set logging to 0x0450 instead of 10, and I 
didn't have time to compare if any different information was caught. If 
you still need me to specify log level 10 or some other setting, let me 
know. The login that these logs are for took 15.371 seconds (checked via 
'time ssh danofsatx at yoda.example.lcl exit'

selinux_child.log: http://fpaste.org/207805/
sssd_sudo.log: http://fpaste.org/207806/
sssd_pac.log: http://fpaste.org/207807/
sssd_pam.log: http://fpaste.org/207808/67775142/
sssd_nss.log: http://fpaste.org/207809/
sssd.log: http://fpaste.org/207810/
sssd_example.lcl.log: http://fpaste.org/207811/36832514/

-- 
Dan Mossor
Systems Engineer at Large
Fedora KDE WG | Fedora QA Team | Fedora Server SIG
Fedora Infrastructure Apprentice
FAS: dmossor IRC: danofsatx
San Antonio, Texas, USA