[Freeipa-users] Very slow kerberos performance after upgrade to IPA 2.2

[Sigbjorn Lie]

On Tue, July 31, 2012 10:20, Petr Spacek wrote:
> On 07/30/2012 10:37 PM, Sigbjorn Lie wrote:
>
>> Hi,
>>
>>
>> I've been having performance issues after I upgraded to RHEL 6.3 / IPA 2.2. I
>> still have a LDAP server having unusual high cpu usage even after it's been removed from the SRV
>> records and is serving almost no clients anymore, but it would seem as my main issues is with
>> the kerberos server.
>>
>> All kerberos services are performing very slowly, and the IPA servers has much
>> higher CPU load now then what they had with IPA 2.1. Some services are timing out, like
>> kerberized web servers, other kerberized services perform authentication very slowly. I had to
>> switch our automounter away from kerberos authentication as it is no longer usable.
>>
>> Using SSH to log on to SSSD enabled hosts are also very slow, a login takes
>> anything from 5 seconds up to 20 seconds. Noticably longer than pre IPA 2.2.
>>
>> The IPA web admin interface is definitely not faster than in IPA 2.1.
>>
>>
>> For a comparison, listing out all the folders in an automount map, causing
>> them to be looked up from LDAP and mounted takes over 5 minutes with IPA 2.2 when using kerberos
>> authentication for the automounter. There are approx 130 folders in that automount map.
>>
>> After unmounting all the mounted folders, and changing to using a username and
>> password authentication with a TLS connection, attempting the same operating again, and it now
>> finishes in about 14 seconds for both the lookup from LDAP and the mount operation.
>>
>> After unmounting all the mounted folders again, changing to username and
>> password authentication with a simple unencrypted bind, and then attempting the same operation
>> and it now finishes both lookup and mount in just over 5 seconds!
>>
>> I don't have any timing for kerberized automount pre IPA-2.2, but we we're not
>> talking about several minutes to mount all the folders in this automount map. Unfortunately
>> mounting all the folders is what happens when the users use konqueror to browse the automount
>> maps, so this is a very noticable issue.
>>
>> Even loading a new gnome-terminal or konsole terminal which causes an
>> automount folder to be mounted takes anything between 5 - 15 seconds after the upgrade. There
>> we're no notiable delay when opening a new terminal window pre IPA-2.2.
>>
>>
>> I am not using SSSD for the automounter.
>>
>>
>> I do notice that the dbmodule for the kerberos server has changed from "kldap"
>> to "ipadb.so" Perhaps there is some issues with the new library?
>>
>>
>>
>>
>> Regards,
>> Siggi
>>
>
>
> Hello,
>
>
> I'm not a Kerberos guy, so I can give only general advice:
> "Overloaded-CPU-problems" can be troubleshooted with OProfile.
>
>
> Oprofile is lightweight statistic profiler (AFAIK it was designed for
> production environment).
>
> Step-by-step documentation for RHEL 6 is available from:
> http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html-single/Deployment_Guide/index.ht
> ml#ch-OProfile
>
> As you can see in section 22.5.1., it allows to break whole CPU usage between
> processes, libraries and even individual symbols (if proper debuginfos are installed).
>
> I recommend to run OProfile on problematic system - results from opreport can
> provide missing clue to us.
>
> OProfile gives best results on bare-metal machines. On virtual machines you
> has to use timer mode in place of hardware performance counters, please see the documentation.
>
>
> Short getting started guide:
> http://oprofile.sourceforge.net/doc/overview.html#getting-started
>
>
> Nice article with theory && examples:
> http://people.redhat.com/wcohen/Oprofile.pdf
>
>
> Homepage with a lot of useful information:
> http://oprofile.sourceforge.net/
>
>
>

Thank you.

All 3 IPA servers are close to idle now after switching from kerberos to user/pwd bind for the
Linux automounter.

Still there is an issue with kerberos failing to issue a ticket every now and then and it's
responding very slowly.

There seem to be low activity on this list just now. Is the kerberos people away on vacation?


Rgds,
Siggi