[Freeipa-users] Very slow kerberos performance after upgrade to IPA 2.2

[Petr Spacek]

On 07/30/2012 10:37 PM, Sigbjorn Lie wrote:
> Hi,
>
> I've been having performance issues after I upgraded to RHEL 6.3 / IPA 2.2. I
> still have a LDAP server having unusual high cpu usage even after it's been
> removed from the SRV records and is serving almost no clients anymore, but it
> would seem as my main issues is with the kerberos server.
>
> All kerberos services are performing very slowly, and the IPA servers has much
> higher CPU load now then what they had with IPA 2.1. Some services are timing
> out, like kerberized web servers, other kerberized services perform
> authentication very slowly. I had to switch our automounter away from kerberos
> authentication as it is no longer usable.
>
> Using SSH to log on to SSSD enabled hosts are also very slow, a login takes
> anything from 5 seconds up to 20 seconds. Noticably longer than pre IPA 2.2.
>
> The IPA web admin interface is definitely not faster than in IPA 2.1.
>
> For a comparison, listing out all the folders in an automount map, causing
> them to be looked up from LDAP and mounted takes over 5 minutes with IPA 2.2
> when using kerberos authentication for the automounter. There are approx 130
> folders in that automount map.
>
> After unmounting all the mounted folders, and changing to using a username and
> password authentication with a TLS connection, attempting the same operating
> again, and it now finishes in about 14 seconds for both the lookup from LDAP
> and the mount operation.
>
> After unmounting all the mounted folders again, changing to username and
> password authentication with a simple unencrypted bind, and then attempting
> the same operation and it now finishes both lookup and mount in just over 5
> seconds!
>
> I don't have any timing for kerberized automount pre IPA-2.2, but we we're not
> talking about several minutes to mount all the folders in this automount map.
> Unfortunately mounting all the folders is what happens when the users use
> konqueror to browse the automount maps, so this is a very noticable issue.
>
> Even loading a new gnome-terminal or konsole terminal which causes an
> automount folder to be mounted takes anything between 5 - 15 seconds after the
> upgrade. There we're no notiable delay when opening a new terminal window pre
> IPA-2.2.
>
> I am not using SSSD for the automounter.
>
> I do notice that the dbmodule for the kerberos server has changed from "kldap"
> to "ipadb.so" Perhaps there is some issues with the new library?
>
>
>
>
> Regards,
> Siggi


Hello,

I'm not a Kerberos guy, so I can give only general advice:
"Overloaded-CPU-problems" can be troubleshooted with OProfile.

Oprofile is lightweight statistic profiler (AFAIK it was designed for 
production environment).

Step-by-step documentation for RHEL 6 is available from:
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html-single/Deployment_Guide/index.html#ch-OProfile

As you can see in section 22.5.1., it allows to break whole CPU usage between 
processes, libraries and even individual symbols (if proper debuginfos are 
installed).

I recommend to run OProfile on problematic system - results from opreport can 
provide missing clue to us.

OProfile gives best results on bare-metal machines. On virtual machines you 
has to use timer mode in place of hardware performance counters, please see 
the documentation.


Short getting started guide:
http://oprofile.sourceforge.net/doc/overview.html#getting-started

Nice article with theory && examples:
http://people.redhat.com/wcohen/Oprofile.pdf

Homepage with a lot of useful information:
http://oprofile.sourceforge.net/


Petr^2 Spacek