[Freeipa-users] Very slow kerberos performance after upgrade to IPA 2.2

Tue Jul 31 21:46:07 UTC 2012

On 07/31/2012 11:25 PM, Simo Sorce wrote:
> On Tue, 2012-07-31 at 21:08 +0200, Sigbjorn Lie wrote:
>> On 07/31/2012 01:50 PM, Simo Sorce wrote:
>>> On Tue, 2012-07-31 at 10:50 +0200, Sigbjorn Lie wrote:
>>>> On Tue, July 31, 2012 10:20, Petr Spacek wrote:
>>>>> On 07/30/2012 10:37 PM, Sigbjorn Lie wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>> I've been having performance issues after I upgraded to RHEL 6.3 / IPA 2.2. I
>>>>>> still have a LDAP server having unusual high cpu usage even after it's been removed from the SRV
>>>>>> records and is serving almost no clients anymore, but it would seem as my main issues is with
>>>>>> the kerberos server.
>>>>>>
>>>>>> All kerberos services are performing very slowly, and the IPA servers has much
>>>>>> higher CPU load now then what they had with IPA 2.1. Some services are timing out, like
>>>>>> kerberized web servers, other kerberized services perform authentication very slowly. I had to
>>>>>> switch our automounter away from kerberos authentication as it is no longer usable.
>>>>>>
>>>>>> Using SSH to log on to SSSD enabled hosts are also very slow, a login takes
>>>>>> anything from 5 seconds up to 20 seconds. Noticably longer than pre IPA 2.2.
>>>>>>
>>>>>> The IPA web admin interface is definitely not faster than in IPA 2.1.
>>>>>>
>>>>>>
>>>>>> For a comparison, listing out all the folders in an automount map, causing
>>>>>> them to be looked up from LDAP and mounted takes over 5 minutes with IPA 2.2 when using kerberos
>>>>>> authentication for the automounter. There are approx 130 folders in that automount map.
>>>>>>
>>>>>> After unmounting all the mounted folders, and changing to using a username and
>>>>>> password authentication with a TLS connection, attempting the same operating again, and it now
>>>>>> finishes in about 14 seconds for both the lookup from LDAP and the mount operation.
>>>>>>
>>>>>> After unmounting all the mounted folders again, changing to username and
>>>>>> password authentication with a simple unencrypted bind, and then attempting the same operation
>>>>>> and it now finishes both lookup and mount in just over 5 seconds!
>>>>>>
>>>>>> I don't have any timing for kerberized automount pre IPA-2.2, but we we're not
>>>>>> talking about several minutes to mount all the folders in this automount map. Unfortunately
>>>>>> mounting all the folders is what happens when the users use konqueror to browse the automount
>>>>>> maps, so this is a very noticable issue.
>>>>>>
>>>>>> Even loading a new gnome-terminal or konsole terminal which causes an
>>>>>> automount folder to be mounted takes anything between 5 - 15 seconds after the upgrade. There
>>>>>> we're no notiable delay when opening a new terminal window pre IPA-2.2.
>>>>>>
>>>>>>
>>>>>> I am not using SSSD for the automounter.
>>>>>>
>>>>>>
>>>>>> I do notice that the dbmodule for the kerberos server has changed from "kldap"
>>>>>> to "ipadb.so" Perhaps there is some issues with the new library?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Siggi
>>>>>>
>>>>> Hello,
>>>>>
>>>>>
>>>>> I'm not a Kerberos guy, so I can give only general advice:
>>>>> "Overloaded-CPU-problems" can be troubleshooted with OProfile.
>>>>>
>>>>>
>>>>> Oprofile is lightweight statistic profiler (AFAIK it was designed for
>>>>> production environment).
>>>>>
>>>>> Step-by-step documentation for RHEL 6 is available from:
>>>>> http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html-single/Deployment_Guide/index.ht
>>>>> ml#ch-OProfile
>>>>>
>>>>> As you can see in section 22.5.1., it allows to break whole CPU usage between
>>>>> processes, libraries and even individual symbols (if proper debuginfos are installed).
>>>>>
>>>>> I recommend to run OProfile on problematic system - results from opreport can
>>>>> provide missing clue to us.
>>>>>
>>>>> OProfile gives best results on bare-metal machines. On virtual machines you
>>>>> has to use timer mode in place of hardware performance counters, please see the documentation.
>>>>>
>>>>>
>>>>> Short getting started guide:
>>>>> http://oprofile.sourceforge.net/doc/overview.html#getting-started
>>>>>
>>>>>
>>>>> Nice article with theory&&   examples:
>>>>> http://people.redhat.com/wcohen/Oprofile.pdf
>>>>>
>>>>>
>>>>> Homepage with a lot of useful information:
>>>>> http://oprofile.sourceforge.net/
>>>>>
>>>>>
>>>>>
>>>> Thank you.
>>>>
>>>> All 3 IPA servers are close to idle now after switching from kerberos to user/pwd bind for the
>>>> Linux automounter.
>>>>
>>>> Still there is an issue with kerberos failing to issue a ticket every now and then and it's
>>>> responding very slowly.
>>>>
>>>> There seem to be low activity on this list just now. Is the kerberos people away on vacation?
>>> Hi Siggi,
>>> some people are on vacation, some are busy covering others :-)
>>>
>>> Would you be able to take a wireshark trace of an automount going on ?
>>> I would like to see precise timing of packets on the wire to make a
>>> first assesment of where is the bottleneck.
>>>
>>> We did change from ldap.so to ipadb.so, but the structure of the drivers
>>> is not much different, so I am surprised it would be much slower,
>>> however it is possible, I would like to find out what is going on with
>>> your help.
>>>
>> OK, I will get that done when I'm back in the office tomorrow. I suspect
>> it will be somewhat better than my first results as the load on the IPA
>> servers are now much lower when the linux automounters are no longer
>> using kerberos for authentication.
>>
>> It seem like there is a race condition going on as the shit didn't hit
>> the fan until the week after the upgrade to IPA 2.2 when people returned
>> to work. The slowness issues then gradually became worse and worse.
>>
>> I will send you the captures in a private email. Do you need anything
>> besides TCP 389, 636 and TCP/UDP 88 ?
> no need for TCP 636, but it may be intresting to see DNS queries, do you
> use the IPA integrated DNS or do you use your own infra ?
>
>

Ok, I'll include DNS requests.

I manage the DNS entries with IPA, but I dump the zone contents from 
LDAP to files and then I run BIND off the files.

Rgds,
Siggi