[Freeipa-users] Very slow kerberos performance after upgrade to IPA 2.2

[Simo Sorce]

On Tue, 2012-07-31 at 10:50 +0200, Sigbjorn Lie wrote:
> On Tue, July 31, 2012 10:20, Petr Spacek wrote:
> > On 07/30/2012 10:37 PM, Sigbjorn Lie wrote:
> >
> >> Hi,
> >>
> >>
> >> I've been having performance issues after I upgraded to RHEL 6.3 / IPA 2.2. I
> >> still have a LDAP server having unusual high cpu usage even after it's been removed from the SRV
> >> records and is serving almost no clients anymore, but it would seem as my main issues is with
> >> the kerberos server.
> >>
> >> All kerberos services are performing very slowly, and the IPA servers has much
> >> higher CPU load now then what they had with IPA 2.1. Some services are timing out, like
> >> kerberized web servers, other kerberized services perform authentication very slowly. I had to
> >> switch our automounter away from kerberos authentication as it is no longer usable.
> >>
> >> Using SSH to log on to SSSD enabled hosts are also very slow, a login takes
> >> anything from 5 seconds up to 20 seconds. Noticably longer than pre IPA 2.2.
> >>
> >> The IPA web admin interface is definitely not faster than in IPA 2.1.
> >>
> >>
> >> For a comparison, listing out all the folders in an automount map, causing
> >> them to be looked up from LDAP and mounted takes over 5 minutes with IPA 2.2 when using kerberos
> >> authentication for the automounter. There are approx 130 folders in that automount map.
> >>
> >> After unmounting all the mounted folders, and changing to using a username and
> >> password authentication with a TLS connection, attempting the same operating again, and it now
> >> finishes in about 14 seconds for both the lookup from LDAP and the mount operation.
> >>
> >> After unmounting all the mounted folders again, changing to username and
> >> password authentication with a simple unencrypted bind, and then attempting the same operation
> >> and it now finishes both lookup and mount in just over 5 seconds!
> >>
> >> I don't have any timing for kerberized automount pre IPA-2.2, but we we're not
> >> talking about several minutes to mount all the folders in this automount map. Unfortunately
> >> mounting all the folders is what happens when the users use konqueror to browse the automount
> >> maps, so this is a very noticable issue.
> >>
> >> Even loading a new gnome-terminal or konsole terminal which causes an
> >> automount folder to be mounted takes anything between 5 - 15 seconds after the upgrade. There
> >> we're no notiable delay when opening a new terminal window pre IPA-2.2.
> >>
> >>
> >> I am not using SSSD for the automounter.
> >>
> >>
> >> I do notice that the dbmodule for the kerberos server has changed from "kldap"
> >> to "ipadb.so" Perhaps there is some issues with the new library?
> >>
> >>
> >>
> >>
> >> Regards,
> >> Siggi
> >>
> >
> >
> > Hello,
> >
> >
> > I'm not a Kerberos guy, so I can give only general advice:
> > "Overloaded-CPU-problems" can be troubleshooted with OProfile.
> >
> >
> > Oprofile is lightweight statistic profiler (AFAIK it was designed for
> > production environment).
> >
> > Step-by-step documentation for RHEL 6 is available from:
> > http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html-single/Deployment_Guide/index.ht
> > ml#ch-OProfile
> >
> > As you can see in section 22.5.1., it allows to break whole CPU usage between
> > processes, libraries and even individual symbols (if proper debuginfos are installed).
> >
> > I recommend to run OProfile on problematic system - results from opreport can
> > provide missing clue to us.
> >
> > OProfile gives best results on bare-metal machines. On virtual machines you
> > has to use timer mode in place of hardware performance counters, please see the documentation.
> >
> >
> > Short getting started guide:
> > http://oprofile.sourceforge.net/doc/overview.html#getting-started
> >
> >
> > Nice article with theory && examples:
> > http://people.redhat.com/wcohen/Oprofile.pdf
> >
> >
> > Homepage with a lot of useful information:
> > http://oprofile.sourceforge.net/
> >
> >
> >
> 
> Thank you.
> 
> All 3 IPA servers are close to idle now after switching from kerberos to user/pwd bind for the
> Linux automounter.
> 
> Still there is an issue with kerberos failing to issue a ticket every now and then and it's
> responding very slowly.
> 
> There seem to be low activity on this list just now. Is the kerberos people away on vacation?

Hi Siggi,
some people are on vacation, some are busy covering others :-)

Would you be able to take a wireshark trace of an automount going on ?
I would like to see precise timing of packets on the wire to make a
first assesment of where is the bottleneck.

We did change from ldap.so to ipadb.so, but the structure of the drivers
is not much different, so I am surprised it would be much slower,
however it is possible, I would like to find out what is going on with
your help.

Simo.

-- 
Simo Sorce * Red Hat, Inc * New York