[Freeipa-users] 'Request is a replay'

[Sigbjorn Lie]

On Tue, July 24, 2012 20:29, Simo Sorce wrote:
> On Tue, 2012-07-24 at 10:22 +0200, Sigbjorn Lie wrote:
>
>> Hi,
>>
>>
>> I keep seing this error message in our production environment "Request is a replay" in variuos
>> services using kerberos like ssh, sssd, automounter, squid +++ after the upgrade to RHEL 6.3 /
>> IPA
>> 2.2.
>>
>>
>>
>> Jul 24 10:16:11 server027 sssd_be: GSSAPI Error: Unspecified GSS failure.  Minor code may
>> provide more information (Request is a replay)
>>
>> Seaching google seem to suggest that this is an error with time. However we have NTP configured
>>  (IPA servers as NTP servers) which is synchronized to external NTP servers. There has been no
>> issue before, and I cannot find issue with the time being out of sync on the machines where this
>>  is happening.
>
> This error usually appears only when a same request is found in the
> replay cache. It shouldn't be related to time issues, in that case you usually get clock-skew.
>
> Can you tell me what operation was being performed by sssd when you
> caught that error ? Can you check if immediately before another identical operation had been
> performed ?
>

That being said, I do have 1 IPA server (out of 3) that has significantly higher CPU usage than
the other 2, the 15-minute load average is sitting at between 0.85 and 0.95 the entire day, where
ns-slapd 389-ds process is running at 100% most of the time.

Load: 1.02, 0.94, 0.87

In comparison the other two IPA servers has a 15-minute average between 0.10 - 0.30 throughout the
day, and the ns-slapd process is far from being such a cpu hog.

On the server having high load, running even a command such as "ipactl status" can take up to 20
seconds to complete, where "Directory Service: RUNNING" returns after a second or so, and to list
the rest of the services takes the remainding 19 seconds.

Also the web interface on this particular IPA server is rendered unusable, returning "Limits
exceeded for the query" for almost any action.

Restarting all the IPA servies (ipactl restart) on the problematic host soemwhat improves the
situation, however that particular server returns to having heavy load quickly.

Using logconv.pl to analyze the dirsrv access log file displays that the server in question has
the lowest search queries per min with 106 queries/min. The other servers have 710 search
queries/sec and 168 queries/sec.

For modifications all the IPA servers has about 5-6 queries/sec. For unindexed searches the
problematic server is the server with the lowest number. It does however have more than twice the
amount of GSSAPI binds than the other servers with over 61000 GSSAPI binds over a 17 hour period.

The problematic server is a physical server with 2 x AMD 2.4GHz Quad core CPU and 8GB of RAM.

This issue is also impacting all the clients, where I see random hangs with anything involving a
ldap or kerberos query to the IPA servers.

Any suggestions?



Regards,
Siggi