<div dir="ltr"><div>I'm pretty sure this is the root of my problem (not confirmed yet, but it's AIX -- that's always the problem):</div><div><br></div><div><a href="http://www-01.ibm.com/support/docview.wss?uid=swg21212940">http://www-01.ibm.com/support/docview.wss?uid=swg21212940</a><br>
</div><div><br></div><div>The takeaway is this:</div><div><br></div><div>"<span style="font-size:13px;color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:15px">The first query (184) is a normal IPV4 lookup for "</span><tt style="font-size:13px;margin:0px;padding:0px;border:0px none;outline:none 0px;vertical-align:baseline;font-family:monospace,sans-serif;word-wrap:break-word;color:rgb(51,51,51);line-height:15px"><a href="http://ldap.austin.texas.com">ldap.austin.texas.com</a></tt><span style="font-size:13px;color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:15px">", which returns "</span><tt style="font-size:13px;margin:0px;padding:0px;border:0px none;outline:none 0px;vertical-align:baseline;font-family:monospace,sans-serif;word-wrap:break-word;color:rgb(51,51,51);line-height:15px">192.168.1.255</tt><span style="font-size:13px;color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:15px">". But then an IPV6 lookup is done for the same name. Because there is no IPV6 address for </span><tt style="font-size:13px;margin:0px;padding:0px;border:0px none;outline:none 0px;vertical-align:baseline;font-family:monospace,sans-serif;word-wrap:break-word;color:rgb(51,51,51);line-height:15px"><a href="http://ldap.austin.texas.com">ldap.austin.texas.com</a></tt><span style="font-size:13px;color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:15px">, it continues searching every </span><tt style="font-size:13px;margin:0px;padding:0px;border:0px none;outline:none 0px;vertical-align:baseline;font-family:monospace,sans-serif;word-wrap:break-word;color:rgb(51,51,51);line-height:15px">search</tt><span style="font-size:13px;color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:15px"> domain in the resolv.conf file (</span><tt style="font-size:13px;margin:0px;padding:0px;border:0px none;outline:none 0px;vertical-align:baseline;font-family:monospace,sans-serif;word-wrap:break-word;color:rgb(51,51,51);line-height:15px"><a href="http://example.austin.texas.com">example.austin.texas.com</a> <a href="http://austin.texas.com">austin.texas.com</a> <a href="http://texas.com">texas.com</a></tt><span style="font-size:13px;color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:15px">) trying to find one."</span></div>
<div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Sep 20, 2013 at 3:07 AM, Petr Spacek <span dir="ltr"><<a href="mailto:pspacek@redhat.com" target="_blank">pspacek@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On 20.9.2013 01:24, KodaK wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
This is ridiculous, right?<br>
<br>
IPA server 1:<br>
<br>
# for i in $(ls access*); do echo -n $i:\ ;grep err=32 $i | wc -l; done<br>
access: 248478<br>
access.20130916-043207: 302774<br>
access.20130916-123642: 272572<br>
access.20130916-201516: 294308<br>
access.20130917-081053: 295060<br>
access.20130917-144559: 284498<br>
access.20130917-231435: 281035<br>
access.20130918-091611: 291165<br>
access.20130918-154945: 275792<br>
access.20130919-014322: 296113<br>
<br>
IPA server 2:<br>
<br>
access: 4313<br>
access.20130909-200216: 4023<br>
access.20130910-200229: 4161<br>
access.20130911-200239: 4182<br>
access.20130912-200249: 5069<br>
access.20130913-200258: 3833<br>
access.20130914-200313: 4208<br>
access.20130915-200323: 4702<br>
access.20130916-200332: 4532<br>
<br>
<br>
IPA server 3:<br>
<br>
access: 802<br>
access.20130910-080737: 3876<br>
access.20130911-080748: 3902<br>
access.20130912-080802: 3678<br>
access.20130913-080810: 3765<br>
access.20130914-080826: 3524<br>
access.20130915-080907: 4142<br>
access.20130916-080916: 4930<br>
access.20130917-080926: 4769<br>
access.20130918-081005: 2879<br>
<br>
IPA server 4:<br>
<br>
access: 2812<br>
access.20130910-003051: 4095<br>
access.20130911-003105: 3623<br>
access.20130912-003113: 3606<br>
access.20130913-003125: 3581<br>
access.20130914-003135: 3758<br>
access.20130915-003150: 3935<br>
access.20130916-003159: 4184<br>
access.20130917-003210: 3859<br>
access.20130918-003221: 5110<br>
<br>
<br>
The vast majority of the err=32 messages are DNS entries.<br>
</blockquote>
<br></div></div>
It depends on your setup. Bind-dyndb-ldap does LDAP search for each non-existent name to verify that the name wasn't added to LDAP in meanwhile. If you have clients doing 1M queries for non-existing names per day, then you will see 1M LDAP queries with err=32 per day.<br>
<br>
Next major version of bind-dyndb-ldap will have reworked internal database and it will support negative caching, so number of err=32 should drop significantly.<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Here are some samples:<br>
<br>
[19/Sep/2013:18:19:51 -0500] conn=9 op=169764 SRCH base="idnsName=<a href="http://xxx.com" target="_blank">xxx.com</a><br>
,idnsname=<a href="http://unix.xxx.com" target="_blank">unix.xxx.com</a>,cn=dns,<u></u>dc=unix,dc=xxx,dc=com" scope=0<br>
filter="(objectClass=<u></u>idnsRecord)" attrs=ALL<br>
[19/Sep/2013:18:19:51 -0500] conn=9 op=169764 RESULT err=32 tag=101<br>
nentries=0 etime=0<br>
</blockquote>
<br></div>
This is interesting, because this LDAP query is equal to DNS query for "<a href="http://xxx.com.unix.xxx.com" target="_blank">xxx.com.unix.xxx.com</a>." Are your clients that crazy? :-)<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
[19/Sep/2013:18:19:51 -0500] conn=9 op=169774 SRCH base="idnsName=<br>
<a href="http://slpoxacl01.unix.xxx.com" target="_blank">slpoxacl01.unix.xxx.com</a>,<u></u>idnsname=<a href="http://unix.xxx.com" target="_blank">unix.xxx.com</a>,cn=dns,<u></u>dc=unix,dc=xxx,dc=com"<br>
scope=0 filter="(objectClass=<u></u>idnsRecord)" attrs=ALL<br>
[19/Sep/2013:18:19:51 -0500] conn=9 op=169774 RESULT err=32 tag=101<br>
nentries=0 etime=0<br>
</blockquote>
<br></div>
This is equivalent to DNS query for "<a href="http://slpoxacl01.unix.xxx.com.unix.xxx.com" target="_blank">slpoxacl01.unix.xxx.com.unix.<u></u>xxx.com</a>.".<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
[19/Sep/2013:18:19:51 -0500] conn=9 op=169770 SRCH base="idnsName=<br>
<a href="http://sla400q1.unix.xxx.com" target="_blank">sla400q1.unix.xxx.com</a>,<u></u>idnsname=<a href="http://unix.xxx.com" target="_blank">unix.xxx.com</a>,cn=dns,<u></u>dc=unix,dc=xxx,dc=com"<br>
scope=0 filter="(objectClass=<u></u>idnsRecord)" attrs=ALL<br>
[19/Sep/2013:18:19:51 -0500] conn=9 op=169770 RESULT err=32 tag=101<br>
nentries=0 etime=0<br>
</blockquote>
<br></div>
And this is "<a href="http://sla400q1.unix.xxx.com.unix.xxx.com" target="_blank">sla400q1.unix.xxx.com.unix.<u></u>xxx.com</a>.".<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
[19/Sep/2013:18:19:51 -0500] conn=9 op=169772 SRCH base="idnsName=<br>
<a href="http://magellanhealth.com" target="_blank">magellanhealth.com</a>,idnsname=<a href="http://unix.magellanhealth.com" target="_blank">un<u></u>ix.magellanhealth.com</a>,cn=dns,<u></u>dc=unix,dc=magellanhealth,dc=<u></u>com"<br>
scope=0 filter="(objectClass=<u></u>idnsRecord)" attrs=ALL<br>
[19/Sep/2013:18:19:51 -0500] conn=9 op=169772 RESULT err=32 tag=101<br>
nentries=0 etime=0<br>
<br>
So far today there are over half a million of these. That can't be right.<br>
</blockquote>
<br></div>
I would recommend you to use network sniffer and check which clients sends these crazy queries.<br>
<br>
My guess is that your resolver library (libc?) causes this.<br>
<br>
On my Linux system with glibc-2.17-14.fc19.x86_64 it behaves in this way:<br>
<br>
client query = <a href="http://nonexistent.example.com" target="_blank">nonexistent.example.com</a>.<br>
(I used $ "ping <a href="http://nonexistent.example.com" target="_blank">nonexistent.example.com</a>.")<br>
search domain in /etc/resolv.conf = <a href="http://brq.redhat.com" target="_blank">brq.redhat.com</a>.<br>
<br>
DNS query #1: <a href="http://nonexistent.example.com" target="_blank">nonexistent.example.com</a>. => NXDOMAIN<br>
DNS query #2: <a href="http://nonexistent.example.com.brq.redhat.com" target="_blank">nonexistent.example.com.brq.<u></u>redhat.com</a>. => NXDOMAIN<br>
DNS query #3: <a href="http://nonexistent.example.com.redhat.com" target="_blank">nonexistent.example.com.<u></u>redhat.com</a>. => NXDOMAIN<div class="im"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Thu, Sep 19, 2013 at 3:05 PM, KodaK <<a href="mailto:sakodak@gmail.com" target="_blank">sakodak@gmail.com</a>> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I didn't realize that DNS created one connection. I thought it was one<br>
connection spanning several days.<br>
</blockquote></blockquote>
<br></div>
In theory, there should be 2-4 LDAP connections from each DNS server and those connections should live until DNS or LDAP server restarts/crashes.<br>
<br>
Petr^2 Spacek<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
On Thu, Sep 19, 2013 at 2:51 PM, Rich Megginson <<a href="mailto:rmeggins@redhat.com" target="_blank">rmeggins@redhat.com</a>>wrote:<br>
<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
On 09/19/2013 12:57 PM, KodaK wrote:<br>
<br>
Well, this is awkward:<br>
<br>
[root@slpidml01 slapd-UNIX-xxx-COM]# grep conn=170902 access* | wc -l<br>
5453936<br>
[root@slpidml01 slapd-UNIX-xxx-COM]#<br>
<br>
<br>
Why is it awkward?<br>
<br>
<br>
<br>
<br>
On Thu, Sep 19, 2013 at 1:48 PM, KodaK <<a href="mailto:sakodak@gmail.com" target="_blank">sakodak@gmail.com</a>> wrote:<br>
<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Thanks. I've been running that against my logs, and this has to be<br>
abnormal:<br>
<br>
err=32 129274 No Such Object<br>
err=0 10952 Successful Operations<br>
err=14 536 SASL Bind in Progress<br>
err=53 39 Unwilling To Perform<br>
err=49 3 Invalid Credentials (Bad Password)<br>
<br>
I'm still trying to figure out why there are so many error 32s. Are<br>
there any usual suspects I should know about? (That's just the current<br>
access log, btw.)<br>
<br>
<br>
On Tue, Sep 17, 2013 at 9:01 AM, Rich Megginson <<a href="mailto:rmeggins@redhat.com" target="_blank">rmeggins@redhat.com</a>>wrote:<br>
<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
On 09/16/2013 07:57 PM, Dmitri Pal wrote:<br>
<br>
On 09/16/2013 12:02 PM, KodaK wrote:<br>
<br>
Yet another AIX related problem:<br>
<br>
The AIX LDAP client is called secldapclntd (sure, they could make it<br>
more awkward, but the budget ran out.) I'm running into the issue detailed<br>
here:<br>
<br>
<a href="http://www-01.ibm.com/support/docview.wss?uid=isg1IV11344" target="_blank">http://www-01.ibm.com/support/<u></u>docview.wss?uid=isg1IV11344</a><br>
<br>
"If an LDAP server fails to answer an LDAP query, secldapclntd caches<br>
the non-answered query negatively. This may happen if the LDAP server<br>
is down for example. After the LDAP server is back again secldapclntd will<br>
use the negative cache entry and the application initiating the original<br>
query will still fail until the cache entry expires."<br>
<br>
IBM is working on porting the fix to our specific TL and SP levels.<br>
<br>
What I'm concerned with here, though, is *why* is it timing out? I<br>
don't know what the current timeout values are (AIX sucks, etc.)<br>
<br>
I don't see timeout issues on my Linux boxes, which leads me to<br>
believe that either the sssd timouts are longer or that sssd is just more<br>
robust when dealing with timeouts.<br>
<br>
I believe I'm seeing similar behavior with LDAP sudo on AIX as well,<br>
because I occasionally have to re-run sudo commands because they initially<br>
fail (and I know I'm using the right passwords.) However, sudo doesn't<br>
appear to have a cache (or it handles caching better.)<br>
<br>
Does anyone have any troubleshooting suggestions? Any general "speed<br>
things up" suggestions on the IPA side?<br>
<br>
Thanks,<br>
<br>
--Jason<br>
<br>
--<br>
The government is going to read our mail anyway, might as well make it<br>
tough for them. GPG Public key ID: B6A1A7C6<br>
<br>
<br>
______________________________<u></u>_________________<br></div></div>
Freeipa-users mailing listFreeipa-users@redhat.<u></u>comhttps://<a href="http://www.redhat.com/mailman/listinfo/freeipa-users" target="_blank">www.redhat.com/<u></u>mailman/listinfo/freeipa-users</a><div class="im"><br>
<br>
<br>
Is the server FreeIPA?<br>
Can see in the server logs what is actually happening is it the server<br>
that really takes time or there is a network connectivity issue or FW is<br>
dropping packets?<br>
I would really start with the server side logs.<br>
<br>
<br>
As far as 389 goes, run <a href="http://logconv.pl" target="_blank">logconv.pl</a> against the access logs in<br>
/var/log/dirsrv/slapd-DOMAIN-<u></u>COM<br>
</div></blockquote></blockquote></blockquote></blockquote></blockquote><div class="HOEnZb"><div class="h5">
<br>
______________________________<u></u>_________________<br>
Freeipa-users mailing list<br>
<a href="mailto:Freeipa-users@redhat.com" target="_blank">Freeipa-users@redhat.com</a><br>
<a href="https://www.redhat.com/mailman/listinfo/freeipa-users" target="_blank">https://www.redhat.com/<u></u>mailman/listinfo/freeipa-users</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>The government is going to read our mail anyway, might as well make it tough for them. GPG Public key ID: B6A1A7C6
</div>