PAM troubleshooting assistance

[Joshua M. Miller]

I have a problem authenticating to a Redhat 3.8 host via PAM, (pam_ldap) 
and could use some pointers on continued troubleshooting.

I've recently upgraded OpenLDAP from 2.0.27 -> 2.3.34.  I have ~100 
hosts authenticating to this directory without any issues, the majority 
of these hosts are CentOS 3/4 hosts.

The problem is with 2 RHEL 3.8 hosts -- they have the exact same 
configuration as all of the other linux hosts (pushed via cfengine) and 
yet they do not bind properly to obtain the userPassword attribute.

The basic flow that I see from the LDAP server for a successful bind:
1. Bind anonymously to obtain uid/homedirectory, etc
2. Bind anonymously to attempt to obtain userPassword -> fail
3. Bind as uid authenticating to obtain userPassword -> success

The 2 hosts that are failing do not perform step 3 and the login fails. 
  I thought the problem was related to nss_ldap but I have now come to 
the point where the issue is inconsisten.  The problem went away for a 
day or two when I installed the CentOS nss_ldap RPM on the RHEL host but 
when I restored the ACL this morning, the host stopped working.

The only logs that I see are access denied from pam_unix:
/var/log/messages
May  7 08:51:09 <host> sshd(pam_unix)[11294]: authentication failure; 
logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=host.example.org
May  7 08:51:15 <host> sshd(pam_unix)[11294]: check pass; user unknown
May  7 08:51:18 <host> sshd(pam_unix)[11294]: 1 more authentication 
failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=host.example.org


My /etc/pam.d/system-auth:
auth        required      /lib/security/$ISA/pam_env.so debug
auth        sufficient    /lib/security/$ISA/pam_unix.so likeauth nullok 
debug
auth        sufficient    /lib/security/$ISA/pam_ldap.so use_first_pass 
debug
auth        required      /lib/security/$ISA/pam_deny.so debug

account     sufficient    /lib/security/$ISA/pam_unix.so debug
account     sufficient    /lib/security/$ISA/pam_ldap.so debug

password    required      /lib/security/$ISA/pam_cracklib.so retry=3 
type= debug
password    sufficient    /lib/security/$ISA/pam_unix.so nullok 
use_authtok md5 shadow debug
password    sufficient    /lib/security/$ISA/pam_ldap.so use_authtok debug
password    required      /lib/security/$ISA/pam_deny.so debug

session     required      /lib/security/$ISA/pam_limits.so debug
session     required      /lib/security/$ISA/pam_unix.so debug
session     optional      /lib/security/$ISA/pam_ldap.so debug


/etc/nsswitch.conf:
passwd:     files ldap
shadow:     files ldap
group:      files ldap


Any ideas for continued troubleshooting?

Thanks,
-- 
Joshua M. Miller - RHCE,VCP