[Freeipa-users] Can not post to list - email floats off into cyberspace

Sat Oct 3 00:35:51 UTC 2015

Sorry about this post.  I sent this email to the list 3 times over the
last 48 hours and it was finally accepted after the 3rd send when I
changed the subject to something totally not descriptive of my problem. 
Original email with original subject also finally posted today :(

> We have a FreeIPA domain running IPA server 4.1.4 on CentOS 7.
>
> We have no per zone forwarding enabled, only a single global forwarder.
> This seems to work fine, but then after a while (several weeks I think)
> will randomly stop working.
>
> We had this issue several weeks ago on a different IPA domain (identical
> setup) in our production network but it was ignored because a server
> restart fixed it.
>
> This issue then re-surfaced in our development domain today (different
> network, different physical hardware, same OS and IPA versions).
>
> I received a report today from a developer that he could not ping a
> machine in another domain so I verified network connectivity and
> everything was fine.  When I tried to resolve the name from the IPA dc
> using ping it would fail, but nslookup directly to the forward server
> worked fine.
>
> ipactl showed no issues, and only after I restarted the server did the
> lookups start working again.
>
> Console log below :
>
> Using username "myipausername".
> Last login: Thu Oct  1 16:36:51 2015 from 10.5.5.57
> [myipausername at dc1 ~]$ sudo su -
> Last login: Tue Sep 29 19:03:39 UTC 2015 on pts/3
>
> ATTEMPT FIRST PING TO UNRESOLVABLE HOST
> =======================================
> [root at dc1 ~]# ping artifactory.externaldomain.net
> ping: unknown host artifactory.externaldomain.net
>
> CHECK IPA STATUS
> ================
> [root at dc1 ~]# ipactl status
> Directory Service: RUNNING
> krb5kdc Service: RUNNING
> kadmin Service: RUNNING
> named Service: RUNNING
> ipa_memcached Service: RUNNING
> httpd Service: RUNNING
> pki-tomcatd Service: RUNNING
> smb Service: RUNNING
> winbind Service: RUNNING
> ipa-otpd Service: RUNNING
> ipa-dnskeysyncd Service: RUNNING
> ipa: INFO: The ipactl command was successful
>
> ATTEMPT PING OF GLOBAL FORWARDER
> ================================
> [root at dc1 ~]# ping 10.21.0.14
> PING 10.21.0.14 (10.21.0.14) 56(84) bytes of data.
> 64 bytes from 10.21.0.14: icmp_seq=1 ttl=64 time=0.275 ms
> 64 bytes from 10.21.0.14: icmp_seq=2 ttl=64 time=0.327 ms
> ^C
> --- 10.21.0.14 ping statistics ---
> 2 packets transmitted, 2 received, 0% packet loss, time 1000ms
> rtt min/avg/max/mdev = 0.275/0.301/0.327/0.026 ms
>
> MANUAL NSLOOKUP OF DOMAIN ON GLOBAL FORWARDER FROM IPA DC
> =========================================================
> [root at dc1 ~]# nslookup
>> server 10.21.0.14
> Default server: 10.21.0.14
> Address: 10.21.0.14#53
>> artifactory.externaldomain.net
> Server:         10.21.0.14
> Address:        10.21.0.14#53
>
> Non-authoritative answer:
> artifactory.externaldomain.net     canonical name =
> van-artifactory1.externaldomain.net.
> Name:   van-artifactory1.externaldomain.net
> Address: 10.20.10.14
>
> RE-ATTEMPT PING SINCE WE KNOW THAT NAME RESOLUTION (at least via nslookup
> IS WORKING FROM THIS MACHINE
> ======================================================================================================
>> ^C[root at dc1 ~]# ping artifactory.externaldomain.net
> ping: unknown host artifactory.externaldomain.net
> [root at dc1 ~]# ping van-artifactory1.externaldomain.net
> ping: unknown host van-artifactory1.externaldomain.net
>
> RESTART IPA SERVICES
> ====================
> [root at dc1 ~]# ipactl restart
> Restarting Directory Service
> Restarting krb5kdc Service
> Restarting kadmin Service
> Restarting named Service
> Restarting ipa_memcached Service
> Restarting httpd Service
> Restarting pki-tomcatd Service
> Restarting smb Service
> Restarting winbind Service
> Restarting ipa-otpd Service
> Restarting ipa-dnskeysyncd Service
> ipa: INFO: The ipactl command was successful
> [root at dc1 ~]# ipa dnsconfig-show
> ipa: ERROR: did not receive Kerberos credentials
> [root at dc1 ~]# kinit myipausername
> Password for myipausername at ipadomain.NET:
>
> OUTPUT GLOBAL FORWARDER CONFIG FOR TROUBLESHOOTING
> ==================================================
> [root at dc1 ~]# ipa dnsconfig-show
>   Global forwarders: 10.21.0.14
>   Allow PTR sync: TRUE
>
> PING NOW WORKS BECAUSE IPA SERVICES WERE RESTARTED
> ==================================================
> [root at dc1 ~]# ping artifactory.externaldomain.net
> PING van-artifactory1.externaldomain.net (10.20.10.14) 56(84) bytes of
> data.
> 64 bytes from 10.20.10.14: icmp_seq=1 ttl=60 time=3.00 ms
> 64 bytes from 10.20.10.14: icmp_seq=2 ttl=60 time=1.42 ms
> 64 bytes from 10.20.10.14: icmp_seq=3 ttl=60 time=2.39 ms
> ^C
> --- van-artifactory1.externaldomain.net ping statistics ---
> 3 packets transmitted, 3 received, 0% packet loss, time 2004ms
> rtt min/avg/max/mdev = 1.420/2.274/3.004/0.653 ms
> [root at dc1 ~]#
>
> Here are some strange enties from my /var/log/messages relating to errors
> from today :
>
> Oct  1 20:39:31 dc1 named-pkcs11[15066]: checkhints: unable to get root NS
> rrset from cache: not found
> Oct  1 20:39:17 dc1 named-pkcs11[15066]: error (network unreachable)
> resolving 'pmdb1.ipadomain.net/A/IN': 2001:500:2f::f#53
> Oct  1 20:39:17 dc1 named-pkcs11[15066]: error (network unreachable)
> resolving 'pmdb1.ipadomain.net/AAAA/IN': 2001:500:2f::f#53
>
> Looking at the log entries, it appears that there may have been a network
> connectivity 'blip' (maybe a switch or router was restarted) at some point
> and even after connectivity was restored, the global forwarding was
> failing because the "we can't contact our forwarder" status seemed to get
> stuck in memory.
>
>
> --
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project
>