[Freeipa-users] DNS forwarding configuration randomly breaks and stops working
nathan at nathanpeters.com
nathan at nathanpeters.com
Fri Oct 2 23:47:28 UTC 2015
This issue has occured again and I am once again trying to troubleshoot it.
show forwarder
--------------
-bash-4.2$ ipa dnsconfig-show
Global forwarders: 10.21.0.14
Allow PTR sync: TRUE
attempt ping
------------
-bash-4.2$ ping stash.externaldomain.net
ping: unknown host stash.externaldomain.net
-attempt nslookup
-----------------
-bash-4.2$ nslookup
> stash.externaldomain.net
Server: 127.0.0.1
Address: 127.0.0.1#53
** server can't find stash.externaldomain.net: NXDOMAIN
*comment* : strange it doesn't work against localhost. Lets make sure
that localhost lookups work at all :
-bash-4.2$ nslookup
> google.com
Server: 127.0.0.1
Address: 127.0.0.1#53
Non-authoritative answer:
Name: google.com
Address: 216.58.216.142
*comment* : yup, I can resolve google.com when talking to localhost...
now lets try to talk to the forwarder configured in the global settings
-----------------------------------------------------------------------
> server 10.21.0.14
Default server: 10.21.0.14
Address: 10.21.0.14#53
> stash.externaldomain.net
Server: 10.21.0.14
Address: 10.21.0.14#53
Non-authoritative answer:
stash.externaldomain.net canonical name = git1.externaldomain.net.
Name: git1.externaldomain.net
Address: 10.20.10.30
more troubleshooting
--------------------
I ran wireshark to see what the freeipa server was sending back to the
client :
3 0.000393 10.178.0.99 10.178.21.2 DNS 163 Standard query response 0x12ca
No such name CNAME git1.externaldomain.net
I've never seen a 'no such CNAME' response before. Lets look at the
contents of the packet:
Frame 4: 163 bytes on wire (1304 bits), 163 bytes captured (1304 bits)
Ethernet II, Src: Vmware_b7:09:c6 (00:50:56:b7:09:c6), Dst:
HewlettP_3c:f9:48 (2c:59:e5:3c:f9:48)
Internet Protocol Version 4, Src: 10.178.0.99 (10.178.0.99), Dst:
10.178.21.2 (10.178.21.2)
User Datagram Protocol, Src Port: 53 (53), Dst Port: 57374 (57374)
Domain Name System (response)
[Request In: 1]
[Time: 0.000414000 seconds]
Transaction ID: 0x12ca
Flags: 0x8183 Standard query response, No such name
1... .... .... .... = Response: Message is a response
.000 0... .... .... = Opcode: Standard query (0)
.... .0.. .... .... = Authoritative: Server is not an authority
for domain
.... ..0. .... .... = Truncated: Message is not truncated
.... ...1 .... .... = Recursion desired: Do query recursively
.... .... 1... .... = Recursion available: Server can do recursive
queries
.... .... .0.. .... = Z: reserved (0)
.... .... ..0. .... = Answer authenticated: Answer/authority
portion was not authenticated by the server
.... .... ...0 .... = Non-authenticated data: Unacceptable
.... .... .... 0011 = Reply code: No such name (3)
Questions: 1
Answer RRs: 1
Authority RRs: 1
Additional RRs: 0
Queries
stash.externaldomain.net: type A, class IN
Name: stash.externaldomain.net
[Name Length: 21]
[Label Count: 3]
Type: A (Host Address) (1)
Class: IN (0x0001)
Answers
stash.externaldomain.net: type CNAME, class IN, cname
git1.externaldomain.net
Name: stash.externaldomain.net
Type: CNAME (Canonical NAME for an alias) (5)
Class: IN (0x0001)
Time to live: 22483
Data length: 20
CNAME: git1.externaldomain.net
Authoritative nameservers
externaldomain.net: type SOA, class IN, mname
van-dns1.externaldomain.net
Name: externaldomain.net
Type: SOA (Start Of a zone of Authority) (6)
Class: IN (0x0001)
Time to live: 518
Data length: 38
Primary name server: van-dns1.externaldomain.net
Responsible authority's mailbox: tech.externaldomain.net
Serial Number: 2015092101
Refresh Interval: 10800 (3 hours)
Retry Interval: 900 (15 minutes)
Expire limit: 604800 (7 days)
Minimum TTL: 86400 (1 day)
> We have a FreeIPA domain running IPA server 4.1.4 on CentOS 7.
>
> We have no per zone forwarding enabled, only a single global forwarder.
> This seems to work fine, but then after a while (several weeks I think)
> will randomly stop working.
>
> We had this issue several weeks ago on a different IPA domain (identical
> setup) in our production network but it was ignored because a server
> restart fixed it.
>
> This issue then re-surfaced in our development domain today (different
> network, different physical hardware, same OS and IPA versions).
>
> I received a report today from a developer that he could not ping a
> machine in another domain so I verified network connectivity and
> everything was fine. When I tried to resolve the name from the IPA dc
> using ping it would fail, but nslookup directly to the forward server
> worked fine.
>
> ipactl showed no issues, and only after I restarted the server did the
> lookups start working again.
>
> Console log below :
>
> Using username "myipausername".
> Last login: Thu Oct 1 16:36:51 2015 from 10.5.5.57
> [myipausername at dc1 ~]$ sudo su -
> Last login: Tue Sep 29 19:03:39 UTC 2015 on pts/3
>
> ATTEMPT FIRST PING TO UNRESOLVABLE HOST
> =======================================
> [root at dc1 ~]# ping artifactory.externaldomain.net
> ping: unknown host artifactory.externaldomain.net
>
> CHECK IPA STATUS
> ================
> [root at dc1 ~]# ipactl status
> Directory Service: RUNNING
> krb5kdc Service: RUNNING
> kadmin Service: RUNNING
> named Service: RUNNING
> ipa_memcached Service: RUNNING
> httpd Service: RUNNING
> pki-tomcatd Service: RUNNING
> smb Service: RUNNING
> winbind Service: RUNNING
> ipa-otpd Service: RUNNING
> ipa-dnskeysyncd Service: RUNNING
> ipa: INFO: The ipactl command was successful
>
> ATTEMPT PING OF GLOBAL FORWARDER
> ================================
> [root at dc1 ~]# ping 10.21.0.14
> PING 10.21.0.14 (10.21.0.14) 56(84) bytes of data.
> 64 bytes from 10.21.0.14: icmp_seq=1 ttl=64 time=0.275 ms
> 64 bytes from 10.21.0.14: icmp_seq=2 ttl=64 time=0.327 ms
> ^C
> --- 10.21.0.14 ping statistics ---
> 2 packets transmitted, 2 received, 0% packet loss, time 1000ms
> rtt min/avg/max/mdev = 0.275/0.301/0.327/0.026 ms
>
> MANUAL NSLOOKUP OF DOMAIN ON GLOBAL FORWARDER FROM IPA DC
> =========================================================
> [root at dc1 ~]# nslookup
>> server 10.21.0.14
> Default server: 10.21.0.14
> Address: 10.21.0.14#53
>> artifactory.externaldomain.net
> Server: 10.21.0.14
> Address: 10.21.0.14#53
>
> Non-authoritative answer:
> artifactory.externaldomain.net canonical name =
> van-artifactory1.externaldomain.net.
> Name: van-artifactory1.externaldomain.net
> Address: 10.20.10.14
>
> RE-ATTEMPT PING SINCE WE KNOW THAT NAME RESOLUTION (at least via nslookup
> IS WORKING FROM THIS MACHINE
> ======================================================================================================
>> ^C[root at dc1 ~]# ping artifactory.externaldomain.net
> ping: unknown host artifactory.externaldomain.net
> [root at dc1 ~]# ping van-artifactory1.externaldomain.net
> ping: unknown host van-artifactory1.externaldomain.net
>
> RESTART IPA SERVICES
> ====================
> [root at dc1 ~]# ipactl restart
> Restarting Directory Service
> Restarting krb5kdc Service
> Restarting kadmin Service
> Restarting named Service
> Restarting ipa_memcached Service
> Restarting httpd Service
> Restarting pki-tomcatd Service
> Restarting smb Service
> Restarting winbind Service
> Restarting ipa-otpd Service
> Restarting ipa-dnskeysyncd Service
> ipa: INFO: The ipactl command was successful
> [root at dc1 ~]# ipa dnsconfig-show
> ipa: ERROR: did not receive Kerberos credentials
> [root at dc1 ~]# kinit myipausername
> Password for myipausername at ipadomain.NET:
>
> OUTPUT GLOBAL FORWARDER CONFIG FOR TROUBLESHOOTING
> ==================================================
> [root at dc1 ~]# ipa dnsconfig-show
> Global forwarders: 10.21.0.14
> Allow PTR sync: TRUE
>
> PING NOW WORKS BECAUSE IPA SERVICES WERE RESTARTED
> ==================================================
> [root at dc1 ~]# ping artifactory.externaldomain.net
> PING van-artifactory1.externaldomain.net (10.20.10.14) 56(84) bytes of
> data.
> 64 bytes from 10.20.10.14: icmp_seq=1 ttl=60 time=3.00 ms
> 64 bytes from 10.20.10.14: icmp_seq=2 ttl=60 time=1.42 ms
> 64 bytes from 10.20.10.14: icmp_seq=3 ttl=60 time=2.39 ms
> ^C
> --- van-artifactory1.externaldomain.net ping statistics ---
> 3 packets transmitted, 3 received, 0% packet loss, time 2004ms
> rtt min/avg/max/mdev = 1.420/2.274/3.004/0.653 ms
> [root at dc1 ~]#
>
> Here are some strange enties from my /var/log/messages relating to errors
> from today :
>
> Oct 1 20:39:31 dc1 named-pkcs11[15066]: checkhints: unable to get root NS
> rrset from cache: not found
> Oct 1 20:39:17 dc1 named-pkcs11[15066]: error (network unreachable)
> resolving 'pmdb1.ipadomain.net/A/IN': 2001:500:2f::f#53
> Oct 1 20:39:17 dc1 named-pkcs11[15066]: error (network unreachable)
> resolving 'pmdb1.ipadomain.net/AAAA/IN': 2001:500:2f::f#53
>
> Looking at the log entries, it appears that there may have been a network
> connectivity 'blip' (maybe a switch or router was restarted) at some point
> and even after connectivity was restored, the global forwarding was
> failing because the "we can't contact our forwarder" status seemed to get
> stuck in memory.
>
>
> --
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project
>
More information about the Freeipa-users
mailing list