Debug DNS BIND

Last weekend I found myself in the position that the name servers in our main datacenter where not responding anymore. To solve the problem I needed to check all kinds of things, I needed to check the network, routing, and the BIND server itself. So for those of you who will be in a position like this someday here are some of the commands I used to debug.

You can debug BIND in a couple of ways. If you would like to see what queries are executed against the BIND server you can put your bind server in querylog mode. This will result in all query information being dumped to /var/log/messages so you can trace all queries executed by doing a tail –f /var/log/messages

Activating the querylog for bind:
Execute the command rndc querylog . This can result in the following error:
rndc: neither /etc/bind/rndc.conf nor /chroot/named/etc/bind/rndc.key was found

This means you have to set the location of your rndc.key which was in my case done with the following:
rndc –k /etc/bind/rndc,key querylog

now all the query information is dumped to /var/log/messages so the tail -f /var/log/messages will show you a ongoing list of queries which are executed at the moment in real-time. For example:
Jul 15 11:51:25 pubmai01 named[15274]: client 10.32.5.2#35242: query: google.com IN A +
Jul 15 11:51:26 pubmai01 named[15274]: client 10.96.96.39#53159: query: www.google.nl IN AAAA +
Jul 15 11:51:26 pubmai01 named[15274]: client 10.96.96.39#53882: query: www.google.nl IN A +
Jul 15 11:51:26 pubmai01 named[15274]: client 10.32.5.253#61121: query: 28.2.32.10.in-addr.arpa IN PTR +
Jul 15 11:51:32 pubmai01 named[15274]: client 127.0.0.1#58547: query: 50.195.67.92.sbl-xbl.spamhaus.org IN TXT +
Jul 15 11:51:32 pubmai01 named[15274]: client 10.32.5.2#35242: query: knmi.nl IN A +
Jul 15 11:51:34 pubmai01 named[15274]: client 10.32.5.253#61121: query: 20.2.32.10.in-addr.arpa IN PTR +
Jul 15 11:51:37 pubmai01 named[15274]: client 10.32.5.2#35242: query: nasa.gov IN A +
Jul 15 11:51:38 pubmai01 named[15274]: client 10.96.96.39#59327: query: www.marktplaats.nl IN AAAA +
Jul 15 11:51:38 pubmai01 named[15274]: client 10.96.96.39#58879: query: www.marktplaats.nl IN A +
Jul 15 11:51:38 pubmai01 named[15274]: client 10.96.96.39#59364: query: statisch.marktplaats.nl IN AAAA +
Jul 15 11:51:38 pubmai01 named[15274]: client 10.96.96.39#59768: query: statisch.marktplaats.nl IN A +

This way you can see if the query you execute is being picked up and handled. To do a manual lookup you can use the dig command:

“dig (domain information groper) is a flexible tool for interrogating DNS name servers. It performs DNS lookups and displays the answers that are returned from the name server(s) that were queried. Most DNS administrators use dig to troubleshoot DNS problems because of its flexibility, ease of use and clarity of output. Other lookup tools tend to have less functionality than dig.”

To make sure that you are calling the server you have set in querylog mode you have to explicitly query this server if it is not your default name server. To do this you have to add the ip of the nameserver you like to query. Let say this is the 10.32.5.1 you have to issue the command to query for the information of www.knmi.nl like this dig @10.32.5.1 www.knmi.nl

Now if all is ok you should see something in the query log files stating something like:
Jul 15 11:51:32 pubmai01 named[15274]: client 10.32.5.2#35242: query: knmi.nl IN A +

If this is not the case the best thing to do is to test if you can get it from another name server for example the ns1.kpn.net which is the first nameserver owned by kpn, it has IP 194.151.228.2. So you can execute a command like dig @194.151.228.2 www.knmi.nl this should give you a result. If this is not the case you have to check if you are allowed by firewalls and other security measures to initiate and receive UDP traffic.

If you do get information back from the server you also can do a dig +trace

“Toggle tracing of the delegation path from the root name servers for the name being looked up. Tracing is disabled by default. When tracing is enabled, dig makes iterative queries to resolve the name being looked up. It will follow referrals from the root servers, showing the answer from each server that was used to resolve the lookup.”

However before we go and check all kinds of firewalls you might want to do some more debugging and logging on the BIND server, you can put bind in a log mode by executing the following command:

rndc -k /etc/bind/rndc.key trace

This will put BIND in a debug trace mode, executing the command again raises the trace level. You can check on which tracelevel you are by executing:

rndc -k /etc/bind/rndc.key status

When running in debug trace mode you can do a tail -f /var/bind/named.run this will give you a good insight in what actions are being executed by you BIND DNS server.