Last weekend I found myself in the position that the name servers in our main datacenter where not responding anymore. To solve the problem I needed to check all kinds of things, I needed to check the network, routing, and the BIND server itself. So for those of you who will be in a position like this someday here are some of the commands I used to debug.
You can debug BIND in a couple of ways. If you would like to see what queries are executed against the BIND server you can put your bind server in querylog mode. This will result in all query information being dumped to /var/log/messages so you can trace all queries executed by doing a tail –f /var/log/messages
Activating the querylog for bind:
Execute the command rndc querylog . This can result in the following error:
rndc: neither /etc/bind/rndc.conf nor /chroot/named/etc/bind/rndc.key was found
This means you have to set the location of your rndc.key which was in my case done with the following:
rndc –k /etc/bind/rndc,key querylog
now all the query information is dumped to /var/log/messages so the tail -f /var/log/messages will show you a ongoing list of queries which are executed at the moment in real-time. For example:
Jul 15 11:51:25 pubmai01 named[15274]: client 10.32.5.2#35242: query: google.com IN A +
Jul 15 11:51:26 pubmai01 named[15274]: client 10.96.96.39#53159: query: www.google.nl IN AAAA +
Jul 15 11:51:26 pubmai01 named[15274]: client 10.96.96.39#53882: query: www.google.nl IN A +
Jul 15 11:51:26 pubmai01 named[15274]: client 10.32.5.253#61121: query: 28.2.32.10.in-addr.arpa IN PTR +
Jul 15 11:51:32 pubmai01 named[15274]: client 127.0.0.1#58547: query: 50.195.67.92.sbl-xbl.spamhaus.org IN TXT +
Jul 15 11:51:32 pubmai01 named[15274]: client 10.32.5.2#35242: query: knmi.nl IN A +
Jul 15 11:51:34 pubmai01 named[15274]: client 10.32.5.253#61121: query: 20.2.32.10.in-addr.arpa IN PTR +
Jul 15 11:51:37 pubmai01 named[15274]: client 10.32.5.2#35242: query: nasa.gov IN A +
Jul 15 11:51:38 pubmai01 named[15274]: client 10.96.96.39#59327: query: www.marktplaats.nl IN AAAA +
Jul 15 11:51:38 pubmai01 named[15274]: client 10.96.96.39#58879: query: www.marktplaats.nl IN A +
Jul 15 11:51:38 pubmai01 named[15274]: client 10.96.96.39#59364: query: statisch.marktplaats.nl IN AAAA +
Jul 15 11:51:38 pubmai01 named[15274]: client 10.96.96.39#59768: query: statisch.marktplaats.nl IN A +
This way you can see if the query you execute is being picked up and handled. To do a manual lookup you can use the dig command:
“dig (domain information groper) is a flexible tool for interrogating DNS name servers. It performs DNS lookups and displays the answers that are returned from the name server(s) that were queried. Most DNS administrators use dig to troubleshoot DNS problems because of its flexibility, ease of use and clarity of output. Other lookup tools tend to have less functionality than dig.”
To make sure that you are calling the server you have set in querylog mode you have to explicitly query this server if it is not your default name server. To do this you have to add the ip of the nameserver you like to query. Let say this is the 10.32.5.1 you have to issue the command to query for the information of www.knmi.nl like this dig @10.32.5.1 www.knmi.nl
Now if all is ok you should see something in the query log files stating something like:
Jul 15 11:51:32 pubmai01 named[15274]: client 10.32.5.2#35242: query: knmi.nl IN A +
If this is not the case the best thing to do is to test if you can get it from another name server for example the ns1.kpn.net which is the first nameserver owned by kpn, it has IP 194.151.228.2. So you can execute a command like dig @194.151.228.2 www.knmi.nl this should give you a result. If this is not the case you have to check if you are allowed by firewalls and other security measures to initiate and receive UDP traffic.
If you do get information back from the server you also can do a dig +trace
“Toggle tracing of the delegation path from the root name servers for the name being looked up. Tracing is disabled by default. When tracing is enabled, dig makes iterative queries to resolve the name being looked up. It will follow referrals from the root servers, showing the answer from each server that was used to resolve the lookup.”
However before we go and check all kinds of firewalls you might want to do some more debugging and logging on the BIND server, you can put bind in a log mode by executing the following command:
rndc -k /etc/bind/rndc.key trace
This will put BIND in a debug trace mode, executing the command again raises the trace level. You can check on which tracelevel you are by executing:
rndc -k /etc/bind/rndc.key status
When running in debug trace mode you can do a tail -f /var/bind/named.run this will give you a good insight in what actions are being executed by you BIND DNS server.
Oracle has announced what the roadmap will be for the fusion middleware stack in combination with the acquisition of BEA. In a webcast by Charles Phillips "president Oracle Corporation" and Thomas Kurian "Senior Vice President Oracle Server Technologies Development", Oracle announced that the WebLogic Server will become the Oracle JEE platform over OC4J.
Part of this roadmap is the promotion of the AquaLogic Server Bus into the Oracle SOA Suite as the preferred ESB.
So what we will, most likely, be seeing is that a lot of Oracle Application Server people will be shifting their interest to BEA products to keep an advantage in knowledge and be prepared for the upcoming BEA wave which will be coming from Oracle.
I just came across a website called visualcomplexity.com, here you can find some of the best and most cool network visualizations out there. Not only visualization of computer networks but also social networks, business networks and more. Cool to have look at.
When you have worked with Oracle OBIEE you will have noticed that it is quiet web2.0 cool looking and nice to work with. To make this even more attractive now you can run it on a Apple iPhone. Oracle releases a press release named “Oracle Announces Oracle® Business Applications for iPhone Available on Apple App Store”.
You can now download for free from the apple appstore the needed tools to access Oracle applications from your iPhone.
So the question was, is the iPhone a business phone or not? Wel the answer is yes, you can now access your Business Intelligence Oracle applications from your iPhone, 2 very good things in your hand. All the knowledge of your business and a fun tool like a iPhone.
I am using my mobile phone a lot, and I have to recharge my phone a lot because for one part my battery is not as it used to be and for the other part I make a lot of calls with my cellphone. Sometimes I am not in the position to recharge my phone, think about when you are camping, yes I carry my work phone with me during my holidays, think about when commuting in the train. The Dutch trains do not (yet) have a powersupply for the travelers.
The solution is then to have a solar powered charger. I knew a friend have orderd one form Japan some time ago. However in Holland you can now get them from some stores. So I have purchased a solar power charger and I have to say,…. I love it… and I will keep using it. So from now on I will be charging my phone green whenever I can. There will however be times I cannot, think about in the middle of the night after a call intensive day. If you are in the Netherlands and like to purchase one of those go and have a look at MyCom.
For the company I am currently working for as an Oracle functional and technical consultant I have to give a presentation from time to time. The first couple of slides are almost every time the same, SmartApps is a Oracle SaaS hosting company and is doing Oracle E-Business suite implementations. SmartApps is targeting the SMB companies with a tendency to the mid market companies. Also we do work for some of the bigger companies in the world for Oracle products.
Some things about datacenters, solutions, the way we use a accelerator solution to implement Oracle E-Business suite…. Now I have made a standard opening. Remember this is version 1.0 so I will be changed it is however giving you a good first view about SmartApps B.V.
I have posted it on slideshare.com which is a great site.
I am currently working on setting up a test system for Oracle XML gateway which should forward purchase orders in Oracle purchasing with the XML gateway as a XML EDI message to an other Oracle E-Business suite system. Because we have been testing all kinds of setup for this project which also includes changes to configuration files on UNIX level I have stopped and started the system a couple of times to be sure that changes would be picked up.
During the last stop start session somehow the workflow manager and the concurrent managers where not coming up again. The first thing we noticed that after approving a purchase order the order “disappeared”. After some searching we found out that not everything was up and running so I started all the processes in the Oracle Applications Manager.
However when reviewing the notification list we found that all the purchase orders who have been approved during the time the background processes where down where in a status of “PO Approval Error” stating as details “Document Manager Failed With Error Number 3 while processing Standard Purchase Order 310”.
Checking metalink note 1074838.6 on this error is stating: The document is left in status IN PROCESS. The WFSTATUS.sql output file shows that the process failed at node Open Document State during Verify PO and defaulted to Document Manager Failed.
And we found that this is indeed the case. To resolve the troubled purchase orders you can go into the notification and use the retry option. When using this the purchase order will be approved and will follow the normal workflows as you have set them up.
Last weekend I found myself in the position that the name servers in our main datacenter where not responding anymore. To solve the problem I needed to check all kinds of things, I needed to check the network, routing, and the BIND server itself. So for those of you who will be in a position like this someday here are some of the commands I used to debug.
