solved Oracle NoSQL java.net.NoRouteToHostException

When you are installing Oracle NoSQL on an Oracle linux machine you can follow the Oracle guide which will run you through the simple installation process. There are however some things to keep into consideration. One of the steps is to do a check if you key-value store database is up and running. When you use a default Oracle Linux installation you will most likely succeed when you do a ping to the same host that you are working on. The issue however is starting to occur when you try to do a ping to another machine.

The following situation:
nosql0.exampledomain.com -- 192.168.1.80
nosql1.exampledomain.com -- 192.168.1.81

When you are on nosql0 and execute the following below command you will have a positive result.

java -jar ./lib/kvstore-1.2.123.jar ping -port 5000 -host nosql0.exampledomain.com

If you are on nosql1 and you execute the below command you will also get a positive result.
java -jar ./lib/kvstore-1.2.123.jar ping -port 5000 -host nosql1.exampledomain.com 

However, if you are on nosql0 and try to ping the nosql1 with the below command you will get an error.
java -jar ./lib/kvstore-1.2.123.jar ping -port 5000 -host nosql1.exampledomain.com  

if you use a ping command you can ping the other server (if not you have another network issue) and you can setup a SSH session however the error message will state that you do not have a route to the mentioned host. The error message will look something like the one below:

[nosql@nosql0 kv-1.2.123]$ java -jar ./lib/kvstore-1.2.123.jar ping -port 5000 -host nosql1.terminalcultexample.org
Exception in thread "main" java.rmi.ConnectIOException: Exception creating connection to: nosql1.terminalcultexample.org; nested exception is:
        java.net.NoRouteToHostException: No route to host
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:632)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
        at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:340)
        at sun.rmi.registry.RegistryImpl_Stub.list(Unknown Source)
        at oracle.kv.util.Ping.getTopology(Ping.java:332)
        at oracle.kv.util.Ping.main(Ping.java:104)
        at oracle.kv.impl.util.KVStoreMain$8.run(KVStoreMain.java:218)
        at oracle.kv.impl.util.KVStoreMain.main(KVStoreMain.java:319)
Caused by: java.net.NoRouteToHostException: No route to host
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
        at java.net.Socket.connect(Socket.java:546)
        at java.net.Socket.connect(Socket.java:495)
        at java.net.Socket.(Socket.java:392)
        at java.net.Socket.(Socket.java:206)
        at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
        at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:146)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
        ... 8 more
[nosql@nosql0 kv-1.2.123]$

As it turns out by default Oracle Linux will have iptables configured and this will block your connection to port 5000. You can check you iptables settings by issuing the following command: iptables -L -n


This will give you something like the below:

[root@nosql1 init.d]# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22
REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
[root@nosql1 init.d]#

What you need to do is configure iptables to allow network traffic to port 5000 or disable iptables. Disabling iptables is never a smart move however you can opt for it in some cases.