Oracle Linux - Using Consul for DNS based service discovery

Consul, developed by hashicorp,  is a solution for service discovery and configuration. Consul is completely distributed, highly available, and scales to thousands of nodes and services across multiple datacenters. Some concrete problems Consul solves: finding the services applications need (database, queue, mail server, etc.), configuring services with key/value information such as enabling maintenance mode for a web application, and health checking services so that unhealthy services aren't used. These are just a handful of important problems Consul addresses.

When developing microservices it is important that as soon as new instance of a microservice comes online it is able to register itself to a central registry, this process we call service registration. As soon as an instance of a microservice is registered at the central registry it can be used in the load balancing mechanism. If a call to a service is to be initiated to a microservice the service needs to be discovered via the central registry, the process is called service discovery.

In effect two ways are common for service discovery. One is based upon an API discovery model where the calling service discovers the service by executing a call to the service registry based upon a HTTP REST API call and receiving an endpoint which can be used. Commonly an URL based upon a IP and a port number.

The other common way is using a DNS based lookup against a service registry. The effect of doing a DNS based lookup is that you need to ensure that all the instances of a service are always running on the same port on all instance. Enforcing the same port number might be somewhat limiting in cases where your port number for each service instance can vary.

Using Consul on Oracle Linux
In an earlier blogpost I already outlined how to install Consul on Oracle Linux. In this post I will provide a quick inisght in how you can configure it on Oracle Linux. We build upon the installation done in the mentioned post.

We take the example of a service where we have two instances for, the name of the service is web and in effect is nothing more than a simple nginx webserver running in a production instance. Every time we call the service we want to discover the call by using a DNS lookup, we do however want to have this balanced, meaning we want to have a different IP being returned from the DNS server.

Configure a service in consul
We configure the service "web" manually in this case by creating a JSON file in /etc/consul.d and ensure we have the information about the two instances in this JSON file. An example of the file is shown below

{
 "services": [{
   "id": "web0",
   "name": "web",
   "tags": ["production"],
   "Address": "191.168.1.10",
   "port": 80
  },
  {
   "id": "web1",
   "name": "web",
   "tags": ["production"],
   "Address": "191.168.1.11",
   "port": 80
  }
 ]
}

As you can see, we have one name, "web" with two ID's; "web0" and "web1". The id's are used to identify the different instances of the service web. as you can see they have both a port noted next to the address. Even though it is good practice to have this in the configuration file it will not be used in the response from the internal consul DNS service as DNS will only return the addresses and not the ports.

Discover the service via Consul DNS
If we want to discover the service we can have our code to undertake a lookup against the DNS server. If we have configured the underlying Oracle Linux instance to have the Consul server in your /etc/resolv.conf file it will happen almost automatically. It will be important to make sure the ordering of your DNS servers is done correctly to improve resolving speed.

In effect all services configured in Consul will be by default part of .service.consul which will mean that if we want to do a DNS resolving for the web service we will have to do a resolve for web.service.consul. In the below example I have consul running on my localhost at port 8600 and I use dig to explicitly force dig to resolve it at this DNS server. As stated, if you configure it correctly you do not have to explicitly call it and you should be able to do a DNS resolve as you always do.

[root@localhost consul.d]#
[root@localhost consul.d]# dig +noall +answer @127.0.0.1 -p 8600 web.service.consul 
web.service.consul. 0 IN A 191.168.1.10
web.service.consul. 0 IN A 191.168.1.11
[root@localhost consul.d]#

As you can see from the above example I will get the two addresses returned from the Consul DNS server.

Consul and load-balancing
As microservices are commonly build up out of a number of instances of the same service we do want to ensure that load-balancing is done. We can already see from the dig example above that there are two instances. However, having them always in the same order returned will not ensure that the load is balanced over the two instances.

Consul will by default do a load-balancing and will return the IP's in a different order by rotating them in the DNS response. In the below example you can see that this is done when we call the DNS server a couple of times.

[root@localhost consul.d]#
[root@localhost consul.d]# dig +noall +answer @127.0.0.1 -p 8600 web.service.consul 
web.service.consul. 0 IN A 191.168.1.10
web.service.consul. 0 IN A 191.168.1.11
[root@localhost consul.d]#
[root@localhost consul.d]# dig +noall +answer @127.0.0.1 -p 8600 web.service.consul 
web.service.consul. 0 IN A 191.168.1.11
web.service.consul. 0 IN A 191.168.1.10
[root@localhost consul.d]#

In conclusion
If you are running a microservices based IT footprint and you are using Oracle Linux you can read in the referenced article how to install Consul to do service discovery and registration. Consul supports both API based as well as DNS based discovery. If you have your service instances always on the same and pre-defined port using DNS is a very good option to use for your service discovery process.