Oracle Linux | Consul ServiceAddress from multiple datacenters

Consul, developed by HashiCorp, is a tool commonly used for service discovery and service registration in microservice based landscapes. When running a distributed landscape spanning multiple datacenters or multiple Oracle Cloud zones you will, most likely, be running multiple cosul datacenter clusters. A consul datacenter cluster will take care of all service registration requests and all service discovery requests for this specific datacenter. Service discovery can be done based upon a REST API interface or it can be done based upon a DNS resolve mechanism.

Having stated that a Consul datacenter cluster takes care of all service registered in that datacenter is not fully true. You can connect the Consul clusters in the different datacenters or Oracle Cloud zones together with a concept called WAN gossip. Using WAN gossip the clusters will be aware of each other and you can use you local cluster to query services in the other datacenter.

An example of the construct of a FQDN is shown below. In this example we query a service named testservice in datacenter dc1. A DNS resolve on this FQDN will give you all the IP addresses for servers (or Docker containers) running this service.

testservice.service.dc1.consul

if we query the same Consul cluster without dc1 (testservice.service.consul) we will get exactly the same IP's as the DNS service from Consul will default the datacenter name to the name he is responsible for. however, if we query testservice.service.dc2.consul we will get a list of all IP's for  instances for the testservice registered in the other datacenter.

In a lot of cases this is a very workable solution and it will solve most of the situations, it also gives a level of safeguard against cross-dc requests. However, in some cases you would like to have a full list of all IP's for a services without taking into consideration in which DC they reside.

In effect, Consul is not supporting this out of the box at this moment. If you use the API and not DNS the below command is the quickest way to get a list of all IP addresses for a service without taking into account in which datacenter they reside.

[vagrant@consul-dc2 testjson]$ { curl -s "http://localhost:8500/v1/catalog/service/testservice?dc=dc1" ; curl -s "http://localhost:8500/v1/catalog/service/testservice?dc=dc2" ; } | jq '.[].ServiceAddress'
"10.0.1.16"
"10.0.1.17"
"10.0.1.15"
"10.0.1.16"
"10.0.2.20"
"10.0.2.15"
"10.0.2.16"
[vagrant@consul-dc2 testjson]$ 

A similar command could be made using dig and leveraging the DNS way of doing things. However, for some reason it feels like this is a missing option in Consul.

Do note, we use JQ also, JQ is available from the Oracle Linux YUM repository, however, you will have to enable an additional channel as it is not available in the default enabled ones.