Service Discovery and Health Checking with Consul

First off, what is Consul? Following from my last post on Sensu, Consul can be used for some monitoring but it is mostly a tool for discovering and configuring services. Key features:

  • Service Discovery:
    Clients of Consul announce to the Consul server that they provide a service and other clients can use the Consul server to discover service providers.
  • Health Checking:
    Similar to Sensu, Consul can run health checks against services or components on the client nodes. This information is sent to the Consul server(s) and is used to monitor cluster health. Consul will use the reported health information in combination with it’s service discovery components to route traffic away from unhealthy hosts.
  • Key/Value Store:
    Stores key/value pairs. Useful for things like dynamic configuration, feature flagging, and other values. The api is pretty easy to use.
  • Multi Datacenter:
    Consul supports the concept of multiple datacenters or physical locations from which it’s clients are running.

Setting up a Consul Server

To get started with Consul, download and install the Consul binary from their website: https://www.consul.io/

wget https://dl.bintray.com/mitchellh/consul/0.5.0\_linux\_amd64.zip

Unzip it:

unzip 0.5.0\_linux\_amd64.zip

Move the binary somewhere useful:

mv consul /bin/

Start Consul in server mode:

consul agent -server -bootstrap-expect 1 -data-dir /tmp/consul

In another terminal window on the Consul server you can run:

consul members

This shows the membership information for your current Consul cluster. Add a -detailed flag to get more information on the nodes.

The downside to the consul members command is that it uses the gossip protocol, and so is just eventually consistent. For a more consistent view, we would use the HTTP API:

curl localhost:8500/v1/catalog/nodes

We have a node running now. Let’s configure a service for Consul to know of.

Adding a Service to Consul

Create a directory to contain the service definition(s):

mkdir /etc/consul.d

Create a file with the following service definition:

{
    "service": {
        "name": "consul-web",
        "tags": \["apache"\],
        "port": 80
    }
}

Back on our terminal window running the server agent - kill it with CTRL-C, and restart it with the new directory provided:

consul agent -server -bootstrap-expect 1 -data-dir /tmp/consul -config-dir /etc/consul.d

You can now query for information about the service! Example:

\# Using DNS:
dig @127.0.0.1 -p 8600 consul-web.service.consul

# DNS again, but with a SRV record so we get the port info:
dig @127.0.0.1 -p 8600 consul-web.service.consul SRV

Joining nodes to a cluser is pretty easy (run this from clients):

consul join \[ip\]

A service definition should also be associated with a health check definition… so modify our service definition so that it looks like:

{
    "service": {
        "name": "consul-web",
        "tags": \["apache"\],
        "port": 80,
        "check": {
            "script": "curl localhost >/dev/null 2>&1",
            "interval": "10s"
        }
    }
}

We’ve added a check field. This drives the magic of health checking.

Restart the server node.

Once the server node is back up, you can query the service health with:

curl http://localhost:8500/v1/health/state/critical

If you query the service via DNS, it will return no results for an unhealthy service.

Storing Key/Values in Consul

We can check to see if there are existing keys with:

curl http://localhost:8500/v1/kv/?recurse

Currently we should have no keys stored. Let’s add a key:

curl -X PUT -d 'test' http://localhost:8500/v1/kv/web/key1

Now lets grab that key:

curl http://localhost:8500/v1/kv/web/key1

This returns:

\[{"CreateIndex":60,"ModifyIndex":62,"LockIndex":0,"Key":"web/key1","Flags":0,"Value":"dGVzdA=="}\]

The value is a base64 encoded blob of data. Fix that with base64 --decode:

curl -s http://localhost:8500/v1/kv/web/key1 | jq -c '.\[\].Value' | tr -d '"' | base64 --decode

The Consul Web UI

All of these command line options are great, but what about a dashboard for humans? Consul does have a dashboard package!

On our Consul server download the web package:

wget https://dl.bintray.com/mitchellh/consul/0.5.0\_web\_ui.zip

Unzip it:

unzip 0.5.0\_web\_ui.zip

Place it somewhere that makes sense:

mv dist /etc/consul-web

Restart the server with a -ui-dir flag:

consul agent -server -bootstrap-expect 1 -data-dir /tmp/consul -config-dir /etc/consul.d -ui-dir /etc/consul-web -client \[ip\]

We also added a -client flag with the IP address from the box. This changes it from listening on the loopback, to an address we can reach.

Consul Web UI end result (http://[ip]:8500/ui/#/dc1/services):