Consul + Fabio + Your Application - monitoring and Load Balance

July 2018

Sysinfo: Centos 7, Consul 0.9.2, hypothetical Pong application

Background

Need a good way to rout users to a custom application (lets call it Pong, running on Port 8300 (via Apache on port 80). There are 3 nodes running Pong and these will be monitored by Consul

Fabio comes in as an add-on to Consul and does the actual Routing and maintains its own Routing tables based on Consul data.

Fabio is great because you dont need to update Routing table, its completely automated. It reads whatever Service and host is on Consul and works off that.

Note: This article is not a good solution for apps that need persistent sessions. Consul and Fabio do not support session persistence. This setup is best used for small applications and microservices. For large applications like Splunk or anything Java-based, use HAProxy to balance nodes

Basic structure:

1. Users go to a shared (virtual) IP, lets say "pong.company.com" -> this is resolved by DNS to 10.155.20.5

2. Virtual IP then passes the request to one of the 2 Consul/Fabio servers

3. Fabio reads healthcheck data from Consul, determines there are 3 active Pong

Configure Consul server cluster

Lets create a 2 node Consul cluster. Consul will monitor application service health, and Fabio will run on top of each Consul and get its routing tables from Consul data.

1. on each Consul server, install Consul

yum install consul

2. generate an Encryption key & Master Token

consul keygen

UAvkAzdjGfQ7J2NlgkrJMA==

geneate a Master Token

uuidgen

dbef8b5a-6110-4575-bf61-dda1c21ca339

3. create Consul dirs

mkdir -p /etc/consul.d/server

mkdir /var/consul

4. add Consul user + group

groupadd consul

useradd consul -g consul

5. change permissions

chown -R consul:consul /var/consul

Consul Config

6. on the 1st Consul node, create new Boostrap config,

{

"bind_addr": "10.185.20.180",

"client_addr": "0.0.0.0",

"data_dir": "/var/consul",

"server": true,

"ui": true,

"bootstrap": true,

"retry_join": ["10.185.20.179","10.185.20.180"],

"datacenter": "mrx",

"enable_script_checks": true,

"encrypt": "UAvkAzdjGfQ7J2NlgkrJMA==",

"enable_syslog": true,

"addresses": {

"http": "10.185.20.180",

"dns": "10.185.20.180"

},

"dns_config": {

"allow_stale": true,

"max_stale": "30s",

"node_ttl": "30s",

"enable_truncate": true,

"only_passing": true

},

"acl_datacenter": "mrx",

"acl_down_policy": "extend-cache",

"acl_default_policy": "allow",

"acl_master_token": "dbef8b5a-6110-4575-bf61-dda1c21ca339"

}

validate the syntax

consul validate /etc/consul.d/*

fix any validation errors

Startup Service

7. create Startup Service

vim /usr/lib/systemd/system/consul.service

[Unit]

Description=Consul service discovery agent

Requires=network-online.target

After=network.target

[Service]

User=consul

Group=consul

Restart=on-failure

ExecStartPre=[ -f "/var/consul/consul.pid" ] && /usr/bin/rm -f /var/consul/consul.pid

ExecStart=/usr/bin/consul agent -config-dir=/etc/consul.d -config-file=/etc/consul.d/server/consul.json

ExecReload=/bin/kill -s HUP $MAINPID

KillSignal=SIGINT

TimeoutStopSec=5

[Install]

WantedBy=multi-user.target

systemctl enable consul.service

systemctl daemon-reload

systemctl start consul.service

8. Do the same for 2 remaining Consul servers, change "bootstrap": false

9. Start service, Web console should be up at <IP>:8500

UI should be available

Create ACL on Consul cluster

Login to Web UI, click on config gear, enter Master ACL token, click Close, this will save the Master token access between your browser and Consul cluster (this is better than using username/password)

Now click ACL button, update Anonymous Token ACL to let Read-Only access

Now create a new ACL for Pong service,

service "" { policy = "write" }

key "pong/" { policy = "write" }

node "" { policy = "write" }

session "" { policy = "write" }

get the Token ID of this Pong ACL policy

go back to each Consul node, change the consul.json setting, change to "Deny"

"acl_default_policy": "deny"

restart Consul service on all server nodes

Configure Consul agent on each Pong instance

on each Pong node, install Consul

Configure Consul agent on each Pong instance, for ACL token add the Pong ACL token from above

pong01> vim /etc/consul.d/client/consul.json

{ "bind_addr": "10.185.20.173", "data_dir": "/var/consul", "ui": false, "bootstrap": false, "server": false, "start_join": ["10.185.20.179","10.185.20.180"], "datacenter": "mrx", "encrypt": "UAvkAzdjGfQ7J2NlgkrJMA==", "enable_syslog": false, "enable_script_checks": true, "pid_file": "/var/consul/consul.pid", "acl_token": "548bb56f-33c9-622a-4351-1a04851ebb1a" }

Service Health Check

pong01> vim /etc/consul.d/pong.json

{

"service": {

"name": "pong",

"status": "critical",

"check": {

"service_id": "pong",

"interval": "10s",

"script": "/usr/bin/netstat -an | grep 8300 | grep LISTEN"

},

"port": 8300,

"tags": ["pong"]

}

}

This will check the Pong app running on Port 8300

To start Consul, add the same start up scripts as for Consul Server

start Consul

Client should register itself and its Pong service with the cluster

Try stopping Pong and watch for Consul to show the service as Orange (failing)

Fabio config

To rout users to any of the 3 operating Pong instances, you will need Fabio to read the health status of each instance (from Consul)

on each Consul sever, create Fabio user + group

useradd -M -d /opt/fabio -s /sbin/nologin fabio

mkdir -p /opt/fabio/bin

get the Binary

wget https://github.com/fabiolb/fabio/releases/download/v1.5.9/fabio-1.5.9-go1.10.2-linux_amd64 -O /opt/fabio/bin/fabio && chmod +x /opt/fabio/bin/fabio

On each Consul server, Add Fabio properties file, update ui.addr and regitstry.consul.addr, by default Fabio will listen on port 9999

Fabio Properties

vim /opt/fabio/fabio.properties

# These two lines are example of running fabio with HTTPS certificates

#proxy.cs = cs=lb;type=file;cert=/opt/fabio/certs.d/mydomain_com.ca-bundle.crt;key=/opt/fabio/certs.d/mydomain_com.key

#proxy.addr = :443;cs=lb;tlsmin=tls11;tlsmax=tls12;tlsciphers="0xc02f,0x9f,0xc030,0xc028,0xc014,0x6b,0x39,0x009d,0x0035",# :80

proxy.addr = :9999

proxy.header.tls = Strict-Transport-Security

proxy.header.tls.value = "max-age=63072000; includeSubDomains"

ui.addr = 10.185.20.180:9998

ui.access = ro

runtime.gogc = 800

log.access.target = stdout

log.access.format = - - [] "" ".Referer" ".User-Agent" "" "" "" ""

log.access.level = INFO

registry.consul.addr = 10.185.20.180:8500

proxy.maxconn = 20000

Create a Fabio startup script

Fabio startup

cat <<EOF > /etc/systemd/system/fabio.service

[Unit]

Description=Fabio Proxy

After=syslog.target

After=network.target

[Service]

LimitMEMLOCK=infinity

LimitNOFILE=65535

Type=simple

WorkingDirectory=/opt/fabio

Restart=always

ExecStart=/opt/fabio/bin/fabio -cfg fabio.properties

# Log to syslog with identifier for syslog to process

StandardOutput=syslog

StandardError=syslog

SyslogIdentifier=fabio

# No need that fabio messes with /dev

PrivateDevices=yes

# Dedicated /tmp

PrivateTmp=yes

# Make /usr, /boot, /etc read only

ProtectSystem=full

# /home is not accessible at all

ProtectHome=yes

# You will have to run “setcap ‘cap_net_bind_service=+ep’ /opt/fabio/bin/fabio”

# to be able to bind ports under 1024. This directive allows it to happen:

AmbientCapabilities=CAP_NET_BIND_SERVICE

# Only ipv4, ipv6, unix socket and netlink networking is possible

# Netlink is necessary so that fabio can list available IPs on startup

RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK

# Unprivileged user

User=fabio

Group=fabio

[Install]

WantedBy=multi-user.target

EOF

set permissions

chown -R fabio:fabio /opt/fabio

systemctl enable fabio.service

systemctl daemon-reload

systemctl start fabio.service

check Fabio service

journalctl -u fabio --no-pager -n100

You should now see Fabio service listed on Consul

Fabio should be listening on port 9998, open up browser to <IP of Consul Server>:9998

But there should be no routes yet

Fabio Routes

We will create 1 Route for our Apache thats running on each Pong server.

Fabio will route all requests coming to Fabio hostname, port 9999, it will route to <IP of Pong server>:80

We now have a total of 2 Health Checks, 1. Pong 2. Apache

Consul monitors both, but Fabio will create a route only for Apache,

Apache service check

lets add the Apache healthcheck, we will want to rout users to our Apache (which will use Reverse Proxy using certs to proxy users to Pong Web via HTTPS). Note: Fabio also supports TLS authentication but that is outside the scope of this article

add a new Apache health check, the Tag is using the "urlprefix-" to tell Fabio to create a Route, followed by a slash,

Fabio will route any requests coming to Fabio host (port 9999)

{

"service": {

"name": "apache-svc",

"status": "critical",

"check": {

"service_id": "apache-svc",

"interval": "10s",

"script": "/usr/bin/systemctl status httpd.service"

},

"port": 80,

"tags": ["urlprefix-/"]

}

}

restart Consul client on the Pong node (make sure the Pong healthcheck passes, otherwise Fabio wont show the route)

you should now see Fabio display a proper route

Check the available routes using API

curl -s http://10.185.20.180:9998/api/routes

[{"service":"apache-svc","host":"","path":"/","src":"/","dst":"http://10.185.20.173:80/","opts":"","weight":1,"cmd":"route add","rate1":0,"pct99":0}]

Test Routes

on the Consul/Fabio host, tail Fabio output,

journalctl -u fabio -f

Turn off either Apache or Pong service, watch the routing table get updated automatically

_{Fabio adds 2 routes, Pong and Apache}

_{I turned off Apache service, Fabio instantly removes Apache route (minus sign)}

In the browser, try going to <IP or Hostname of Consul/Fabio host>:9999

It should redirect you to <Pong host>:80, and from there, Apache will take over and reverse proxy you to Pong service running on its own different port