[TBD]Understanding Prometheus

Introduction

Laymen explanation

Technical explanation

Prometheus is an open-source monitoring system based on time-series database with embedded notification and visualization mechanisms.

Prometheus is based on the pooling approach. That means you have to install a set of so-called "exporters" on your hosts and register them in the main server configuration. Thanks god, that is not a problem with Ansible. Also, you have to install a separate service for sending notifications ("alertmanager") and a separate service for data visualization (plain old "grafana" which is already integrated with Prometheus).

Exporters

Common place for listing of exporters is https://prometheus.io/docs/instrumenting/exporters/

Using haproxy

HAProxy offers following metrics

• • Frontend metrics such as client connections and requests. Frontend metrics provide information about the client’s interaction with the load balancer itself(for example, throughput, session rate).

• • Backend metrics such as availability and health of backend servers. It measures the communication between HAProxy and your backend servers that handle client requests(for example, backend throughput, errors count).

• • Health metrics that reflect the state of your HAProxy setup. It exposes information about the health of each front and backend server, in addition to the metrics listed above.Monitoring the health of your HAProxy servers gives you the information you need to quickly respond to outages as they occur.

Below HAProxy config provide example config to enable monitoring

HAProxy sample configuration for enabling monitoring(filename:config)

defaults

mode http

timeout server 5s

timeout connect 5s

timeout client 5s

frontend frontend

bind *:1234

use_backend backend

backend backend

# Use the node_exporter (if any).

server node_exporter 127.0.0.1:9100

frontend monitoring

bind *:1235

no log

stats uri /

stats enable

For above config, below is the example output of monitoring metrics

Example metric

root@ubuntu:~/personal/reseach/prometheus/haproxy_exporter/haproxy-1.6.2# ./haproxy -f config &

[1] 1513

root@ubuntu:~/personal/reseach/prometheus/haproxy_exporter/haproxy-1.6.2# curl -k 'http://localhost:1235/;csv'

# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,

frontend,FRONTEND,,,0,0,2000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,1,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,0,0,0,0,,,,,,,,

backend,node_exporter,0,0,0,0,,0,0,0,,0,,0,0,0,0,no check,1,1,0,,,,,,1,2,1,,0,,2,0,,0,,,,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,

backend,BACKEND,0,0,0,0,200,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,0,4,0,,1,2,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,0,0,0,0,-1,,,0,0,0,0,

monitoring,FRONTEND,,,1,1,2000,1,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,3,0,,,,0,1,0,1,,,,0,0,0,0,0,0,,1,1,1,,,0,0,0,0,,,,,,,,

For learning first time, I used following as it is easy to setup and understand

For first time trial

Use steps mentioned below line by line

https://www.robustperception.io/haproxy-dashboards-with-prometheus/?lipi=urn%3Ali%3Apage%3Ad_flagship3_pulse_read%3BRVGw5vLzT6SVFvG8bUmhFw%3D%3D

Later on try with https://github.com/prometheus/haproxy_exporter which is official link. Note that there is a gap in the installation steps mentioned in this doc. You need to perform 'go get' operation.

Using https://github.com/prometheus/haproxy_exporter

$git clone https://github.com/prometheus/haproxy_exporter.git

$go get github.com/prometheus/haproxy_exporter

$ ./haproxy_exporter

INFO[0000] Starting haproxy_exporter (version=0.8.0, branch=master, revision=2894f78b2ac6b3bb270dbe6920367ac6309aff9e) source=haproxy_exporter.go:494

INFO[0000] Build context (go=go1.6.2, user=root@ubuntu, date=20171218-08:07:42) source=haproxy_exporter.go:495

INFO[0000] Listening on :9101 source=haproxy_exporter.go:520

====

Thereafter access metrics via browser (http://<nachine-ip>:9101)

Prometheus exporter as a container + HAProxy as container

Prometheus exporter as a container + HAProxy as container

root@ubuntu:~/docker-configs/haproxy/deepak# ifconfig ens2f0

ens2f0 Link encap:Ethernet HWaddr 0c:c4:7a:e5:64:a4

inet addr:10.106.73.191 Bcast:10.106.73.255 Mask:255.255.255.128

inet6 addr: fe80::ec4:7aff:fee5:64a4/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:871631 errors:0 dropped:0 overruns:0 frame:0

TX packets:149969 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:1144588823 (1.1 GB) TX bytes:21528212 (21.5 MB)

Memory:fbb00000-fbbfffff

root@ubuntu:~/docker-configs/haproxy/deepak# cat config

defaults

mode http

timeout server 5s

timeout connect 5s

timeout client 5s

frontend frontend

bind *:1234

use_backend backend

backend backend

# Use the node_exporter (if any).

server node_exporter 127.0.0.1:9100

frontend monitoring

bind *:1235

no log

stats uri /

stats enable

root@ubuntu:~/docker-configs/haproxy/deepak# docker run -dt -p 1235:1235 --net=host -v /root/docker-configs/haproxy/deepak/config:/usr/local/etc/haproxy/haproxy.cfg haproxy:latest

22f5a3bdfa6ba8df78ca99f4fd65382edb12053de8e7cce79af3f1bde1d12341

root@ubuntu:~/docker-configs/haproxy/deepak# netstat -atpun | grep 1235

tcp 0 0 0.0.0.0:1235 0.0.0.0:* LISTEN 57033/haproxy

root@ubuntu:~/docker-configs/haproxy/deepak# curl -k 'http://localhost:1235/;csv'

# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,agent_duration,check_desc,agent_desc,check_rise,check_fall,check_health,agent_rise,agent_fall,agent_health,addr,cookie,mode,algo,conn_rate,conn_rate_max,conn_tot,intercepted,dcon,dses,

frontend,FRONTEND,,,0,0,2000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,1,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,http,,0,0,0,0,0,0,

backend,node_exporter,0,0,0,0,,0,0,0,,0,,0,0,0,0,no check,1,1,0,,,7,,,1,2,1,,0,,2,0,,0,,,,0,0,0,0,0,0,,,,,0,0,,,,,-1,,,0,0,0,0,,,,,,,,,,,,,,http,,,,,,,,

backend,BACKEND,0,0,0,0,200,0,0,0,0,0,,0,0,0,0,UP,1,1,0,,0,7,0,,1,2,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,0,0,0,0,0,0,0,-1,,,0,0,0,0,,,,,,,,,,,,,,http,,,,,,,,

monitoring,FRONTEND,,,1,1,2000,1,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,3,0,,,,0,1,0,1,,,,0,0,0,0,0,0,,1,1,1,,,0,0,0,0,,,,,,,,,,,,,,,,,,,,,http,,1,1,1,1,0,0,

root@ubuntu:~/docker-configs/haproxy/deepak# netstat -atpun | grep 9101

root@ubuntu:~/docker-configs/haproxy/deepak# docker run -dt -p 9101:9101 prom/haproxy-exporter --haproxy.scrape-uri="http://10.106.73.191:1235/;csv"

2683b270f5ca21b5bc39181df0d00936cfff4a502ffc8d23de71ee7daa655741

root@ubuntu:~/docker-configs/haproxy/deepak# netstat -atpun | grep 9101

tcp6 0 0 :::9101 :::* LISTEN 57105/docker-proxy

root@ubuntu:~/docker-configs/haproxy/deepak# curl -k http://localhost:9101/metrics

# HELP go_gc_duration_seconds A summary of the GC invocation durations.

# TYPE go_gc_duration_seconds summary

go_gc_duration_seconds{quantile="0"} 0

go_gc_duration_seconds{quantile="0.25"} 0

go_gc_duration_seconds{quantile="0.5"} 0

go_gc_duration_seconds{quantile="0.75"} 0

go_gc_duration_seconds{quantile="1"} 0

go_gc_duration_seconds_sum 0

go_gc_duration_seconds_count 0

# HELP go_goroutines Number of goroutines that currently exist.

# TYPE go_goroutines gauge

go_goroutines 9

....

.....

Collecting node data using node_exporter

Most of all, we want to gather linux-specific metrics such as cpu, rss memory and free disk space. For that, you can use node-exporter:

Use steps mentioned below line by line

https://github.com/prometheus/node_exporter

Python help

Useful modules

Useful modules

from prometheus_client import start_http_server

from prometheus_client.core import GaugeMetricFamily, CounterMetricFamily, REGISTRY

Setting up Prometheus server

Start Prometheus server on default port 9090

Setting up prometheus server

# wget https://github.com/prometheus/prometheus/releases/download/v2.2.1/prometheus-2.2.1.linux-amd64.tar.gz

# tar -zxvf prometheus-2.2.1.linux-amd64.tar.gz

# cd prometheus-2.2.1.linux-amd64/

# cat prometheus.yml

global:

scrape_interval: 15s # By default, scrape targets every 15 seconds.

# Attach these labels to any time series or alerts when communicating with

# external systems (federation, remote storage, Alertmanager).

external_labels:

monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:

# Here it's Prometheus itself.

scrape_configs:

# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

- job_name: 'prometheus'

# Override the global default and scrape targets from this job every 5 seconds.

scrape_interval: 5s

static_configs:

- targets: ['10.97.5.192:8080']

labels:

group: 'ingresscpx'

# kubectl describe svc exporter

Name: exporter

Namespace: default

Labels: app=exporter

Annotations: <none>

Selector: app=exporter

Type: ClusterIP

IP: 10.97.5.192

Port: http 8080/TCP

TargetPort: 8080/TCP

Endpoints: 10.244.1.97:8080

Session Affinity: None

Events: <none>

root@master:/var/triton/prometheus/promotheus_server# prometheus-2.2.1.linux-amd64/prometheus --config.file=prometheus.yml

level=info ts=2018-03-16T22:56:26.209697817Z caller=main.go:220 msg="Starting Prometheus" version="(version=2.2.1, branch=HEAD, revision=bc6058c81272a8d938c05e75607371284236aadc)"

level=info ts=2018-03-16T22:56:26.20977861Z caller=main.go:221 build_context="(go=go1.10, user=root@149e5b3f0829, date=20180314-14:15:45)"

level=info ts=2018-03-16T22:56:26.209815006Z caller=main.go:222 host_details="(Linux 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018 x86_64 master (none))"

level=info ts=2018-03-16T22:56:26.209852609Z caller=main.go:223 fd_limits="(soft=1024, hard=1048576)"

level=info ts=2018-03-16T22:56:26.213259093Z caller=web.go:382 component=web msg="Start listening for connections" address=0.0.0.0:9090

level=info ts=2018-03-16T22:56:26.21320263Z caller=main.go:504 msg="Starting TSDB ..."

level=info ts=2018-03-16T22:56:26.252444514Z caller=main.go:514 msg="TSDB started"

level=info ts=2018-03-16T22:56:26.252523541Z caller=main.go:588 msg="Loading configuration file" filename=prometheus.yml

level=info ts=2018-03-16T22:56:26.253085157Z caller=main.go:491 msg="Server is ready to receive web requests."

Useful link: https://prometheus.io/download/

https://prometheus.io/docs/prometheus/latest/getting_started/

Start Prometheus server on custom port

Start prometheus at custom listen port 9091

root@master:/var/triton/prometheus/promotheus_server# prometheus-2.2.1.linux-amd64/prometheus --config.file=prometheus.yml --web.listen-address=0.0.0.0:9091

level=info ts=2018-03-20T04:58:51.842197378Z caller=main.go:220 msg="Starting Prometheus" version="(version=2.2.1, branch=HEAD, revision=bc6058c81272a8d938c05e75607371284236aadc)"

level=info ts=2018-03-20T04:58:51.842275216Z caller=main.go:221 build_context="(go=go1.10, user=root@149e5b3f0829, date=20180314-14:15:45)"

level=info ts=2018-03-20T04:58:51.842321412Z caller=main.go:222 host_details="(Linux 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018 x86_64 master (none))"

level=info ts=2018-03-20T04:58:51.842348997Z caller=main.go:223 fd_limits="(soft=1024, hard=1048576)"

level=info ts=2018-03-20T04:58:51.847293212Z caller=main.go:504 msg="Starting TSDB ..."

level=info ts=2018-03-20T04:58:51.84801314Z caller=web.go:382 component=web msg="Start listening for connections" address=0.0.0.0:9091

level=info ts=2018-03-20T04:58:51.889216483Z caller=main.go:514 msg="TSDB started"

Useful link: https://groups.google.com/forum/#!topic/prometheus-users/vX5XAgfGjp4

https://stackoverflow.com/questions/28303978/changing-grafana-port

Integration of Prometheus with Grafana

Start Grafana container and then configure prometheus server via GUI

Start Grafana as docker container

root@minion1:/var/triton/prometheus# docker run -d --name=grafana -p 3020:3000 grafana/grafana

b80be501fbc791e4e816520847a5f3d5deb0575f7d3b33b0de05c0bc58a9ec7f

root@minion1:/var/triton/prometheus# docker ps | grep grafana

b80be501fbc7 grafana/grafana "/run.sh" 13 seconds ago Up 13 seconds 0.0.0.0:3020->3000/tcp grafana

Useful link:https://prometheus.io/docs/visualization/grafana/

Below example depicts configuration of prometheus via Grafana web-UI.

Access Prometheus server via browser

Access via http://<host machine IP>:9090

Kubernetes specific

Exporter running as kube service

Example yaml to run CPX exporter as service

apiVersion: v1

kind: Service

metadata:

name: exporter

labels:

app: exporter

spec:

type: ClusterIP

ports:

- port: 8080

protocol: TCP

name: http

selector:

app: exporter

---

apiVersion: v1

kind: ReplicationController

metadata:

name: exporter

spec:

replicas: 1

template:

metadata:

labels:

app: exporter

spec:

containers:

- name: exporter

image: exporter:1.0

ports:

- containerPort: 8080

args: ["--target-nsip", "10.102.53.236:5080", "--port", "8080"]

Troubleshoot

How to handle haproxy-exporter error 'Can't scrape HAProxy'

handling error 'Can't scrape HAProxy'

root@ubuntu:~/personal/reseach/prometheus/haproxy_exporter/haproxy_exporter# ./haproxy_exporter --haproxy.scrape-uri="http://localhost:1235/;csv"

INFO[0000] Starting haproxy_exporter (version=0.8.0, branch=master, revision=2894f78b2ac6b3bb270dbe6920367ac6309aff9e) source=haproxy_exporter.go:494

INFO[0000] Build context (go=go1.6.2, user=root@ubuntu, date=20171218-08:07:42) source=haproxy_exporter.go:495

INFO[0000] Listening on :9101 source=haproxy_exporter.go:520

ERRO[0012] Can't scrape HAProxy: Get http://localhost:1235/;csv: dial tcp [::1]:1235: getsockopt: connection refused source=haproxy_exporter.go:314

root@ubuntu:~/personal/reseach/prometheus/haproxy_exporter/haproxy_exporter# cd -

/root/personal/reseach/prometheus/haproxy_exporter

root@ubuntu:~/personal/reseach/prometheus/haproxy_exporter# cd haproxy-1.6.2

root@ubuntu:~/personal/reseach/prometheus/haproxy_exporter/haproxy-1.6.2# cat config

defaults

mode http

timeout server 5s

timeout connect 5s

timeout client 5s

frontend frontend

bind *:1234

use_backend backend

backend backend

# Use the node_exporter (if any).

server node_exporter 127.0.0.1:9100

frontend monitoring

bind *:1235

no log

stats uri /

stats enable

root@ubuntu:~/personal/reseach/prometheus/haproxy_exporter/haproxy-1.6.2# ./haproxy -f config &

[1] 26273

root@ubuntu:~/personal/reseach/prometheus/haproxy_exporter/haproxy-1.6.2# netstat -atpun | grep haproxy

tcp 0 0 0.0.0.0:1234 0.0.0.0:* LISTEN 26273/haproxy

tcp 0 0 0.0.0.0:1235 0.0.0.0:* LISTEN 26273/haproxy

root@ubuntu:~/personal/reseach/prometheus/haproxy_exporter/haproxy-1.6.2# cd ../haproxy_exporter/

root@ubuntu:~/personal/reseach/prometheus/haproxy_exporter/haproxy_exporter# ./haproxy_exporter --haproxy.scrape-uri="http://localhost:1235/;csv"

INFO[0000] Starting haproxy_exporter (version=0.8.0, branch=master, revision=2894f78b2ac6b3bb270dbe6920367ac6309aff9e) source=haproxy_exporter.go:494

INFO[0000] Build context (go=go1.6.2, user=root@ubuntu, date=20171218-08:07:42) source=haproxy_exporter.go:495

INFO[0000] Listening on :9101 source=haproxy_exporter.go:520

====

Above steps shows that error didn't happen when browsed the metrics(http://<machine-ip>:9101/metrics)

Prometheus recommendations

• When working with applications, you should aim for an exporter that requires no custom configuration by the user beyond telling it where the application is.

• You may also need to offer the ability to filter out certain metrics if they may be too granular and expensive on large setups, for example the HAProxy exporter allows filtering of per-server stats.

• Ensuring the exporter works out of the box without configuration, and providing a selection of example configurations for transformation if required, is advised.

• YAML is the standard Prometheus configuration format, all configuration should use YAML by default.

Refer: https://prometheus.io/docs/instrumenting/writing_exporters/

Pull design reasons

https://prometheus.io/docs/introduction/faq/#why-do-you-pull-rather-than-push

https://prometheus.io/blog/2016/07/23/pull-does-not-scale-or-does-it/#:~:text=Prometheus%20was%20inspired%20by%20Google's,due%20its%20pull%20approach%20either.

Reference

http://trustmeiamadeveloper.com/2016/07/03/bringing-the-light-of-monitoring-with-prometheus/

https://www.datadoghq.com/blog/monitoring-haproxy-performance-metrics/#health-metrics