Swarm
Swarm is a native Docker solution for clustering which can cluster many docker hosts into a single large pool, so they look like single large engine. So, it can be managed at scale.
Alternative of Swarm are Kubernetes, Mesosphere, Apache Mesos etc.
Architecture:
It is client-server architecture where docker client sends commands(through tcp 2375) to docker demon.
Docker client could be Docker Engine(native client) or any 3rd party tools
Docker Demon(server) - Docker engine or 3rd party tools
Something that implements docker remote API can server as docker demon.
For a swarm cluster there are 2 things which are needed
1. Discovery Service
2. Swarm Manager
Discovery Service - We could use any of the key value stores like zookeeper, etcd or consel. The 'libkv' sits on top of the kv store and abstracts the differences. libkv is library build in go to abstract the underlying key value store.
If the backend kv store dies swarm data can be populated when a new kv store is spun up. But networking details cannot be populated.
Swarm Manager: Required to administer any program in the cluster. It supports HA. Only the primary sends all commands to the cluster. Secondaries automatically take over at the event of primary going down.
Filtering and Scheduling:
Filtering runs first and scheduling follows.
Filters: 3 types -
Affinity - Run new containers on the same Monde as existing containers or images i.e say run the containers where the images is already there
Constraint - 2 types Standard -- Data returned from docker info command. Custom - Labels assigned to engine demons.
Resource - Run new containers on nodes with particular resources free
Scheduling - 2 types - random, spread( deploys in least container host) and binpack ( deploys in most container host unit max out and move to next host)
To initialise Docker Swarm type docker init this will init the docker swarm mode and if you inspect docker info, you can find swarm in active mode.
PORTS TO BE OPENED BETWEEN MACHINES
4789 UDP - used by docker for overlay network
7946 - TCP and UDP - Used by Swarm
While creating overlay network docker connects to bridge network, so it can access external network it can be prevented by using --internal flags. You can encrypt your network by providing encrypted=true flag, but this will have overhead and reduce the performance of system.
You can use weavenet for overlay network and does not require etcd or consul. Works with Swarm IPAM and DNS discovery but can provide its own without swarm. Mesh works across hosts locally or in cloud or both. High performance peer-to-peer encryption. Allow full UDP multicast.
Ports required for weavenet - TCP 6783, UDP 6873 and 6784 to your docker host. Weave net can route through multiple hops in your cluster.
To initialise docker swarm
docker swarm init
To create service
docker service create --name web -p 8088:80 nginx
To list service
docker service ps web
To update service update
docker service update --replicas=2 web
To remove service
docker service rm web
To scale service
docker service scale web=2
To check the nodes in swarm
docker node ls
Swarm manager accepts your service definition as the desired state for your application.
Tor testing
To run customer api - there are many images are swarmgs (swarm getting started)
docker service --name customer-api --p 3000:3000 swarmgs/customer
curl localhost:3000/customer/1
TESTING
You can use apache 'ab' tool to benchmark http servers
https://httpd.apache.org/docs/2.4/programs/ab.html
Below example makes 1000 requests with 4 concurrent. You can add more containers of the customer-api and see the difference in the request per second value.
ab -n 1000 -c 4 http://127.0.0.1:3000/customer/1
Requests per second: 150.39 [#/sec] (mean)
Time per request: 26.598 [ms] (mean)
Time per request: 6.649 [ms] (mean, across all concurrent requests)
Transfer rate: 35.83 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.2 0 3
Processing: 12 26 12.8 24 152
Waiting: 12 26 12.2 24 139
Total: 12 26 12.8 24 152
To test linking linking between containers
docker run --rm -d -p 4000:3000 --name balance-api -e MYWEB_CUSTOMER_API=127.0.0.1:3000 swarmgs/nodewebstress
VISUALISE
To visualise docker swarm cluster use manomarks/visualizer image from docker hub
docker service create \ --name=viz \ --publish=8080:8080/tcp \ --constraint=node.role==manager \ --mount=type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \ dockersamples/visualizer
DEPLOY DOCKER STACK
docker stack deploy -c ~/Projects/docker/myapp.yml myapp
MANUAL SCHEDULING
# Schedule containers on a specific node environment: - "constraint:node==node-1" # Schedule containers on a node that has the 'storage' label set to 'ssd' environment: - "constraint:storage==ssd" # Schedule containers where the 'redis' image is already pulled environment: - "affinity:image==redis"
COMMAND COMMANDS WHEN USING STACK
To Deploy Stack
docker stack deploy -c app.yml <app name>
Here c denotes that json is passed
To List services
docker stack services <servicename>
To list stack process
Docker stack ps is equivalent to docker service ps
docker stack ps <appname>
Update Service
To update service modify the yaml file and use the same deploy command to update
docker stack deploy -c app.yml <app name>
Remove Stack
docker stack rm <appname>
Tip: docker run is similar to docker service create. docker-compose is equivalent to docker stack deploy
To pass the image name to get the container id - Just a short cut
docker exec -it $(docker ps -ef name=imagename -q) bash
Monitoring Docker Containers
To check the statistics of container and with 'LogEntries' you can collect stats
docker stats
Docker Remote API :/container/{container-name|cid}/stats
Docker Universal Control Plane
cAdvisor - Data is a available only for 60 seconds, so we need to backend with Prometheus or InfluxDB and frontend with ELK dashboard. ie. cAdvisor + Prometheus/Influx + Kibana
NewRelic
ELK can be used for logging
LOGGING
Services send logs to standard output
Logspout fetches the logs and send to Logstash
Logstash parse the logs ans end them to central db
Visaualize it with dashboard
Elastic Search - is an open source RESTfule distributed search and analytics engine build on Apache Lucene. It can be used to provide near-real time analytics using large volumes of log data.
LogStash - open source server side data processing pipeline that ingests data from multiple sources, transforms it and sends it to stash.
KIbana - open source data visualisation and exploration tool. Kibana offers tight integration with elasticsearch which makes it the default choice for visualising data stored in Elasticsearch.
LOGSpout - logspout is a log router for Docker container that runs inside docker. It attaches to all containers on a host, then routes their logs wherever you want. Logspout will be deployed globally and it collects logs from all the containers the routes accordingly.
Create LogStash Config
echo '
input {
syslog { port => 51415 }
}
output {
elasticsearch {
hosts => ["http://elasticsearch:9200"]
}
}
' | docker config create logstash.conf -
Monitoring and Metrics
https://landing.google.com/sre/book/chapters/monitoring-distributed-systems.html
cAdvisor from google can be used for collecting metrics, but UI is not very helpful. But the metrics collected by cAdvisor will be helpful for prometheus. (metrics can be accessed by http:<ip>:<port>/metrics)
Prometheus - opens source system for monitoring and alerting build by SoundCloud.
Node Exporter 0 is an official exporter written by prometheus team an it is capable of collecting metrics from Hardware and OS metrics and exposing to prometheus.
https://github.com/prometheus/node_exporter. There are lots of exporter for different systems.
Prometheus sends alerts to an alert manager. The alert manager then manages those alerts by notification by email, pager duty, Slack, hip chat.
Grafana - open source metric analytics and visualisation suite for time series date visualisation. Can be used with elastic search and prometheus.
HEALTH CHECK
To update a service i.e to restart . This will restart the container
docker service update --force <service-name>
Inside the yaml files(of compose )
add a healthcheck section
health check:
test: curl -f -s -S http://localhost/testurl || exit 1
interval: 1m5s //default is 30s
timeout: 10s //default 30s
retries: 3 //default 3
Docker check the url, and it if the url returns and non-zero, it exits with 1. This is because docker will execptect 0, 1 or 2 as the values to validate the responses, so value 1 is returned for failures.
You can embed the health check as part of the image
HEALTHCHECK --interval=5s CMD /opt/knowesis/healthcheck || exit 1
CONSTRAINTS