Cloud Computing‎ > ‎Load Balancing‎ > ‎HAProxy‎ > ‎

Dynamic config with LUA








Setup VM

========

Login with root user
$ ssh root@<ip>


Add user 'cloud'
$ sudo bash
$ adduser cloud
$ passwd cloud
Password : cloud

Add user cloud to sudo list
$ sudo echo "cloud ALL=(ALL) NOPASSWD:ALL" | sudo tee -a /etc/sudoers
exit

Login with cloud user
$ ssh cloud@<ip>

Stop firewall for Centos-7
$ sudo systemctl stop firewalld
$ sudo systemctl disable firewalld
$ sudo chkconfig firewalld off
$ sudo service status firewalld

Disable SElinux 
$ sudo setenforce 0 
$ getenforce


Compiling HAProxy with Lua

==============================

Install LUA 5.3.1


HAProxy 1.6\1.7 requires Lua 5.3. Lua 5.3 offers some features which make easy the
integration. Lua 5.3 is young, and some distros do not distribute it. Luckily,
Lua is a great product because it does not require exotic dependencies, and its
build process is really easy.

Install Dependencies: http://jackatlinux.blogspot.com/2013/11/installing-lua-and-luarocks-in-centos.html
 - Install gcc
   $ sudo yum install -y gcc gcc-c++ kernel-devel 

The compilation process for linux is easy:

 - download the source tarball
   wget http://www.lua.org/ftp/lua-5.3.1.tar.gz

 - untar it
   tar xf lua-5.3.1.tar.gz

 - enter the directory
   cd lua-5.3.1

 - build the library for linux
   http://www.lua.org/manual/5.3/readme.html
   If you're running Linux and get compilation errors, make sure you have installed the readline development package (which is probably named libreadline-dev or readline-devel).

   sudo yum install -y readline-devel
   make linux

 - install it:
   sudo make INSTALL_TOP=/usr install [This is to overwrite system default lua 5.1.0 to 5.3.0]
   sudo make INSTALL_TOP=/opt/lua-5.3.1 install [This is used by HAProxy from /opt location]


LUA-cjson

=========

The LUA-cjson is a library required for any json parsing in LUA code.

$ wget http://www.kyne.com.au/~mark/software/download/lua-cjson-2.1.0.tar.gz
$ tar -xzvf lua-cjson-2.1.0.tar.gz

$ cd lua-cjson-2.1.0
vi Makefile
LUA_VERSION =       5.3
PREFIX =             /usr

$ vi lua-cjson-2.1.0-1.rockspec
dependencies = {
    "lua >= 5.3"
}

$ vi lua-cjson.spec

%define luaver 5.3

#sudo make install
Console logs -
$ cp cjson.so //usr/lib/lua/5.3
$ chmod 755 //usr/lib/lua/5.3/cjson.so

$ sudo cp /usr/lib/lua/5.3/cjson.so /opt/lua-5.3.1/lib/cjson.so
$ sudo cp /usr/lib/lua/5.3/cjson.so /opt/lua-5.3.1/lib/lua/5.3/cjson.so



Install HAProxy 1.7\1.6 with LUA

============================

HAProxy builds with your favourite options, plus the following options for
embedding the Lua script language:

 - download the source tarball
   wget http://www.haproxy.org/download/1.7/src/snapshot/haproxy-ss-20160518.tar.gz

 - untar it
   tar -xzvf haproxy-ss-20160518

 - enter the directory
   cd haproxy-ss-20160518

 - build HAProxy:
   To fix openssl errors
   sudo yum install -y openssl-devel
   
   make TARGET=linux \
        USE_DL=1 \
        USE_LUA=1 \
USE_OPENSSL=1 \
        LUA_LIB=/opt/lua-5.3.1/lib \
        LUA_INC=/opt/lua-5.3.1/include

   [ backup - if wanna install in /usr]
   make TARGET=linux \
        USE_DL=1 \
        USE_LUA=1 \
USE_OPENSSL=1 \
        LUA_LIB=/usr/lib/lua/5.3/ \
        LUA_INC=/usr/include

   
   

   
 - install it:
   sudo make PREFIX=/opt/haproxy-1.7 install

   [ backup - if wanna install in /usr]
   sudo make PREFIX=/usr install
   

Configure HAProxy

================

 - copy binary file to sbin
   sudo cp /opt/haproxy-1.7/sbin/haproxy /usr/sbin/

 - copy init file
   sudo cp /home/cloud/haproxy/haproxy-1.7-dev3/examples/haproxy.init /etc/init.d/haproxy
   sudo chmod 755 /etc/init.d/haproxy

 - Create these directories and the statistics file for HAProxy to record in.

   sudo mkdir -p /etc/haproxy
   sudo mkdir -p /run/haproxy
   sudo mkdir -p /var/lib/haproxy
   sudo touch /var/lib/haproxy/stats

 - Then add a new user for HAProxy.
   sudo useradd -r haproxy

 - After the installation you can double check the installed version number with the following
    $ sudo haproxy -v
      HA-Proxy version 1.7-dev3-27b639d 2016/05/17
      Copyright 2000-2016 Willy Tarreau <willy@haproxy.org>

 - Configure the load balancer
   sudo vi /etc/haproxy/haproxy.cfg
   Add the config


HAProxy Script


$ cat /etc/haproxy/haproxy.cfg


#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    lua-load /usr/sbin/lua-choose-backend.lua
    


    # turn on stats unix socket
    stats socket /var/run/haproxy.sock mode 600 level admin
    stats timeout 2s
    log 127.0.0.1 local0

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  main
    bind *:7000
    acl url_static       path_beg       -i /static /images /javascript /stylesheets
    acl url_static       path_end       -i .jpg .gif .png .css .js

    use_backend static          if url_static
    default_backend             backend1


frontend http-in
    mode http

    log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ {%[ssl_c_verify],%{+Q}[ssl_c_s_dn],%{+Q}[ssl_c_i_dn]}\ %{+Q}r


    bind *:443 ssl crt /etc/haproxy/server.pem ca-file /etc/haproxy/ca.crt verify required

    use_backend %[lua.choose_backend]
    #default_backend backend1
    reqadd X-Forwarded-Proto:\ https if { ssl_fc }
    option forwardfor

# With HAProxy Enterprise HAPEE, this is for dynamic disk write for map updates
#dynamic-update
#       update id cdpload.map url https://10.128.41.10:80/cdpload.map delay 30s timeout 5s retries 3 map


frontend ssl-in
    mode tcp
    bind *:8443 ssl crt /etc/haproxy/server.pem ca-file /etc/haproxy/ca.crt verify required
    default_backend backendssh
    option tcplog
    tcp-request inspect-delay 5s
    tcp-request content accept if HTTP



listen stats
   bind *:8181
   stats enable
   stats uri /
   stats realm Haproxy\ Statistics
   stats auth cloud:cloud

#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
backend static
    balance     roundrobin
    server      static 127.0.0.1:4321 check

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend backend1
    balance     roundrobin
    server app1 1.0.0.1:80 check
    server app2 1.0.0.2:80 check


backend backend2
    balance     roundrobin
    server app3 1.0.0.3:80 check
    server app4 1.0.0.4:80 check


backend backendssh
    mode tcp
    balance     roundrobin
    server app1 1.0.0.1:80 check
    server app2 1.0.0.2:80 check
    server app3 1.0.0.3:80 check
    server app4 1.0.0.4:80 check
    timeout server 2h




LUA Script



/usr/sbin/lua-choose-backend.lua


-- Cloud map

cloudmetadata =  Map.new("/home/cloud/cloudmetadata.map", Map.str);

clouddynamicdata = Map.new("/home/cloud/clouddynamicdata.map", Map.str);


local function split( inSplitPattern, outResults )

   if not outResults then
      outResults = {}
   end
   local theStart = 1
   local theSplitStart, theSplitEnd = string.find( self, inSplitPattern, theStart )
   while theSplitStart do
      table.insert( outResults, string.sub( self, theStart, theSplitStart-1 ) )
      theStart = theSplitEnd + 1
      theSplitStart, theSplitEnd = string.find( self, inSplitPattern, theStart )
   end
   table.insert( outResults, string.sub( self, theStart ) )
   return outResults
end



local function find_backend(cn)

        

        --local json = require('cjson')

        core.log(core.info, cn)
        -- backend1


        --[[
        local clouddynamic = clouddynamicdata:lookup(cn);
        if(clouddynamic == nil) then
                core.log(core.info, "clouddynamic nil")
        else
                core.log(core.info, clouddynamic)
                return clouddynamic
        end
        ]]--

        local metadata = cloudmetadata:lookup(cn);

        if(metadata == nil) then
                core.log(core.info, "metadata nil")
        end

        core.log(core.info, metadata)
        -- 10

        
        core.set_map("/home/cloud/clouddynamicdata.map", cn, 14) 
-- this is in-memory write won't get flushed to disk
-- to flush to disk, run external service which would periodically poll /home/cloud/clouddynamicdata.map and write to disk map. 

        return metadata
end



function choose_backend(txn)

        local arg1 = txn.f:ssl_c_s_dn("CN")
        backend = find_backend (arg1)




        return backend
end

core.register_fetches("choose_backend", choose_backend)





Setup logging in HAproxy

==========================

Step 1: In Global Section of haproxy.cfg put the value log 127.0.0.1 local0 .Like given below

global
        log 127.0.0.1   local0
Step 2: Create new haproxy configuration file in /etc/rsyslog.d . Here we are keeping the log in localhost or in other words we should say HAproxy server

Note:
local0.=info -/var/log/haproxy.log defines the http log will be saved in haproxy.log
local0.notice -/var/log/haproxy-status.log defines the Server status like start,stop,restart,down,up etc. will be saved in haproxy-status.log
UDPServerRun 514 means opening UDP port no. 514 to listen haproxy messages
sudo vi /etc/rsyslog.d/haproxy.conf

$ModLoad imudp
$UDPServerRun 514 
$template Haproxy,"%msg%\n"
local0.=info -/var/log/haproxy.log;Haproxy
local0.notice -/var/log/haproxy-status.log;Haproxy
### keep logs in localhost ##
local0.* ~ 

Step 3: Now restart the HAproxy service

$ sudo service rsyslog restart
$ sudo service https restart
After restarting the haproxy service two logs will be created itself i.e haproxy.log and haproxy-status.log


Install netcat (nc)
$ sudo yum install -y nc


$ sudo tail -f /var/log/haproxy.log


Start HAProxy

==============

  - Copy SSL files
    /etc/haproxy/server.pem
    /etc/haproxy/ca.crt
    /usr/sbin/lua-choose-backend.lua

    /home/cloud/cloudmetadata.map

   $ cat cloudmetadata.map
   backend1 10
   backend2 2

  - Start
    sudo service haproxy start



Dynamic map

==============

1. Turn on stat socket
   $ sudo vi /etc/haproxy/haproxy.cfg
   $ stats socket /var/lib/haproxy.sock mode 600 level admin

2. Install socat
   $ sudo yum install socat

3. 
   $ echo "show map /home/cloud/cloudmetadata.map" | sudo socat /var/run/haproxy.sock stdio
   0x16cf160 backend1 10
   0x16cf300 backend2 2


   $ echo "set map /home/cloud/cloudmetadata.map backend2 14" | sudo socat /var/run/haproxy.sock stdio
   
   $ echo "show map /home/cloud/cloudmetadata.map" | sudo socat /var/run/haproxy.sock stdio
   0x8de070 backend1 10
   0x8de0a0 backend2 14


   $ cat /home/cloud/cloudmetadata.map
   backend1 10
   backend2 2   


   The in-memory state of /home/cloud/cloudmetadata.map is dynamically set by set map cmd or set_map LUA API
   The on-disk /home/cloud/cloudmetadata.map remains un-altered, means dynamic changes are not persistent and would vanish on HAProxy reload.
   
   To make the dynamic changes persistence, run a separate external service to also write the dynamic changes to disk map /home/cloud/cloudmetadata.map
   This would ensure changes to persist as well as keep the disk write load and latency limited to external service and not to HAProxy.






References









Comments