A Kubernetes approach to configure iptables route to host from within POD

Introduction

Laymen explanation

If your any work needs to change the default behaviour of data packet processing, you may be easily do this in a Linux environment. For example, if you want network traffic to be redirect to your app, you can create iptables rule. This is possible in docker container environment as well. This document provides a way to do the same in Kubernetes environment.

Technical explanation

Docker host mode provides access to the host networking. Alongwith this, its provide --cap-add NET_ADMIN option so that container can modify the host networking.

Kubernetes provides similar capability to do this.

Kubernetes hostNetwork option provides capability to access host networking. For example, flannel CNI uses this as Daemonset.

Steps to configure using test ubuntu container

Deploy testnetadmin.yaml

Note that this YAML has defined NET_ADMIN capability. This capability allows to execute iptables command

Host mode is needed to access host network from within the container

Test YAML to Create Daemonset with ability to edit host machine network

root@ubuntu-232:~/deepak/netadmin# cat testnetadmin.yaml

apiVersion: apps/v1

kind: DaemonSet

metadata:

name: testnetadmin

spec:

selector:

matchLabels:

name: testnetadmin

template:

metadata:

labels:

name: testnetadmin

spec:

hostNetwork: true

containers:

- name: testnetadmin

image: "ubuntu:14.04"

command:

- /bin/sleep

- "10000"

securityContext:

capabilities:

add:

- NET_ADMIN

Find the node on which the test POD is running

Login to the node on which POD is running

root@ubuntu-232:~/deepak/netadmin# kubectl get pods -o wide | grep netadmin

testnetadmin-lfjh2 1/1 Running 0 1m 10.106.73.231 ubuntu-231

root@ubuntu:~# ssh root@10.106.73.231

root@10.106.73.231's password:

Permission denied, please try again.

root@10.106.73.231's password:

root@ubuntu:~# ssh root@10.106.73.231

root@10.106.73.231's password:

Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.4.0-133-generic x86_64)

* Documentation: https://help.ubuntu.com

* Management: https://landscape.canonical.com

* Support: https://ubuntu.com/advantage

75 packages can be updated.

8 updates are security updates.

Last login: Fri Sep 6 10:43:04 2019 from 10.252.249.2

root@ubuntu-231:~#

Create a new host iptables chain via POD and verify

We will create a new chain TESTIPTABLE and see if it is reflected in host or not

Login to the host machine and see that TESTIPTABLE chain doesn't exist in the host machine

Verify that hostname doesn't have any preexisting chain TESTIPTABLE

root@ubuntu:~# ssh root@10.106.73.231

root@10.106.73.231's password:

Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.4.0-133-generic x86_64)

* Documentation: https://help.ubuntu.com

* Management: https://landscape.canonical.com

* Support: https://ubuntu.com/advantage

75 packages can be updated.

8 updates are security updates.

New release '18.04.3 LTS' available.

Run 'do-release-upgrade' to upgrade to it.

Last login: Fri Nov 15 03:43:20 2019 from 10.106.102.141

root@ubuntu-231:~# iptables -t nat --list-rules | grep TESTCHAIN

root@ubuntu-231:~# logout

Connection to 10.106.73.231 closed.

Login to the POD for creation of the host rules

Create host rule using POD

root@ubuntu-232:~/deepak/netadmin# kubectl exec -it testnetadmin-lfjh2 bash

root@ubuntu-231:/# iptables -t nat --list-rules | grep TESTIPTABLE

root@ubuntu-231:/# iptables -t nat -N TESTIPTABLE

root@ubuntu-231:/# iptables -t nat --list-rules | grep TESTIPTABLE

-N TESTIPTABLE

root@ubuntu-231:/# iptables -t nat -A TESTIPTABLE -p tcp -m tcp --dport 8888 -j DNAT --to-destination 192.168.1.2:8888

root@ubuntu-231:/# iptables -t nat --list-rules | grep TESTIPTABLE

-N TESTIPTABLE

-A TESTIPTABLE -p tcp -m tcp --dport 8888 -j DNAT --to-destination 192.168.1.2:8888

Now login to the host machine and again check for the chain TESTIPTABLE

Check for TESTIPTABLE rule in the host machine

root@ubuntu:~# ssh root@10.106.73.231

root@10.106.73.231's password:

Welcome to Ubuntu 16.04.4 LTS (GNU/Linux 4.4.0-133-generic x86_64)

* Documentation: https://help.ubuntu.com

* Management: https://landscape.canonical.com

* Support: https://ubuntu.com/advantage

75 packages can be updated.

8 updates are security updates.

New release '18.04.3 LTS' available.

Run 'do-release-upgrade' to upgrade to it.

Last login: Fri Nov 15 03:52:58 2019 from 10.106.102.141

root@ubuntu-231:~# iptables -t nat --list-rules | grep TESTIPTABLE

-N TESTIPTABLE

-A TESTIPTABLE -p tcp -m tcp --dport 8888 -j DNAT --to-destination 192.168.1.2:8888

Cleanup the setup

Setup cleanup steps

root@ubuntu-231:/# iptables -t nat -F TESTIPTABLE

root@ubuntu-231:/# iptables -t nat --list-rules | grep TESTIPTABLE

-N TESTIPTABLE

root@ubuntu-231:/# iptables -t nat -X TESTIPTABLE

Another example via POD yaml

POD yaml for doing the same

root@ubuntu-232:~/deepak/netadmin# cat netadmin.yaml_pod

apiVersion: v1

kind: Pod

metadata:

name: mypod

spec:

hostNetwork: true

containers:

- name: myshell

image: "ubuntu:14.04"

command:

- /bin/sleep

- "10000"

securityContext:

capabilities:

add:

- NET_ADMIN

Reference

https://www.digitalocean.com/community/tutorials/how-to-list-and-delete-iptables-firewall-rules

https://stackoverflow.com/questions/52161937/create-daemonset-using-kubectl

https://www.weave.works/blog/container-capabilities-kubernetes/

http://alesnosek.com/blog/2017/02/14/accessing-kubernetes-pods-from-outside-of-the-cluster/