Reverse IP lookup solution for k8s services

Introduction

Laymen explanation

If you are using kubernetes for your product and wish to understand data traffic (IP packet), then you may need to know that this packet belongs to which services. You may want to use this info to draw service graph. You may need this info to understand communication between services (number of requests, response time). Note that IP packet will have source and destination IPs, not the associated service info. So, You need to devise way to get service info. This document helps in this regard.

Technical explanation

K8s DNS provides mapping between service and its associated IPs. But other way is not available in general. It is quoted that it is as per design (due to multiple level IP-clusterIP-SVC ) . Reference link is https://github.com/kubernetes/kubernetes/issues/33470

K8s offering

Headless service

For headless service, k8s offers reverse lookup (Ref: https://github.com/kubernetes/dns/pull/25)

Non-headless service

For non-headless service (service with cluster-ip), it doesn't provide. It recommends solution based on listening to endpoint/POD (Ref: https://github.com/kubernetes/kubernetes/issues/33470)

Enumeration of possible solutions

Ensure that head-less services are used in the deployment

In this approach, application will query k8s-DNS directly for getting service

Pros

k8s dns can be used for reverse dns
Istio uses headless services (need to double confirm)

Cons

Need to find separate solution for non-headless deployment

Separate App providing reverse lookup using k8s watch

In this approach, you will write your own reverse DNS lookup service.

Pros

Generic approach

Cons

Need to maintain separate app
Take care of endpoint IP reuse in case it is stored in DB
Take care of app failure

Handling of Database inconsistency

<IP, service name> can be inconsistent in case k8s decides to reuse this IP for service. If we don't handle this case, then service graph can go wrong.

In order to handle this case, a possible exploration point is the endpoint IP timestamp. The flow will be as below.

In DB, keep <IP , service name, Timestamp>. Get the startedAt field of the corresponding POD
In case timestamp changes, then App will add another entry in DB

Below flow suggests how to create <IP , service name, Timestamp> info

Design consideration

Endpoint IP timestamp calculation

POD based approach

In K8s, POD is assigned endpoint IP and so, if we can get the timestamp of IP assignment in POD, then we will be sure that any future assignment of IP address will always have higher timestamp. So, at any time, we can uniquely find the <IP, service mapping>

In K8s, the IP is assignment when the POD changes to running state. This state also have a timestamp which is used here for our purpose

Below is the example output for POD event

{u'status': {u'phase': u'Pending', u'qosClass': u'BestEffort'}, u'kind': u'Pod', u'spec': {u'dnsPolicy': u'ClusterFirst', u'securityContext': {}, u'serviceAccountName': u'default', u'schedulerName': u'default-scheduler', u'serviceAccount': u'default', u'terminationGracePeriodSeconds': 30, u'restartPolicy': u'Always', u'volumes': [{u'secret': {u'defaultMode': 420, u'secretName': u'default-token-g5svl'}, u'name': u'default-token-g5svl'}], u'tolerations': [{u'operator': u'Exists', u'tolerationSeconds': 300, u'effect': u'NoExecute', u'key': u'node.kubernetes.io/not-ready'}, {u'operator': u'Exists', u'tolerationSeconds': 300, u'effect': u'NoExecute', u'key': u'node.kubernetes.io/unreachable'}], u'containers': [{u'terminationMessagePath': u'/dev/termination-log', u'name': u'web-frontend', u'image': u'in-docker-reg.eng.citrite.net/cpx-dev/web-test:latest', u'volumeMounts': [{u'readOnly': True, u'mountPath': u'/var/run/secrets/kubernetes.io/serviceaccount', u'name': u'default-token-g5svl'}], u'terminationMessagePolicy': u'File', u'imagePullPolicy': u'Never', u'ports': [{u'protocol': u'TCP', u'containerPort': 80}], u'resources': {}}]}, u'apiVersion': u'v1', u'metadata': {u'name': u'web-frontend-jf6s6', u'labels': {u'app': u'web-frontend'}, u'namespace': u'default', u'ownerReferences': [{u'kind': u'ReplicationController', u'name': u'web-frontend', u'apiVersion': u'v1', u'controller': True, u'blockOwnerDeletion': True, u'uid': u'eb88921d-d76f-11e8-b4a7-eabc5652ca9b'}], u'resourceVersion': u'12280862', u'generateName': u'web-frontend-', u'creationTimestamp': u'2018-10-25T04:38:49Z', u'selfLink': u'/api/v1/namespaces/default/pods/web-frontend-jf6s6', u'uid': u'ddf9a74d-d80f-11e8-b4a7-eabc5652ca9b'}}

{u'status': {u'phase': u'Pending', u'conditions': [{u'status': u'True', u'lastProbeTime': None, u'type': u'PodScheduled', u'lastTransitionTime': u'2018-10-25T04:38:49Z'}], u'qosClass': u'BestEffort'}, u'kind': u'Pod', u'spec': {u'dnsPolicy': u'ClusterFirst', u'securityContext': {}, u'serviceAccountName': u'default', u'schedulerName': u'default-scheduler', u'serviceAccount': u'default', u'terminationGracePeriodSeconds': 30, u'restartPolicy': u'Always', u'volumes': [{u'secret': {u'defaultMode': 420, u'secretName': u'default-token-g5svl'}, u'name': u'default-token-g5svl'}], u'tolerations': [{u'operator': u'Exists', u'tolerationSeconds': 300, u'effect': u'NoExecute', u'key': u'node.kubernetes.io/not-ready'}, {u'operator': u'Exists', u'tolerationSeconds': 300, u'effect': u'NoExecute', u'key': u'node.kubernetes.io/unreachable'}], u'containers': [{u'terminationMessagePath': u'/dev/termination-log', u'name': u'web-frontend', u'image': u'in-docker-reg.eng.citrite.net/cpx-dev/web-test:latest', u'volumeMounts': [{u'readOnly': True, u'mountPath': u'/var/run/secrets/kubernetes.io/serviceaccount', u'name': u'default-token-g5svl'}], u'terminationMessagePolicy': u'File', u'imagePullPolicy': u'Never', u'ports': [{u'protocol': u'TCP', u'containerPort': 80}], u'resources': {}}], u'nodeName': u'ubuntu-231'}, u'apiVersion': u'v1', u'metadata': {u'name': u'web-frontend-jf6s6', u'labels': {u'app': u'web-frontend'}, u'namespace': u'default', u'ownerReferences': [{u'kind': u'ReplicationController', u'name': u'web-frontend', u'apiVersion': u'v1', u'controller': True, u'blockOwnerDeletion': True, u'uid': u'eb88921d-d76f-11e8-b4a7-eabc5652ca9b'}], u'resourceVersion': u'12280863', u'generateName': u'web-frontend-', u'creationTimestamp': u'2018-10-25T04:38:49Z', u'selfLink': u'/api/v1/namespaces/default/pods/web-frontend-jf6s6', u'uid': u'ddf9a74d-d80f-11e8-b4a7-eabc5652ca9b'}}

{u'status': {u'qosClass': u'BestEffort', u'containerStatuses': [{u'restartCount': 0, u'name': u'web-frontend', u'image': u'in-docker-reg.eng.citrite.net/cpx-dev/web-test:latest', u'imageID': u'', u'state': {u'waiting': {u'reason': u'ContainerCreating'}}, u'ready': False, u'lastState': {}}], u'startTime': u'2018-10-25T04:38:49Z', u'hostIP': u'10.106.73.231', u'phase': u'Pending', u'conditions': [{u'status': u'True', u'lastProbeTime': None, u'type': u'Initialized', u'lastTransitionTime': u'2018-10-25T04:38:49Z'}, {u'status': u'False', u'lastTransitionTime': u'2018-10-25T04:38:49Z', u'reason': u'ContainersNotReady', u'lastProbeTime': None, u'message': u'containers with unready status: [web-frontend]', u'type': u'Ready'}, {u'status': u'True', u'lastProbeTime': None, u'type': u'PodScheduled', u'lastTransitionTime': u'2018-10-25T04:38:49Z'}]}, u'kind': u'Pod', u'spec': {u'dnsPolicy': u'ClusterFirst', u'securityContext': {}, u'serviceAccountName': u'default', u'schedulerName': u'default-scheduler', u'serviceAccount': u'default', u'terminationGracePeriodSeconds': 30, u'restartPolicy': u'Always', u'volumes': [{u'secret': {u'defaultMode': 420, u'secretName': u'default-token-g5svl'}, u'name': u'default-token-g5svl'}], u'tolerations': [{u'operator': u'Exists', u'tolerationSeconds': 300, u'effect': u'NoExecute', u'key': u'node.kubernetes.io/not-ready'}, {u'operator': u'Exists', u'tolerationSeconds': 300, u'effect': u'NoExecute', u'key': u'node.kubernetes.io/unreachable'}], u'containers': [{u'terminationMessagePath': u'/dev/termination-log', u'name': u'web-frontend', u'image': u'in-docker-reg.eng.citrite.net/cpx-dev/web-test:latest', u'volumeMounts': [{u'readOnly': True, u'mountPath': u'/var/run/secrets/kubernetes.io/serviceaccount', u'name': u'default-token-g5svl'}], u'terminationMessagePolicy': u'File', u'imagePullPolicy': u'Never', u'ports': [{u'protocol': u'TCP', u'containerPort': 80}], u'resources': {}}], u'nodeName': u'ubuntu-231'}, u'apiVersion': u'v1', u'metadata': {u'name': u'web-frontend-jf6s6', u'labels': {u'app': u'web-frontend'}, u'namespace': u'default', u'ownerReferences': [{u'kind': u'ReplicationController', u'name': u'web-frontend', u'apiVersion': u'v1', u'controller': True, u'blockOwnerDeletion': True, u'uid': u'eb88921d-d76f-11e8-b4a7-eabc5652ca9b'}], u'resourceVersion': u'12280868', u'generateName': u'web-frontend-', u'creationTimestamp': u'2018-10-25T04:38:49Z', u'selfLink': u'/api/v1/namespaces/default/pods/web-frontend-jf6s6', u'uid': u'ddf9a74d-d80f-11e8-b4a7-eabc5652ca9b'}}

{u'status': {u'qosClass': u'BestEffort', u'containerStatuses': [{u'restartCount': 0, u'name': u'web-frontend', u'image': u'10.217.6.101:5000/web-test:latest', u'imageID': u'docker-pullable://10.217.6.101:5000/web-test@sha256:a9866688908482a8adbe25937bfdc5bf1f4cb5830e2843d99d3d0ef068ddcd7b', u'state': {u'running': {u'startedAt': u'2018-10-25T04:39:02Z'}}, u'ready': True, u'lastState': {}, u'containerID': u'docker://a5519ce3bcff4178660f35845ef99404c75070ea2708365917e60a248145917b'}], u'podIP': u'10.244.1.162', u'startTime': u'2018-10-25T04:38:49Z', u'hostIP': u'10.106.73.231', u'phase': u'Running', u'conditions': [{u'status': u'True', u'lastProbeTime': None, u'type': u'Initialized', u'lastTransitionTime': u'2018-10-25T04:38:49Z'}, {u'status': u'True', u'lastProbeTime': None, u'type': u'Ready', u'lastTransitionTime': u'2018-10-25T04:39:02Z'}, {u'status': u'True', u'lastProbeTime': None, u'type': u'PodScheduled', u'lastTransitionTime': u'2018-10-25T04:38:49Z'}]}, u'kind': u'Pod', u'spec': {u'dnsPolicy': u'ClusterFirst', u'securityContext': {}, u'serviceAccountName': u'default', u'schedulerName': u'default-scheduler', u'serviceAccount': u'default', u'terminationGracePeriodSeconds': 30, u'restartPolicy': u'Always', u'volumes': [{u'secret': {u'defaultMode': 420, u'secretName': u'default-token-g5svl'}, u'name': u'default-token-g5svl'}], u'tolerations': [{u'operator': u'Exists', u'tolerationSeconds': 300, u'effect': u'NoExecute', u'key': u'node.kubernetes.io/not-ready'}, {u'operator': u'Exists', u'tolerationSeconds': 300, u'effect': u'NoExecute', u'key': u'node.kubernetes.io/unreachable'}], u'containers': [{u'terminationMessagePath': u'/dev/termination-log', u'name': u'web-frontend', u'image': u'in-docker-reg.eng.citrite.net/cpx-dev/web-test:latest', u'volumeMounts': [{u'readOnly': True, u'mountPath': u'/var/run/secrets/kubernetes.io/serviceaccount', u'name': u'default-token-g5svl'}], u'terminationMessagePolicy': u'File', u'imagePullPolicy': u'Never', u'ports': [{u'protocol': u'TCP', u'containerPort': 80}], u'resources': {}}], u'nodeName': u'ubuntu-231'}, u'apiVersion': u'v1', u'metadata': {u'name': u'web-frontend-jf6s6', u'labels': {u'app': u'web-frontend'}, u'namespace': u'default', u'ownerReferences': [{u'kind': u'ReplicationController', u'name': u'web-frontend', u'apiVersion': u'v1', u'controller': True, u'blockOwnerDeletion': True, u'uid': u'eb88921d-d76f-11e8-b4a7-eabc5652ca9b'}], u'resourceVersion': u'12280888', u'generateName': u'web-frontend-', u'creationTimestamp': u'2018-10-25T04:38:49Z', u'selfLink': u'/api/v1/namespaces/default/pods/web-frontend-jf6s6', u'uid': u'ddf9a74d-d80f-11e8-b4a7-eabc5652ca9b'}}

Alternate approaches

We had following alternative options for this.

- Time at which App receives endpoint change notification for the IP
  - - App will record that time at which it has seen notification for the IP
    - MAS will have timestamp from traffic flow and so, it can relate relevant service name.
    - There will be slight time difference these two timestamps, but it can be fine considering that there will be some pause time before k8s reassigns the same IP

- - TODO: On App reboot, App will see IPs in bulk and so, it needs to be handled specially

- Endpoint genration timestamp

- - A k8s endpoint IP is bound to a endpoint Object which has a timestamp. In case, endpoint IP is bound to different endpoint object, then timestamp will change.
  - MAS has timestamp from traffic flow and so, it can relate relevant service name.

- TODO: Here assumption is that K8s will ensure that endpoint IP is reassigned to endpoints with increasingly higher timestamp. For example, if an IP (say 10.244.0.100) s assigned to endpoint with Timestamp T1, and then reassigned to endpoint with Timestamp T2, then T2>T1 . Needs to handle otherwise.

Example output for Endpoint generation timestamp

- Feel of timestamp for an IP

- New IP addition to endpoint object

Reference

https://github.com/kubernetes/kubernetes/issues/33470

Page updated

Google Sites

Report abuse