White-paper on Certificate Authority for clients in Kubernetes environment

Introduction

Laymen explanation

A client container which wants to perform HTTPS with the server needs the CA certificate of server. Do you think that this problem is similar to your browser connecting to HTTPS server? In reality, it is not exactly same. And so, we need to tackle this condition separately.

Technical explanation

There are following differences between browser and container regarding server CA validation.

Candidate approaches for CA authority selection

• You could have one CA per Kubernetes cluster, that’s responsible for signing all the private keys in that cluster (this is the model Kubernetes docmentation usually assumes)

• You could have one global CA that’s responsible for signing ALL your private keys and container certificates

• You could have two CAs

  • one CA you use for services that are externally visibile and

  • another CA you use for internal-only services

Case study from Kubernetes infra component

• Every Kubernetes cluster has a cluster root Certificate Authority (CA). The CA is generally used by cluster components to validate the API server’s certificate, by the API server to validate kubelet client certificates, etc.

• At the same time, Kubernetes facilitates user to provide CA certificate.

  • • For example, the apiserver connects to the kubelet process in each node. API Server has a flag to set CA authority to verify Kubelet's server certificate(Ref: below screenshot taken from here)

Requirement on server side [R-Ser]

• No matter what PKI model admin want to use, admin should be able to make it work with Kubernetes.

  • For North-south apps (AKA frontend apps), its CA certificate authority should be validated via internet based client and so, it should be signed by public CA authority

Requirement on client side [R-Client]

• No matter what PKI model server chooses, client should be able to validate CA certificate of server

  • So, client should facilitate admin to inject server CA authority

• Client should allow admin to inject more than one CA authorities

• Client should have default CA bundle and so, admin needed not explicitly inject it [NFR: ease of use]

• Client should facilitate re-provision of certificate (expiry, revocation, rotation etc can be the use-cases)

Requirement on proxy (Load balancer) [R-Proxy]

• Proxy in K8s or outside acts as client for the server and so, all requirements mentioned above [R-Client] is applicable for proxy as well.

• Since proxy acts on behalf of multiple clients which is known at runtime. So, there are additional requirements (listed below)

• Proxy offloads multiple servers and some server might not be SSL

  • So, in server plane(proxy acting as client), default interaction should be non SSL

  • So, by default communication should be insecure and CA certificate is not needed

  • For any SSL communication with server, proxy should have CA certificate of the server and it should be provided by means mentioned in [R-Client]

Reference

https://jvns.ca/blog/2017/08/05/how-kubernetes-certificates-work/

https://kubernetes.io/docs/concepts/architecture/master-node-communication/

https://blog.digitalocean.com/vault-and-kubernetes/

https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/2983b28f13b294c6422a5600bb6f14142f5e7a26/docs/02-certificate-authority.md