IP Networking: Kubernetes and VPC Additional IP Address Ranges
Posted on January 31, 2024 — 2 Minutes Read
Traditional cloud service providers, whether it is the hyperscalers that count Amazon Web Services (AWS), Microsoft Azure and Google Cloud Platform (GCP), or the niche cloud service providers such as DigitalOcean (DO) or Linode, often build their managed Kubernetes (K8s) offerings on top of their virtual machines (VM) products. It is a natural trajectory for those who started with VM offerings, and allows for resource mobility between VMs and K8s as workloads containerise.
In this model, a K8s cluster will have multiple VMs operating as nodes, and each of them will house numerous pods, on which the containerised workloads will run. Pods on different nodes communicate, either through an encapsulation overlay, often using VXLAN or IP-in-IP, configured by the Container Network Interface (CNI) plugin such as Calico or Cilium; or by leveraging the routing capabilities of the underlay network on which the cluster runs. While the former is relatively simple as its only requirement is for the nodes to be able to reach one another over the underlying network, it incurs MTU overheads which translates directly to a reduced maximum throughput. The latter does not incur any additional overhead or performance cost. Yet it does rely on the underlying network to connect between the pods on the nodes. This, on various cloud platforms, means the Virtual Private Cloud (VPC) or equivalent networking product, on which the cluster runs, will need to assign to every node i.e. VM, an IP address range for its pods, and will need to know which pod resides in which node in order to route the traffic accordingly.
Assigning an additional IP address range from the VPC to a VM is often as simple as an API call. Yet once assigned, the VM will need to be informed that traffic to this range will be routed to it. To do so, one trick I have been using is to add the additional IP address range to its loopback interface.
ip address add *additional-ip-address-range* dev lo
$ ip address add 10.0.0.16/28 dev lo
This allows the VM to receive the traffic regardless of which network interface e.g. eth0, eth1 or eth2 etc. it comes in from. It allows the pods or any other services listening on the loopback interface to respond to it. It is simple and neat.