Deep dive into the Kubernetes network model and network communication

Kubernetes defines a simple and consistent network model based on a flat network structure design. It can communicate efficiently without mapping host ports to network ports, and does not require other components to forward. This model also makes it easy to migrate applications from virtual machines or host physical machines to pods managed by Kubernetes.

This article mainly explores the Kubernetes network model in depth and understands how containers and pods communicate with each other. The implementation of the network model will be introduced in the following article.

Kubernetes Network Model

The model defines:

• Each pod has its own IP address, which is reachable within the cluster.
• All containers in a Pod share the pod IP address (including MAC address), and containers can communicate with each other (using localhost)
• Pods can communicate with other pods on any node in the cluster using the pod IP address, without NAT
• Kubernetes components can communicate with each other and with pods
• Network isolation can be achieved through network policies

Several related components are mentioned in the above definition:

• Pod: A pod in Kubernetes is somewhat similar to a virtual machine with a unique IP address. Pods on the same node share the network and storage.
• Container: A pod is a collection of containers that share the same network namespace. Containers in a pod are like processes on a virtual machine. Processes can communicate using localhost. Containers have their own independent file systems, CPUs, memory, and process spaces. You need to create a pod to create a container.
• Node: pods run on nodes. A cluster contains one or more nodes. The network namespace of each pod is connected to the namespace of the node to open up the network.

We have talked about network namespaces so many times, but how does it work?

How Network Namespaces Work

Create a pod in the Kubernetes distribution k3s. This pod has two containers: the curl container that sends the request and the httpbin container that provides the web service.

Although the distribution is used, it still uses the Kubernetes network model, which does not prevent us from understanding the network model.

 apiVersion : v1
 kind : Pod
 metadata :
 name : multi - container - pod
 spec :
 containers :
 - image : curlimages / curl
 name : curl
 command : [ "sleep" , "365d" ]
 - image : kennethreitz / httpbin
 name : httpbin

Log in to the node and use lsns -t net​ to check the network namespace on the current host, but no httpbin​ process is found. There is a namespace command /pause​, which is actually an invisible sandbox in each pod.

 lsns - t net
 NS TYPE NPROCS PID USER NETNSID NSFS COMMAND
 4026531992 net 126 1 root unassigned / lib / systemd / systemd --system --deserialize 31
 4026532247 net 1 83224 uuidd unassigned / usr / sbin / uuidd --socket-activation
 4026532317 net 4 129820 65535 0 / run / netns / cni - 607 c5530 - b6d8 - ba57 - 420 e - a467d7b10c56 / pause

Since each container has its own process space, we can change the command to view the space of the process type:

 lsns -t pid
 NS TYPE NPROCS PID USER COMMAND
 4026531836 pid 127 1 root / lib / systemd / systemd --system --deserialize 31
 4026532387 pid 1 129820 65535 / pause
 4026532389 pid 1 129855 systemd - network sleep 365 d
 4026532391 pid 2 129889 root / usr / bin / python3 / usr / local / bin / gunicorn - b 0.0 .0 .0 : 80 httpbin : app - k gevent

Through the process PID 129889, you can find the namespace it belongs to:

 ip netns identify 129889
 cni - 607c5530 - b6d8 - ba57-420e - a467d7b10c56

You can then use exec to execute commands in this namespace:

 ip netns exec cni - 607 c5530 - b6d8 - ba57 - 420 e - a467d7b10c56 ip a
 1 : lo : < LOOPBACK , UP , LOWER_UP > mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
 link / loopback 00 : 00 : 00 : 00 : 00 : 00 brd 00 : 00 : 00 : 00 : 00 : 00
 inet 127.0 .0 .1 / 8 scope host lo
 valid_lft forever preferred_lft forever
 inet6 :: 1 / 128 scope host
 valid_lft forever preferred_lft forever
 2 : eth0@if17 : < BROADCAST , MULTICAST , UP , LOWER_UP > mtu 1450 qdisc noqueue state UP group default
 link / ether f2 : c8 : 17 : b6 : 5 f : e5 brd ff : ff : ff : ff : ff : ff link - netnsid 0
 inet 10.42 .1 .14 / 24 brd 10.42 .1 .255 scope global eth0
 valid_lft forever preferred_lft forever
 inet6 fe80 :: f0c8 : 17 ff : feb6 : 5 fe5 / 64 scope link
 valid_lft forever preferred_lft forever

From the result, we can see that the pod’s IP address 10.42.1.14​ is bound to the interface eth0​, and eth0​ is connected to the interface 17.

On the node host, check the information of interface 17. veth7912056b is the virtual Ethernet device in the host root namespace. It is the tunnel connecting the pod network and the node network. The other end is the interface eth0 in the pod namespace.

 ip link | grep -A1 ^ 17
 17 : veth7912056b@if2 : < BROADCAST , MULTICAST , UP , LOWER_UP > mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default
 link / ether d6 : 5 e : 54 : 7 f : df : af brd ff : ff : ff : ff : ff : ff link - netns cni - 607 c5530 - b6d8 - ba57 - 420 e - a467d7b10c56

From the above results, we can see that the veth is connected to the network bridge cni0.

The bridge works at the data link layer (Layer 2 of the OSI model), connecting multiple networks (multiple network segments). When a request arrives at the bridge, the bridge will ask all connected interfaces (here the pod is connected to the bridge via veth) whether they have the IP address in the original request. If there is an interface response, the bridge will record the matching information (IP -> veth) and forward the data.

What if there is no interface response? The specific process depends on the implementation of each network plug-in. I plan to introduce commonly used network plug-ins in the following articles, such as Calico, Flannel, Cilium, etc.

Next, let's look at how network communication is accomplished in Kubernetes. There are several types:

• Communication between containers in the same pod
• Communication between pods on the same node
• Communication between pods on different nodes

How Kubernetes Networking Works

Communication between containers in the same pod

Communication between containers in the same pod is the simplest. These containers share a network namespace, and each namespace has a lo​ loopback interface, which can be used to communicate through localhost.

Communication between pods on the same node

When we run the curl​ container and httpbin​ in two pods respectively, these two pods may be scheduled to the same node. The request sent by curl​ reaches the eth0​ interface in the pod according to the routing table in the container. Then it reaches the root network space of the node through the tunnel veth1 connected to eth0​.

veth1​ is connected to other pods through the virtual Ethernet interface vethX​ via the bridge cni0​. The bridge will ask all connected interfaces whether they have the IP address in the original request (such as 10.42.1.9​ here). After receiving the response, the bridge will record the mapping information (10.42.1.9​ => veth0​) and forward the data. Finally, the data enters the pod httpbin through the veth0​ tunnel.

Communication between pods on different nodes

Communication between pods across nodes is more complicated, and different network plug-ins handle it differently. Here we choose an easy-to-understand way to briefly explain it.

The process in the first half is similar to the communication between pods on the same node. When the request reaches the bridge, the bridge asks which pod has the IP but does not receive a response. The process enters the host's routing addressing process and reaches a higher cluster level.

At the cluster level, there is a routing table that stores the Pod IP network segment of each node (a Pod network segment (Pod CIDR) is assigned to a node when it joins the cluster. For example, the default Pod CIDR in k3s is 10.42.0.0/16​, and the network segments obtained by the node are 10.42.0.0/24, 10.42.1.0/24, 10.42.2.0/24, and so on). The node requesting the IP can be determined through the Pod IP network segment of the node, and then the request is sent to the node.

Summarize

Now you should have a preliminary understanding of Kubernetes network communication.

The entire communication process requires the cooperation of various components, such as the Pod network namespace, the Pod Ethernet interface eth0​, the virtual Ethernet interface vethX​, the network bridge cni0, etc. Some of these components correspond to pods one by one and have the same life cycle as the pods. Although they can be created, associated, and deleted manually, non-permanent resources such as pods are frequently created and destroyed, and too much manual work is unrealistic.

In fact, all these tasks are delegated by the container to the network plug-in, and the network plug-in follows the CNI (Container Network Interface) specification.

What do network plugins do?

• Create a network namespace for the pod (container)
• Creating an interface
• Create veth pair
• Setting up namespace network
• Setting up static routing
• Configuring Ethernet Bridge
• Assigning IP addresses
• Creating NAT Rules

...

• refer to

https://www.tigera.io/learn/guides/kubernetes-networking/

https://kubernetes.io/docs/concepts/services-networking/

https://matthewpalmer.net/kubernetes-app-developer/articles/kubernetes-networking-guide-beginners.html

https://learnk8s.io/kubernetes-network-packets