Comparative Analysis of Kubernetes Network Plugins (Flannel, Calico, Weave)

[[269494]]

This article will focus on exploring and comparing the most popular CNI plugins in detail, based on the introduction of technical principles and corresponding terminology:

• Flannel
• Calico
• Weave

introduce

Network architecture is one of the more complex aspects of Kubernetes that causes headaches for many users. The Kubernetes network model itself has certain requirements for certain specific network functions, but it also has certain flexibility in implementation. Therefore, there are many different network solutions in the industry to meet specific environments and requirements.

CNI stands for Container Network Interface, which is a standard design that allows users to more easily configure container networks when creating or destroying containers. In this article, we will focus on exploring and comparing the most popular CNI plugins: Flannel, Calico, Weave, and Canal (technically a combination of multiple plugins). These plugins can ensure that Kubernetes' network requirements are met and provide Kubernetes cluster administrators with certain specific network functions they need.

background

Container networking is the mechanism by which containers choose to connect to other containers, the host, and external networks (such as the Internet). The container runtime provides a variety of network modes, each of which produces a different experience.

For example, Docker can configure the following networks for containers by default:

• none: Add the container to a container-specific network stack with no external connectivity.
• host: Adds the container to the host's network stack, with no isolation.
• default bridge: Default network mode. Each container can connect to each other via IP addresses.
• Custom Bridge: A user-defined bridge with more flexibility, isolation, and other convenience features.

Docker also allows users to configure more advanced networking (including multi-host overlay networks) through additional drivers and plugins.

The original intention of CNI was to create a framework for dynamically configuring appropriate network configuration and resources when configuring or destroying containers. The CNI specification in the following link outlines the plug-in interface for configuring the network, which allows the container runtime to coordinate with the plug-in:

https://github.com/containernetworking/cni/blob/master/SPEC.md

The plugin is responsible for configuring and managing IP addresses for interfaces, and generally provides functionality related to IP management, IP allocation per container, and multi-host connectivity. The container runtime calls the network plugin to allocate IP addresses and configure the network when the container starts, and calls it again to clean up these resources when the container is deleted.

The runtime or orchestrator decides which network the container should join and which plugin it needs to call. The plugin then adds the interface to the container network namespace as one side of a veth pair. Next, it makes changes on the host, including connecting the other part of the veth to the bridge. After that, it allocates IP addresses and sets up routing by calling a separate IPAM (IP Address Management) plugin.

In Kubernetes, kubelet can call the plugins it finds at the appropriate time to automatically configure the network for the pods started by kubelet.

the term

Before comparing CNI plugins, we can first have an overall understanding of the related terms that we will see on the Internet. Whether reading this article or coming into contact with other CNI-related content in the future, it is always very useful to understand some common terms.

Some of the most common terms include:

• Layer 2 Network: The "data link" layer of the OSI (Open Systems Interconnections) network model. Layer 2 networks handle the delivery of frames between two adjacent nodes on the network. A notable example of a layer 2 network is Ethernet, where MAC is represented as a sublayer.
• Layer 3 Networking: The "Network" layer of the OSI networking model. The primary focus of layer 3 networking is routing packets between hosts above layer 2 connections. IPv4, IPv6, and ICMP are examples of layer 3 network protocols.
• VXLAN: Stands for "Virtual Extensible LAN". VXLAN was first used to help large cloud deployments by encapsulating Layer 2 Ethernet frames in UDP datagrams. VXLAN virtualization is similar to VLANs, but offers greater flexibility and functionality (VLANs are limited to 4096 network IDs). VXLAN is an encapsulation and overlay protocol that runs on existing networks.
• Overlay network: An overlay network is a virtual logical network built on top of an existing network. Overlay networks are often used to provide useful abstractions on top of existing networks and to separate and protect different logical networks.
• Encapsulation: Encapsulation is the process of wrapping a network packet in an additional layer to provide additional context and information. In overlay networks, encapsulation is used to translate from the virtual network to the underlying address space so that it can be routed to different locations (the packet can then be decapsulated and continue on to its destination).
• Mesh network: A mesh network is one where each node is connected to many other nodes to collaborate on routing and achieve greater connectivity. A mesh network allows routing through multiple paths, providing a more reliable network. The downside of a mesh is that each additional node adds a significant amount of overhead.
• BGP: Stands for "Border Gateway Protocol" and is used to manage how packets are routed between edge routers. BGP helps figure out how to send packets from one network to another by taking into account available paths, routing rules, and specific network policies. BGP is sometimes used as a routing mechanism in CNI plugins instead of encapsulated overlay networks.

Now that we understand the technical terminology and the various technologies that support each type of plugin, we can start exploring some of the most popular CNI plugins.

CNI Comparison

Flannel

Link: https://github.com/coreos/flannel

Flannel, a project developed by CoreOS, is probably the most direct and advanced CNI plugin. It is one of the most mature network structure examples in container orchestration systems, aiming to achieve better inter-container and inter-host networking. With the rise of the CNI concept, the Flannel CNI plugin is an early entry.

Compared with other solutions, Flannel is relatively easy to install and configure. It is packaged as a single binary file FlannelD, and many common Kubernetes cluster deployment tools and many Kubernetes distributions can install Flannel by default. Flannel can use the existing etcd cluster of the Kubernetes cluster to store its state information using the API, so no dedicated data storage is required.

Flannel configures a Layer 3 IPv4 Overlay network. It creates a large internal network that spans every node in the cluster. In this Overlay network, each node has a subnet for internally assigning IP addresses. When you configure a Pod, the Docker bridge interface on each node assigns an address to each new container. Pods in the same host can communicate using Docker bridging, while pods on different hosts use flanneld to encapsulate their traffic in UDP packets so that they can be routed to the appropriate destination.

Flannel has several different types of backends that can be used for encapsulation and routing. The default and recommended method is to use VXLAN, as VXLAN performs better and requires less manual intervention.

Overall, Flannel is a good choice for most users. From a management perspective, it provides a simple network model, and users only need some basic knowledge to set up an environment suitable for most use cases. Generally speaking, using Flannel in the early stages is a safe and secure choice until you start to need something it can't provide.

Calico

Link: https://github.com/projectcalico/cni-plugin

Calico is another popular network choice in the Kubernetes ecosystem. While Flannel is recognized as the simplest choice, Calico is known for its performance and flexibility. Calico is more comprehensive, not only providing network connectivity between hosts and pods, but also network security and management. The Calico CNI plugin encapsulates the functionality of Calico within the CNI framework.

On a newly configured Kubernetes cluster that meets the system requirements, users can quickly deploy Calico by applying a single manifest file. If you are interested in Calico's optional network policy features, you can apply additional manifests to the cluster to enable these features.

Although the actions required to deploy Calico seem fairly simple, the network environment it creates has both simple and complex attributes. Unlike Flannel, Calico does not use an overlay network. Instead, Calico configures a Layer 3 network that uses the BGP routing protocol to route packets between hosts. This means that packets do not need to be wrapped in an additional encapsulation layer when moving between hosts. The BGP routing mechanism can direct packets locally without the need to additionally package traffic in the traffic layer.

In addition to the performance benefits, users can also use more conventional methods to troubleshoot when network problems occur. While encapsulation using technologies such as VXLAN is also a good solution, the way the process handles packets is also difficult to track. With Calico, standard debugging tools can access the same information as in simple environments, making it easier for more developers and administrators to understand the behavior.

In addition to network connectivity, Calico is also known for its advanced network features. Network policy is one of its most sought-after features. In addition, Calico can also be integrated with the service mesh Istio to interpret and implement policies for workloads within the cluster at the service mesh layer and the network infrastructure layer. This means that users can configure powerful rules that describe how Pods should send and accept traffic, improve security, and control the network environment.

If supporting network policies is important to your environment and you have other performance and functionality requirements, Calico is an ideal choice. In addition, if you may want technical support now or in the future, Calico offers commercial support. In general, Calico is a good choice when you want to be able to control your network long-term rather than just configure it once and forget it.

Weave

Link: https://www.weave.works/oss/net/

Weave is a Kubernetes CNI networking option provided by Weaveworks that provides a different model than all the networking solutions we have discussed so far. Weave creates a mesh overlay network between each node in the cluster, with flexible routing between participants. This feature, combined with some other unique features, allows Weave to route intelligently in certain situations that may cause problems.

To create a network, Weave relies on routing components installed on each host in the network. These routers then exchange topology information to maintain a comprehensive view of the available network environment. When traffic needs to be sent to a Pod located on a different node, the Weave routing component automatically decides whether to send it through the "fast data path" or fall back to the "sleeve" packet forwarding method.

The fast datapath relies on the kernel's native Open vSwitch datapath module to forward packets to the appropriate Pod without having to move in and out of userspace multiple times. The Weave router updates the Open vSwitch configuration to ensure that the kernel layer has accurate information about how to route incoming packets. Conversely, sleeve mode can be used as a backup when the network topology is not suitable for fast datapath routing. It is a slower encapsulation mode that can be used to route packets in situations where the fast datapath lacks the necessary routing information or connectivity. As traffic passes through routers, they learn which peers are associated with which MAC addresses, allowing them to route subsequent traffic more intelligently with fewer hops. This same mechanism can help each node correct itself when network changes cause available routes to change.

Like Calico, Weave also provides network policy capabilities for Kubernetes clusters. When setting up Weave, network policy is automatically installed and configured, so users do not need to perform additional configuration other than adding network rules. A feature that is unique to Weave and does not exist in other network solutions is simple encryption of the entire network. Although this will add considerable network overhead, Weave can use NaCl encryption to automatically encrypt all routing traffic for sleeve traffic, and for fast data path traffic, because it needs to encrypt VXLAN traffic in the kernel, Weave uses IPsec ESP to encrypt fast data path traffic.

Weave is a great choice for those looking for a feature-rich network without adding a lot of complexity or management difficulty. It is relatively easy to set up, provides many built-in and automatically configured features, and can provide intelligent routing in scenarios where other solutions may fail. The mesh topology does limit the size of the network that can be reasonably accommodated, but for most users, this is not a big problem. In addition, Weave also provides paid technical support, which can provide troubleshooting and other technical services for enterprise users.

Conclusion

The CNI standard adopted by Kubernetes has allowed a wide variety of network solutions to flourish in the Kubernetes ecosystem. More diverse choices mean that most users will be able to find a CNI plug-in that suits their current needs and deployment environment, while also being able to find new solutions when the environment changes.

The operational requirements of different enterprises vary greatly, so having a range of mature solutions with different levels of complexity and feature richness greatly helps Kubernetes to provide a consistent user experience while meeting the unique needs of different users.