VXLAN technology introduction: Building a virtual Layer 2 network with a Layer 3 network

1. VXLAN Overview

1. What is VXLAN

VXLAN (Virtual Extensible LAN) is an overlay network technology that encapsulates the original Layer 2 Ethernet frame with UDP (MAC-in-UDP), adds an 8-byte VXLAN header, an 8-byte UDP header, a 20-byte IP header, and a 14-byte Ethernet header, for a total of 50 bytes.

2. VXLAN Advantages

Compared with VLAN, VXLAN provides better scalability and flexibility, with the following features:

• Flexible application deployment: Layer 2 Ethernet frames encapsulated by VXLAN can cross Layer 3 network boundaries, making networking and application deployment more flexible while solving IP address conflicts in multi-tenant network environments.
• Better scalability: The traditional VLANID field is 12-bit, and the maximum number of VLANs is 4096; VXLAN uses 24-bit VNID (VXLAN network identifier) ​​and supports a maximum of 16,000,000 logical networks.
• Improve network utilization: Traditional Ethernet uses STP to prevent loops. STP causes network redundant paths to be blocked. VXLAN messages are transmitted based on the Layer 3 IP header, which can effectively utilize network paths and support ECMP (equal-cost multipath) and link aggregation protocols.

(1) Flexible application deployment

As shown in Figure 1-1, application deployment in a VXLAN environment is not restricted by physical location and Layer 3 network boundaries. For example, if the address segment of an application is 192.168.1.0/24, all application servers or virtual machines must be deployed in the same Layer 3 network in a traditional network. Otherwise, routing or address conflicts will occur.

(2) Better scalability

Traditional networks use VLANs to logically isolate customer networks. The VLAN ID field is 12-bit and the maximum number of VLANs is 4096. VXLAN uses a 24-bit VNID (VXLAN network identifier) ​​and supports up to 16,000,000 logical networks, greatly enhancing scalability.

(3) Improving network utilization

Figure 1-3 Using Layer 3 interfaces to interconnect after VXLAN is used to eliminate spanning tree blocked ports

Traditional Ethernet frames cannot traverse Layer 3 networks. After VXLAN is deployed, data between VTEPs is based on Layer 3 addressing, and network interconnection interfaces are no longer Layer 2 interfaces. The interconnection interfaces between switches can be deployed in Layer 3 mode, eliminating spanning tree blocked ports, improving network utilization, and supporting ECMP (equal-cost multipath) and link aggregation protocols.

2. VXLAN Terminology

1. VTEP

VXLAN

Tunnel Endpoint (VTEP). VXLAN uses VTEP devices to encapsulate and decapsulate VXLAN packets, including ARP request packets and normal VXLAN data packets. VTEP encapsulates the original Ethernet frame through VXLAN and sends it to the peer VTEP device. After receiving the VXLAN packet, the peer VTEP decapsulates it and forwards it according to the original MAC. VTEP can be implemented by a physical switch, physical server, or other hardware device or software that supports VXLAN.

2. VNI

Virtual Network ID (VNI), VNI is encapsulated in the VXLAN header, a total of 24 bits, and supports up to 16,000,000 logical networks.

3. VXLAN Gateway

The VXLAN gateway is used to connect the VXLAN network and the traditional VLAN network. The VXLAN gateway implements the mapping between VNI and VLAN ID. The VXLAN gateway is actually also a VTEP device.

4. Multicast group

VTEP devices need to join the same multicast group, which is mainly used for control plane address learning.

3. VXLAN Encapsulation

VXLAN uses UDP to encapsulate a complete Ethernet frame (MAC-in-UDP), with a total of 50 bytes of encapsulation message header. The specific message format is as follows:

1. Inner MAC

Inner MAC: The inner MAC is the MAC address of the original Ethernet frame.

2. VXLAN Header

There are 8 bytes in total. Currently, an 8-bit identification bit in Flags and a 24-bit VNI (Vxlan Network identifier) ​​are used. The rest is undefined, but must be set to 0x0000 when used.

3. Outer UDP Header

A total of 8 bytes. The standard destination port assigned by IANA is 4798, but manufacturers can modify it as needed. At the same time, the UDP checksum must be set to all 0s.

4. Outer IP Header

A total of 20 bytes, the destination IP address can be a unicast address or a multicast address. In the unicast case, the destination IP address is the IP address of the destination VTEP; when used for the VXLAN control plane, a multicast address is used.

Outer IP: The outer IP address is the Layer 3 IP address after VTEP encapsulation. The source IP is the local VTEP device IP. When used for the control plane, the destination IP can be a multicast address. When used for the forwarding plane, the destination IP is the remote VTEP device IP.

5. Outer Ethernet Header

A total of 14 bytes, the outer Ethernet frame header. Outer MAC, the outer MAC is the Layer 2 MAC after VTEP encapsulation, the source MAC is the MAC of the local VTEP device, and the destination MAC can be the MAC of the remote VTEP device or the MAC of the Layer 3 network device in the middle of the transmission path.

4. VXLAN Data Forwarding

1. Control Plane

In the implementation of VXLAN, when the control plane path discovery is implemented through multicast, a stateless tunnel is used between VTEP devices, and the stateful long connection is not maintained between VTEP devices. VXLAN needs to learn the address information of the remote device through the control plane and build the control plane table entry locally. The control plane table entry consists of a triple of VNI, Inner Source MAC, and Outer Source IP.

2. Forwarding plane

After the control plane learns the address mapping information, the forwarding plane is responsible for forwarding the actual data. VTEP adds a UDP header to the original data frame, and the new header will be removed only after it reaches the destination VTEP. The network devices in the intermediate path will only forward data according to the destination address in the outer packet header.

3. VXLAN ARP request

As shown in the figure above, terminal device A needs to communicate with terminal device B. The ARP request process is as follows:

• Terminal device A sends an ARP request, requesting the MAC address of terminal device B;
• VTEP-1 receives the ARP request sent by terminal device A. At this time, VTEP-1 does not have the address mapping table entry corresponding to terminal device B. VTEP-1 encapsulates the ARP request into VXLAN, sets VNI to 10, outer-src-ip is the IP of VTEP-1, outer-dst-ip is the multicast group address to be joined, and forwards it to the VXLAN multicast group after encapsulation.
• VTEP-2 and VTEP3 join the same multicast group. All group members will receive the multicast message sent by VTEP-1. After decapsulation, they will check whether the VNI matches the local VNI. If they match, they will send an ARP request to the local network. At the same time, they will record the correspondence between VNI, inner MAC, and outer IP, and build a control plane address mapping table entry. If the VNI does not match, the data packet will be discarded.
• After receiving the ARP request, terminal device B sends an ARP response in unicast mode;
• After VTEP-2 receives the ARP response from terminal device B, it performs VXLAN encapsulation. At this time, VTEP-2 has built the control plane address mapping table entry and sends it in unicast mode after VXLAN encapsulation. Outer-src-ip is the IP address of VTEP-2, and outer-dst-ip is the IP address of VTEP-1;
• After receiving the encapsulated ARP response, VTEP-1 decapsulates and compares the VNI. If a match is found, the ARP response is sent to terminal device A. At the same time, the corresponding relationship between VNI, inner MAC, and outer IP is recorded to construct a control plane table entry.
• At this point, VTEP-1 and VTEP-2 have successfully constructed the control plane address mapping information, and subsequent VXLAN data is transmitted between VTEP-1 and VTEP-2 using unicast.

4. VXLAN Data Transmission

• After the ARP request is completed, terminal device A sends data to terminal device B. VTEP-1 receives the data and searches for the address mapping table entry, encapsulates the original data into VXLAN, and forwards it to VTEP-2.
• After receiving the VXLAN data packet, VTEP-2 checks whether the VNI matches the local VNI. If so, it decapsulates the original Ethernet frame and forwards it to terminal device B.

VXLAN Deployment

6. Supplement:

• When performing ARP processing, in order to transmit broadcasts through multicast, it is necessary to set up a mapping from VNI to multicast groups. This mapping belongs to the management layer and is used to establish management channels between VTEPs. Unknown MAC destinations will also be multicast encapsulated and processed in the same way as broadcasts.
• VXLAN messages cannot be fragmented. Intermediate devices may fragment VXLAN messages, but VTEP will discard the fragmented messages. To ensure that VXLAN messages are not fragmented, the MTU of all devices along the way needs to be modified. The RFC document does not explain why VTEP must discard fragmented messages.
• VLAN TAG information will be stripped during encapsulation and decapsulation unless otherwise configured.