IPv6 upgrade causes major communication failure

This article is reprinted from the WeChat public account "Network Optimization Mercenary", the author is in correspondence. To reprint this article, please contact the WeChat public account "Network Optimization Mercenary".

At the beginning of the new year, a major communication failure occurred again.

On February 1, 2022, Japanese mobile operator NTT DoCoMo suffered a major nationwide communications failure, which caused some 4G and 5G users to be unable to access the Internet and make VoLTE voice calls for up to 5 hours. On February 7, the president of NTT held a press conference to apologize for the accident and announced the cause of the failure.

Fault description

1. Time of failure:

7:30am to 12:13pm, February 1, 2022

2. Impact of the failure:

Some mobile users are unable to use mobile Internet services and voice call services (VoLTE)

3. Cause of failure:

When the IPv6 single stack mode is introduced, the server load increases dramatically and sends control and network communication signaling to the terminal.

4. Number of affected users:

About 18,000

5. Scope of impact:

Nationwide

6. Solution:

In response to the failure of the IPv6 single-stack network upgrade, DoCoMo took the temporary measure of falling back to the "IPv4/IPv6 dual-stack mode" to quickly restore services. In the future, the operator said it will restart the IPv6 single-stack mode upgrade by increasing server capacity and optimizing design.

Why introduce IPv6 single stack mode?

As we all know, IPv4 addresses are 32 bits long, with about 4.2 billion (2^32) IP addresses, which is equivalent to 2 IP addresses for every 3 people in the world. With the popularity of PCs and smartphones, these addresses have been basically allocated and have reached their limit. IPv6 addresses are 128 bits long, and the number of IP addresses available is very large, so that "every grain of sand on the earth has an IP address."

Nowadays, with the increase in IoT devices and the popularization of 5G, the number of devices connected to the Internet continues to increase, and the demand for IP addresses is also increasing. To this end, operators, ISPs, cloud providers and other sectors are actively promoting IPv6 upgrades to solve the problem of IPv4 address exhaustion.

During the transition from IPv4 to IPv6, people generally adopt the IPv4/IPv6 dual-stack mode, that is, assigning two sets of IP addresses, IPv4 and IPv6, to the terminal. If the other end is an IPv4 server, IPv4 is used for communication; if the other end is IPv6, IPv6 is used for communication.

However, in the dual-stack mode, all devices in the mobile network are required to support both IPv4 and IPv6 protocol stacks, which results in slow upgrade cycles, high maintenance costs, and is unable to fundamentally resolve the IPv4 address shortage problem.

In this context, in order to maximize the utilization of IPv6, DoCoMo proposed to introduce IPv6 single stack mode in mobile networks in the spring of 2022.

IPv6 single stack, also called pure IPv6 or IPv6-only, means that only IPv6 addresses are allocated to terminals in mobile data communications.

However, since the IPv6 protocol and the IPv4 protocol are incompatible, as shown in the figure above, after upgrading to an IPv6 single-stack mobile network, the network only allocates IPv6 addresses to terminals. If the target server accessed by the terminal is still IPv4, address conversion processing is required through the server or switch equipment in the DoCoMo mobile network to convert the IPv6 address to IPv4 before the terminal can access the target node with only an IPv4 address.

In response to this, DoCoMo adopted two IP address conversion methods, DNS64/NAT64 and 464XLAT. The former can solve the problem of Native IPv6 users connecting to IPv4 servers, and the latter allows IPv4-Only applications on the terminal to communicate on the IPv6 single-stack network.

According to DoCoMo's official announcement, the communication failure was caused by the introduction of the "IPv6 single stack" mode in the mobile network, which increased the server load and sent signaling to the terminal to control network communication, causing some mobile users to be unable to access ISP (sp-mode) and IMS (VoLTE), and unable to conduct data communication and voice calls. Subsequently, the operator took emergency measures to fall back to the "IPv4/IPv6 dual stack mode", and returned to normal after about 5 hours.