Cutover failure leads to major communication failure

[[429302]]

Japan has once again experienced a major network incident since SoftBank's major communications failure in 2018.

At around 5 pm local time on October 14, 2021, operator NTT DoCoMo's mobile network suffered a major communication failure across Japan, causing a large number of mobile phone users to be unable to make calls and data communications.

The accident had a wide impact and shocked the whole of Japan.

After the accident, Japan's Minister of Internal Affairs and Communications said at a press conference after the cabinet meeting:

It is regrettable that a large-scale failure of the mobile network, an important infrastructure that affects people's daily lives, occurred. The Ministry of Internal Affairs and Communications takes this matter very seriously and has asked NTT DoCoMo to promptly investigate and report the cause and extent of the accident so as to provide a full explanation to the majority of users. We hope that NTT DoCoMo will fulfill its social responsibility and take all possible measures to prevent similar accidents from happening again.

NTT DoCoMo management publicly apologized, expressed deep regret for the inconvenience caused to customers and many people by the accident, and said it would work hard to prevent the accident from happening again.

[[429303]]

NTT DoCoMo management publicly apologizes

Fault description

NTT DoCoMo's official website stated that starting at around 5 pm on October 14, 2021, a network incident occurred that made DoCoMo's voice calls and data communication services difficult to use.

At 7:57 pm on October 14, 2021, the operator took emergency network operations and the fault began to gradually recover, but due to network congestion, some customers were still unable to connect to the network.

At 5:05 am on October 15, 2021, 5G and 4G networks returned to normal, but the 3G network in some areas is still difficult to use, and efforts are being made to restore it. We have informed the majority of users that users who have subscribed to 4G packages and have 3G signals can connect to the 4G network by restarting their phones to obtain normal communication.

On the afternoon of October 15, 2021, the vice president of NTT DoCoMo stated at a press conference that "no clear time can be given" for the restoration of the 3G network, explaining that the outlook is unclear.

Cause

On October 14, NTT DoCoMo’s official website explained that the network congestion was caused by an increase in network signaling due to “switching” operations during network construction.

On October 15, NTT DoCoMo held a press conference to further explain the cause of the failure.

The specific reasons are as follows:

In the early morning of October 14, the operator began to upgrade the server that stores user/location information of IoT terminal devices, replacing old devices with new ones.

Image source: NTT DoCoMo

But then something unexpected happened: problems arose in the process of migrating the location information of about 200,000 IoT terminals from old devices to new devices.

At 4:36 p.m. on October 14, after the problem was discovered, a rollback operation was initiated, returning to the old equipment.

Image source: NTT DoCoMo

However, this rollback operation caused a large number of IoT terminals to re-initiate location registration information to the old server. A surging "signaling storm" quickly caused network congestion and affected the voice and data packet core equipment of the 3G/4G/5G network at around 5 pm, causing a large number of users to be unable to make calls and data communications.

Image source: NTT DoCoMo

NTT DoCoMo said the sudden influx of signaling traffic was much higher than expected.

It appears that this was a major network failure caused by a failed network cutover.

The accident triggered reflections from all walks of life in Japan. After the failure, many netizens said that online classes were forced to stop, taxis could not be paid with smartphones, and electronic tickets could not be displayed on site at the movies... It brought great inconvenience to daily life.

Communications industry research institutions and media commented that at present, mobile networks have become social infrastructure like water, electricity, and gas. Once a major failure occurs, it will have a great impact on people's daily lives. In the 5G era, the network will connect home appliances, factory machines, self-driving cars, etc. If a major failure occurs, the impact will be even greater. Therefore, regulators and the telecommunications industry should strengthen redundancy backup to ensure the reliability and stability of the network.