Exploration and practice of multi-link transmission technology in Volcano Engine RTC

Traditional data transmission methods mostly use a link and select the default network card of the device for transmission. When using this method to implement real-time audio and video calls, if there is a problem with the default network (such as disconnection, weak network, etc.), the user's communication will be interrupted or stuck, affecting the user experience.

Multi-link transmission, as the name implies, is a technology that uses multiple links to transmit data. In recent years, the technology of supporting multiple available network cards on a single device has become more and more common. For example, our mobile phones support both wireless network cards and 4G/5G network cards. Some dual-SIM phones can also support two 4G/5G network cards at the same time. Multi-link technology is to make full use of multiple network resources on the user's device for data transmission. When a problem occurs on a certain network, other available networks can continue to transmit data uninterruptedly, avoiding call interruptions or freezes due to a single network problem, and improving the availability and fluency of user calls.

At present, multi-link transmission technology has been polished to a basic maturity in Volcano Engine RTC, and has been implemented in business scenarios such as Douyin and Feishu Conference, effectively reducing the user's call freeze rate and improving the user experience.

1. Industry status and challenges

Multi-link technology has been applied in some industries. For example, Apple's MPTCP (Multi-Path TCP) is a multi-link technology. Apple uses it in applications such as Siri, Apple Maps, and Apple Music to solve the problem of unsmooth application usage caused by the system network frequently switching between Wifi and cellular networks when users are moving outdoors. When we use WeChat's audio and video call function, WeChat will also enable the audio dual-send function, that is, use Wifi and cellular networks to send audio data at the same time to improve the user's call experience.

The use of multi-link transmission technology adds an extra layer of "reliability" to data transmission and makes network switching smoother. Of course, while enjoying the benefits of multiple network transmissions, we also need to solve some problems in the implementation of multi-link technology.

First, the architecture of multi-link transmission is more complex than that of single-link transmission . For example, it is necessary to consider the data packet redundancy and deduplication problems caused by multiple link transmissions, and how to accurately evaluate the bandwidth and quality of multiple links. Second, it is necessary to balance user experience and traffic and power consumption . When using multiple links to transmit data, it will inevitably cause traffic consumption to increase, especially the user's cellular traffic. The increase in traffic consumption will not only affect bandwidth consumption and crowding out, but also affect the power consumption of the user's mobile phone, seriously affecting the user experience.

There are two main ways to use multi-links in the industry: one is fully redundant dual-link transmission , that is, as long as the dual-link is turned on, double data will be transmitted under any circumstances. The disadvantages of this method are obvious. It does not judge the quality of the user's network and performs dual-link transmission "brainlessly" , which will obviously increase traffic and power consumption, and may cause problems such as heating of the user's mobile phone, affecting the user experience; the other is to switch between the two links . This method can avoid the increase in power consumption and traffic caused by the first fully redundant transmission method, but this method also has disadvantages. For example, when the two links are switched, it may cause unevenness and jamming, and this transmission method cannot fully utilize the benefits of "increased reliability" brought by the two link resources.

Based on the full redundancy mode, Volcano Engine RTC has developed a transmission mode that takes into account both user experience and traffic and power consumption: weak network redundant transmission . As the name implies, weak network redundant transmission means that the dual-link capability is enabled to transmit data only when the RTC system detects a weak network. This mode can fully utilize two links to transmit data when the user's network is weak, so that the user experience will basically not be affected by the weak network; it can also use single-link transmission when the user's network is normal, reducing the consumption of user traffic and power consumption.

2. Evolution of multi-link transmission technology

2.1 Fully redundant transmission

Fully redundant transmission means that data, including audio, video, signaling, messages, etc., is transmitted between the client and the server through two independent paths. The data of the main link will be completely copied to the backup link. Fully redundant transmission relies on multi-path architecture, multi-path protocol, and deduplication algorithm, among which multi-path protocol and deduplication algorithm are the key parts. Fully redundant transmission can effectively avoid communication interruption caused by problems in a single path, thereby improving the availability of user calls.

2.1.1 Architecture Design

In terms of architectural design, fully redundant transmission only exists between the SDK connection module and the edge media node. On the sending end, the SDK transmission module is responsible for redundancy of the upper layer data and sending it out through two channels at the same time. The edge media node is responsible for deduplication and restoration of the data when it is received, and then handing it over to the upper layer service. On the receiving end, the edge media node is responsible for redundant transmission of data, while the SDK transmission module is responsible for deduplication and restoration of the received data and handing it over to the upper layer module.

Fully redundant multi-link transmission architecture

2.1.2 Multipath Protocol

The design of the multipath protocol must meet at least two requirements: one is to be able to distinguish redundant multipath packets from normal data packets (such as RTP/RTCP packets, STUN packets, etc.), and the other is to be able to deduplicate redundant packets.

Therefore, we designed the 4-byte multipath packet header as shown above: the Packet Type field is used to identify redundant multipath packets and distinguish them from normal RTP/RTCP and STUN packets; the Packet Sequence Number is used to number multipath packets for deduplication; the last one is a reserved field.

For example, the format of a multipath RTP packet is as follows:

 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 0xc8 | packet sequence number | default to 0 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |V=2|P|1| CC |M| PT | sequence number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | timestamp | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | synchronization source (SSRC) identifier | +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ | 0x10 | 0x00 | length=0 | +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ | RTP payload | | .... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

2.1.3 Deduplication Algorithm

We use the transmission module to save a bit array bitArrary of 65536 (16 bits) in length, initialize it to 0, and use a sliding window to filter data packets with duplicate sequence numbers.

Illustration of deduplication algorithm

As shown in the figure above, resetSeq is the leftmost value of the window, latestSeq is the rightmost value of the window, and the effective judgment window is between resetSeq and latestSeq, that is, windowSize, which is set to 30000. When the module receives a packet, it records the seqNum of this packet as curSeq and performs the following processing:

1. If this packet is the first packet received by the module, set latestSeq = seqNum, resetSeq = latestSeq - windowSize, and notify the upper layer that a new packet has been received;
2. If this packet is not the first packet received by the module, the distance between curSeq and latestSeq is determined. If it is greater than windowSize, the packet is considered to be an invalid packet (sequence number jump is too large) and is discarded; if it is not greater than windowSize, it is considered to be a valid packet.
3. In the second step, if curSeq < latestSeq, check whether the bit corresponding to curSeq in bitArray is 1. If not, it means that we have received a new packet, set this bit to 1, and inform the upper layer; if it is already 1, it means that we have received a duplicate packet, which is discarded directly. If curSeq > latestSeq, it is also considered that a new packet has been received, inform the upper layer, and move the window, set latestSeq = seqNum, resetSeq = latestSeq - windowSize.

2.1.4 Effect

We simulated various network environments to test the impact of fully redundant multi-link transmission (hereinafter referred to as "full dual transmission") on audio and video quality, device performance, and power consumption.

The following test data is the audio and video call test results when a weak network is added to the main link (Wifi) when both Wifi and cellular dual-link networks are enabled.

User Experience

Overall, when full dual-transmission is turned on, the weak network of a single path has basically no effect on the overall audio and video quality and end-to-end delay performance, and users can still get a high-quality audio and video call experience; when full dual-transmission is turned off, the weak network of a single path will cause obvious lag and delay problems.

After turning on/off full dual transmission, the impact of a single-path weak network on audio and video quality

As shown in the figure above, after turning on full dual transmission, the sound quality (MOS score) and picture quality (video frame rate) are stable in different weak network environments, and the sound quality and picture quality are basically not affected by the weak network. After turning off full dual transmission, the sound quality and picture quality will be affected to varying degrees depending on the weak network scenario, especially when the speed limit is 200kbps, the sound quality and picture quality will be severely degraded.

After turning on/off full dual transmission, the impact of a single-path weak network on the end-to-end delay of audio and video

As shown in the figure above, after turning on full dual-transmission, the end-to-end delay performs well and stably in different weak network environments and is basically not affected by weak networks. After turning off full dual-transmission, when there is high packet loss, high latency and weak network, the end-to-end delay increases significantly and the user experience decreases significantly.

CPU and memory performance

In various network environments, there is not much difference in CPU and memory consumption when full dual-transmission is enabled or disabled.

After turning on/off full dual transmission, the impact of a single-path weak network on the end-to-end delay of audio and video

Power consumption

The test results show that turning on full dual-transmission will have a certain impact on the power consumption of the device and increase the power consumption of calls.

The impact of turning on/off full dual transmission on the power consumption of the mobile phone (the picture shows the test results using iPhone 11)

We used the same mobile phone to make continuous audio and video calls. Turning on full dual-transmission consumed more power and faster than turning it off. After a 1-hour call, turning on dual-transmission consumed 4% more power than turning it off.

2.1.5 Pros and Cons of the Solution

Fully redundant multi-link transmission can effectively solve the audio and video freeze and delay problems caused by problems with a single network, improve user experience, and its implementation is relatively simple. However, its disadvantages are also obvious. First, it consumes a lot of traffic, especially in video scenarios, which will consume a lot of user mobile traffic; second, it also consumes a lot of power consumption. Therefore, full dual transmission does not meet the actual business needs and user experience. We need to improve the user call experience while minimizing traffic consumption and power consumption.

2.2 Redundant transmission in weak network

Fully redundant transmission means that no matter what the situation is, the data will be copied to another link for transmission. This will cause users to consume more traffic and power consumption. In fact, in most cases, the network of users is normal and there is no need to enable redundant transmission. For a normal network, enabling redundant transmission is a waste and has a negative effect. Therefore, we need to enable multi-link transmission in a more targeted manner, which can not only improve the user's call experience, but also save traffic and power consumption.

As the name suggests, weak network redundant transmission is to add weak network judgment on the basis of fully redundant transmission, that is, multi-link transmission is only enabled when the network is weak. The key lies in the timeliness and accuracy of weak network judgment. The system needs to accurately judge the occurrence of weak network and enable multi-link transmission in time (hereinafter referred to as "weak network dual transmission"), so as to effectively avoid network freezes.

2.2.1 Architecture Design

The multipath protocol and deduplication algorithm used by weak network dual-transmission are basically the same as those of full dual-transmission. The main difference is that weak network dual-transmission adds the judgment of weak network conditions on the basis of full dual-transmission. Only when a weak network is detected in the main link will the system start dual-transmission (using the mobile network to send data), otherwise it will not be turned on. This ensures that no extra traffic and power are consumed under normal network conditions, and improves user experience under weak network conditions.

Comparison of multi-link transmission processes with full redundancy and weak network redundancy

2.2.2 Effect

We compared the effects of full dual-transmission, dual-transmission in weak networks, and closed dual-transmission on audio and video quality, device performance, and power consumption under different weak network conditions.

User Experience

Overall, whether it is full dual-transmission or weak network dual-transmission, the weak network of a single path has little effect on the overall sound quality and end-to-end delay performance. After turning off dual-transmission, the weak network of a single path has a greater impact on the sound quality and delay performance, with a significant decrease in sound quality and a significant increase in delay. Under the same weak network conditions, the end-to-end delay of full dual-transmission is lower than that of weak network dual-transmission.

The impact of full dual transmission, dual transmission with weak network, and single-path weak network on audio and video quality when dual transmission is turned off

The impact of single-path weak network on end-to-end delay when full dual transmission, weak network dual transmission, and dual transmission is disabled

CPU and memory performance

In various network environments, there is not much difference in CPU and memory consumption between full dual-transmission, weak network dual-transmission, and closed dual-transmission.

Power consumption

Whether it is full dual-transmission or weak-network dual-transmission, the power consumption of calls will increase, but the power consumption of weak-network dual-transmission is smaller than that of full dual-transmission, which can reduce power consumption more effectively.

The impact of full dual-transmission, weak network dual-transmission, and closed dual-transmission on mobile phone power consumption

2.2.3 Pros and Cons of the Solution

Compared with full redundant transmission, weak network redundant transmission only starts dual transmission of corresponding data when the main link is detected to be weak, thus saving more traffic; at the same time, since the data transmission on the mobile network is reduced, power consumption is correspondingly reduced. However, since it takes time to judge the weak network itself, the accuracy of the weak network judgment will also affect the effect. Therefore, the end-to-end delay of weak network dual transmission is slightly larger than that of full dual transmission, but in general real-time audio and video usage scenarios, this delay difference is not much for users to experience.

3. Practice and benefits of multi-link transmission technology

Currently, weak network dual-transmission has been launched in services such as Lark Conference and Douyin Social. The following is the performance of Douyin Social Service after enabling weak network dual-transmission compared with the control group:

It can be seen that after enabling the dual-send strategy for audio in weak networks, the online audio freeze rate has dropped significantly, and the performance indicators are almost unaffected.

4. Future Outlook

The above is the evolution and practice of multi-link transmission technology in Volcano Engine RTC.

However, the two multi-link design methods mentioned above are both multi-link transmission modes implemented at the bottom layer of the connection. This multi-link transmission mode is completely transparent to the upper-layer modules. The upper-layer congestion control, retransmission, FEC, PACER, codec and other modules do not "know" whether the bottom layer is a single-link or multi-link transmission mode, which will cause two problems:

(1) The upper-layer congestion control module is not aware of the underlying links and cannot independently evaluate the bandwidth and congestion of each link, and therefore cannot allocate data more accurately based on the quality of each link.

(2) Redundant data transmission is implemented at the connection layer. The connection layer cannot perceive more attributes of the data and cannot implement richer transmission strategies. For example, if we only want to use the backup link to transmit key frames, FEC and other data, it is difficult to implement this strategy in the above two architectures.

In the future, we will use a more advanced architecture to replace the weak network redundant transmission architecture, and continue to optimize the multi-link transmission strategy for audio and video to minimize the use of additional bandwidth and power consumption, and maximize the user's audio and video call experience.

Join us

Volcano Engine RTC is committed to providing high-quality, low-latency real-time audio and video communication capabilities across the global Internet, helping developers quickly build rich scenario functions such as voice calls, video calls, interactive live broadcasts, and retweeted live broadcasts. It currently covers rich real-time audio and video interactive scenarios such as interactive entertainment, education, conferences, games, automobiles, finance, IoT, etc., serving hundreds of millions of users.