Architecture design of MQTT messaging platform for 10 million-level Internet of Vehicles

Preface

With the advancement of the new four trends (electrification, intelligence, networking and sharing) in the entire automotive travel field, various automobile manufacturers are gradually building a car networking system with intelligent driving and intelligent networking as the core. The new generation of car networking systems has put forward higher requirements for the platform architecture of underlying message collection, transmission and processing.

In the previous article of this series, "MQTT Protocol in IoV Scenarios", we have mentioned that MQTT protocol is currently the most suitable communication protocol for building IoV scenario data platforms. Based on this, in this article, we will continue to discuss MQTT message collection and transmission in IoV scenarios, and how to build a 10 million-level IoV MQTT message platform, in order to provide a platform architecture design reference for enterprise users who are currently engaged in IoV business.

The foundation of Internet of Vehicles: message collection and transmission

Evolution of Internet of Vehicles Transmission Protocols

As we all know, vehicle-to-everything (V2X) refers to the interaction between vehicles and clouds, vehicles and networks, vehicles and vehicles, vehicles and roads, vehicles and people, vehicles and sensor devices, etc., and a dynamic mobile communication system that realizes communication between vehicles and public networks. It is a heterogeneous communication network established to meet the efficiency, safety, management and other elements in every link related to the vehicle. The communication protocol running in it has become the key and core of the construction of the vehicle networking system.

In the course of the development of the Internet of Vehicles, there are two main mainstream communication technologies that have promoted the overall development of the Internet of Vehicles:

• DSRC (Dedicated Short Range Communication): Developed by the American Society for Testing and Materials (ASTM) in 1992 for ETC business scenarios, it evolved into the IEEE (802.1X) vehicle networking communication technology standard after years of improvement and iteration. For quite a long time, DSRC technology was the mainstream vehicle networking communication protocol used in the international mainstream automobile production and consumer markets.
• **C-V2X (Cellular Vehicle to Everything): **C-V2X relies on existing cellular base stations. In addition to supporting direct communication of PC5, RSU and vehicles can be connected to the V2X platform through 4/5G channels (using Uu interface) to achieve vehicle-road collaborative communication. Compared with DSRC, C-V2X is technically superior. It enhances the security and confidentiality of communication, supports high network capacity, and can support high bandwidth and large data volume requirements.

The competition between DSRC and C-V2X technologies is very fierce, and both hope to become the mainstream Internet of Vehicles communication standard. At present, my country has the most complete 5G communication network infrastructure, so it is more inclined to adopt C-V2X (LTE-V, 5G-V2X) communication technology, and realize the new generation of Internet of Vehicles architecture based on autonomous driving through the systematic construction of V2X vehicle-road system + single-vehicle intelligent system.

The significance of message platform construction for the Internet of Vehicles

As the construction of the Internet of Vehicles develops rapidly, all OEMs have reached a consensus: the purpose of building the Internet of Vehicles is not to connect for the sake of connecting, nor for the sake of in-vehicle entertainment, but to connect for data. With the Internet of Vehicles, there is data. With data, supplemented by a complete data governance and application system, there is everything.

The target data of this business is not limited to the relevant data on the vehicle side. In the V2X framework, it is necessary to solve the interconnection between vehicles (V2V), vehicles and roads (V2R), vehicles and networks (V2I), vehicles and clouds (V2C), vehicles and people (V2H), etc., to achieve comprehensive data collection and analysis for vehicles, roads, clouds, networks, and people. The C-V2X protocol and communication method based on 5G provide basic capability guarantee for the construction of the entire system.

From traditional OTA applications to smart cockpits, high-precision map adaptation, centimeter-level positioning, long-term connection between vehicle and computer, mobile phone message collection, vehicle-road cloud map, vehicle-road collaboration and many other new intelligent application scenarios, the demand for message platforms and data processing systems in the Internet of Vehicles business has expanded from the original vehicle-cloud to the overall architecture construction of man-vehicle-road-network-cloud, which has also put forward higher requirements for the construction of the entire message platform.

How to build a message communication and transmission system architecture with massive connections, high concurrent throughput, and low latency to ensure the ubiquity, convenience, high availability, reliability, security, and high concurrency of the entire system has become the key to the construction of a new generation of Internet of Vehicles systems based on autonomous driving and vehicle-road collaboration scenarios.

Architecture design of a 10 million-level Internet of Vehicles messaging platform

Next, we will take EMQ's Internet of Vehicles messaging platform and data processing overall solution as an example to introduce how to build a 10 million-level Internet of Vehicles messaging platform.

Business Challenges

(1) Secure access to vehicle computer, road test unit and mobile phone system

The vehicle side needs to cover new vehicle-connected services such as vehicle data reporting, POI distribution, file push, configuration distribution, message push, and operation care. The massive message topics generated need more secure and stable access and transmission to achieve message subscription and publishing. The road side needs to achieve secure access to roadside RSU, message collection and transmission, and map data transmission.

(2) Real-time and reliability of large concurrent message delivery

Application scenarios such as high-precision maps, centimeter-level positioning, and vehicle-road collaboration all require millisecond-level low-latency and highly reliable transmission capabilities for massive vehicle-road map messages. The message processing platform needs to have high performance, low latency, and high reliability to support tens of millions of connections and millions of concurrent business scenarios.

(3) Rich application scenario integration

In the Internet of Vehicles system with autonomous driving as the core, a message platform is needed to connect various applications based on people, roads, maps, and clouds. The vehicle-side data is connected to application scenarios such as high-precision maps, centimeter-level positioning, vehicle-road collaboration, and mobile phone connections through the message platform. The consumption supply of applications is guaranteed through the message platform, and a high-performance, low-latency, and highly reliable data architecture is provided.

(4) Massive data storage, processing and distribution

After massive IoT data from people, cars, roads, clouds, maps, and networks are collected, it is necessary to conduct full life cycle management of the access, storage, processing, and distribution of these large-scale real-time data streams, provide applications with database support for dynamic and continuous data streams, and support applications to make in-depth use of Internet of Vehicles data to serve consumers and make business decisions.

Total Solution

In the solution, we mainly use EMQ's cloud-native distributed IoT access platform - EMQX, to achieve data connection, movement and processing between the vehicle side, the human side and the road side in the Internet of Vehicles system. EMQX's integrated distributed MQTT message service and powerful IoT rule engine can provide a basic capability base for high-reliability, high-performance real-time IoT data movement, processing and integration, helping enterprises quickly build IoT platforms and applications for key businesses.

(1) Message processing on the vehicle side

EMQX uses the MQTT protocol to access the vehicle connection system. The vehicle-side is connected to the EMQX distributed cluster through load balancing. The horizontal expansion capability of EMQX can achieve data communication capabilities of tens of millions of vehicle-side connections and millions of concurrent responses. Through the rule engine, it can achieve massive message bridging message queues, persistent warehousing, offline message storage and other capabilities in one stop, while providing rich API atomic capabilities northbound integration.

In terms of security, EMQX not only supports TLS/DTLS or GMSSL security protocols to ensure system reliability and stability, but also provides multiple protection mechanisms such as heartbeat monitoring, will messages, QoS levels, etc., and realizes real-time, secure and reliable vehicle-machine message communication in a complex network environment through offline message storage.

(2) Message processing for people and roads

EMQX provides a message collection and processing platform for mobile APP, RSU and other terminals for people and roads. Based on the 5G network slicing capability, ultra-low latency traffic information services are achieved through the nearby access of personal terminals and roadside units. Through protocols such as MQTT, the road condition information perceived by people and roadside facilities is pushed to the cloud control platform, and the cloud control platform integrates V2X algorithms to realize intelligent traffic scenarios such as road collaborative perception, safety reminders, and remote collaborative control.

In terms of security, it supports international standard TLS/DTLS encryption or national secret algorithm GMSSL encryption, and ensures the secure communication of the human-vehicle-road information system by expanding the PKI/CA certificate authentication system.

Tens of millions of messages access framework model

For the next generation of Internet of Vehicles scenarios, the reference architecture of EMQ's overall message access and data processing platform with tens of millions of connections and millions of concurrent connections is as follows:

• Business scenario: Vehicles, mobile apps, roadside RSUs and other devices in the Internet of Vehicles system are connected through MQTT, achieving concurrent access capabilities for tens of millions of terminals.
• **System architecture: **Terminal devices are connected through protocols such as MQTT and HTTP, and connected to the distributed messaging platform EMQ X through load balancing components. Distributed multi-cluster deployment meets the needs of tens of millions of concurrent connections. According to the million-level message throughput capacity, the rule engine is connected to the Kafka cluster to realize data forwarding. The Internet of Vehicles service platform, high-precision map service, V2X cloud control service, positioning service and other vehicle network-related applications can directly consume by subscribing to Kafka data. At the same time, EMQ provides three southbound interface services: REST, MQTT and MQ message queue to realize two-way communication of vehicle control (remote control) messages.

Through the above reference framework, EMQ can achieve the business needs of tens of millions of connections and millions of concurrent throughputs in the Internet of Vehicles scenario through the EMQX cloud-native distributed IoT access platform.

Tens of millions of message access tests

Test environment and purpose

A car company plans to verify the following capabilities of the EMQX cluster based on the test environment in the Internet of Vehicles scenario, and prepare the corresponding technical architecture and capability support for subsequent business growth:

• It can support 10 million concurrent connections and 100,000 to 150,000 QoS 0 messages per second with a payload of 100 bytes to be bridged to Kafka through the rule engine;
• 10 million concurrent connections subscribing and consuming OTA broadcast topics;
• 3 million users connecting simultaneously will not cause cluster avalanche, and test the time required for connection.

In addition, after completing all the above tests, we will continue to explore the maximum message sending and bridging throughput that can be supported under the current configuration with 10 million concurrent connections and forwarded to Kafka (increase the client message sending frequency according to the EMQX cluster resource usage), and test the maximum message throughput that meets QoS 2 and an average response time of less than 50 milliseconds under 10 million connections.

Test Preparation

The client connects to the load balancing ELB through TLS encryption, then terminates TLS on the client in HAProxy, and finally connects to the EMQX cluster through TCP. By terminating TLS on HAProxy, the support capability of the EMQX cluster can be improved. In this deployment mode, the processing capability of EMQX is exactly the same as that of the client directly connecting through MQTT TCP. On the other hand, compared with MQTT TCP connection, the client also consumes more resources through TLS connection. The scale of this test is tens of millions, and the number of test machines required is large. In order to reduce the required test resources without affecting the test objectives of the EMQX cluster, this test will directly use TCP connection.

Test scenario

Test Results

The following are the results of this test:

summary

As shown in the above results, under the current deployment architecture, the car company's verification requirements for 10 million concurrent connections + 200,000 messages bridged to Kafka, message broadcasting, and 3 million instantaneous concurrent connections can be met. In the exploratory test, the maximum TPS of 1.2 million messages (QoS 0, payload 1kB) was tested under 10 million connections. The test lasted for 10 hours and the EMQX cluster was stable, the CPU idle was as low as 20%, and the memory usage was stable.

From the above, we can see that EMQX has outstanding performance in supporting tens of millions of connections in the Internet of Vehicles scenario, and its architecture is stable and reliable.

Introduction and use of stress testing tools

Since this test requires a large number of test machines and their management is complex, the commercial version of EMQ's test software XMeter performance testing platform and JMeter-MQTT plug-in are used for this test.

• XMeter is a performance testing platform based on the open source testing tool JMeter. In view of the characteristics of IoT, such as large access scale, elastic expansion requirements, multiple access protocols, and mixed scenarios, XMeter has modified JMeter to support large-scale, high-concurrency performance testing, such as achieving tens of millions of concurrent MQTT connections and message throughput tests. In addition to testing the MQTT protocol, it can also support testing of mainstream applications such as HTTP/HTTPS.
• The JMeter-MQTT plugin is an open source MQTT performance testing plugin implemented by XMeter. It has been used in many projects and is currently the most popular MQTT plugin in the JMeter community.

Final Thoughts

Through this article, we introduced the design of the MQTT message platform architecture for the Internet of Vehicles with tens of millions of concurrent connections based on the cloud-native distributed IoT access platform EMQX, and verified the performance of the architecture in a scenario with tens of millions of concurrent connections, providing a possible design reference for the construction of a message data platform for the Internet of Vehicles system.

As a world-leading IoT data infrastructure software provider, EMQ is committed to building high-performance, low-latency, high-availability and high-reliability products, providing overall solutions for message collection, mobility, processing and analysis for the new generation of Internet of Vehicles systems, and providing infrastructure service guarantees for the autonomous driving and intelligent connected vehicle businesses of vehicle manufacturers, T1 suppliers, aftermarket service providers, travel service companies and government management agencies, realizing the intelligent connection between people, vehicles, roads and clouds.