Using tunneling protocol to achieve transparent communication between different dubbo clusters

[[375332]]

Preface

The author has recently completed a very interesting tunnel mechanism (already running in the production line), which allows dubbo clusters registered with different zookeepers to communicate normally. As shown in the following figure:

For example, in the figure, the two isolated clusters A/B can only communicate through dedicated lines. However, for the applications in them, calling the dubbo service of another cluster (for example, app1 calling app3) is still exactly the same as the original method, without any modification. This feature is very useful for scenarios such as new units (computer rooms) and business network isolation.

This article will briefly talk about this mechanism.

Scenario

This dubbo cluster communication mechanism can be used in the following scenarios.

New machine room

When we build a new computer room, normally, we need to deploy all applications and related facilities of a whole link to the new computer room. As shown in the following figure:

In the author's new mechanism, if the cluster does not have a corresponding interface, it will look for a cluster with a corresponding interface. Even if some systems are missing, the entire computer room can still work, making the new computer room iterative, which greatly reduces the complexity of building a new computer room.

New business unit

Due to the limitation of rack space in a single computer room or some other reasons, some businesses want to be separated into a separate unit (inside the computer room). However, the business does require a lot of basic services of the original unit. However, the network between different units cannot be connected (security requirements).

If we follow the traditional model, we will have to transform the business system, such as establishing a business gateway to communicate with the basic system. This gateway is obviously time-consuming and labor-intensive and has little technical content. For example, in the business code, the dubbo call is forcibly converted into an http call to the business gateway, as shown in the following figure:

Moreover, every time an interface call is added, it must be converted in the business gateway, the corresponding interface package must be added, and then released. Such a gateway is definitely a pit to maintain! With increasingly stringent security requirements, the requirements for network isolation between different businesses will increase day by day.

The author is engaged in middleware, and firmly believes that a good design is one in which the basic services are transparent to the business and imperceptible to the business. Once the business needs to cooperate with the transformation caused by the infrastructure change, it is undoubtedly very unfriendly and even a failed design.

Fault Isolation

In fact, a large part of the reason why I developed this tunnel mechanism is to isolate faults. For example, I encountered several cases where the business system used Zookeeper improperly and wrote a lot of data to Zookeeper, which put the entire cluster at risk of being unavailable. The new mechanism allows different businesses to register with different Zookeepers. If Zookeeper crashes, only this business will crash, and other businesses will not be affected.

In fact, it is not only for Zookeeper, because the author also made a similar tunnel mechanism for messages (such as ActiveMQ). This enables our entire business to better isolate faults!

Tunnel Mechanism

The convenience of this mechanism is that it is less invasive to the business. There is no need to modify the application of the basic cluster. In order to meet this requirement, the author introduced the concept of tunnel, which is very common on the network. You may have come into contact with it in your daily life. Network protocols such as Vxlan all use tunnels.

Tunnel penetration

Let's first look at the most basic principle. When system A calls system B through Dubbo, the dubbo protocol is used in the same cluster. When it crosses clusters, the author carries the dubbo original bit stream on the http protocol and sends it out on a dedicated line.

Since system B sees the same byte stream as what it receives, it cannot (and does not need to) distinguish whether it is a dedicated line or a direct call. Therefore, system B does not need to change any code.

Tunnel Implementation

So, how is this tunnel implemented specifically, and how does system A know that there is no corresponding interface in this cluster and that it needs to call another cluster through the http tunnel? This is where our tunnel gateway comes in.

The concept here is similar to the default gateway on the network. If the corresponding recipient cannot be found in this cluster, it will be delivered to a default gateway, and this tunnel gateway will pass the call on our behalf.

How to discover this gateway

In order to make full use of the registration and discovery mechanism of the dubbo interface, the author also exposes the tunnel gateway as a dubbo interface, and its input and output are as follows:

// Tunnel Gateway Interface Request Body class TunnelInterfaceReq { // dubbo meta information, such as specific call interface information MetaData dubbo // Original request A calls the serialized bit stream byte[] body;} // Tunnel Gateway Interface Return Body class TunnelInterfaceResp { // dubbo meta information MetaData dubbo // Return value calls the serialized bit stream, which is returned by the corresponding system of another cluster byte[] resp;}

With this dubbo interface, we can easily transmit data to the default gateway.

Note that a layer of tunnel protocol is actually implemented here, that is, the dubbo protocol is used to carry the dubbo protocol. This nesting doll-like method effectively utilizes the registration and discovery mechanism of dubbo itself.

Gateways communicate via http

Since different clusters communicate through dedicated lines, the author uses http communication. After App1's request reaches the tunnel gateway, the gateway will take the original body bit stream from TunnelInterfaceRequest and then put it into an http request for transmission. As shown in the following figure:

It is worth noting that since the transmitted data is a byte stream and does not carry any business information (such as type information, etc.), our tunnel gateway can tunnel any dubbo request, unlike the traditional gateway that needs to add jar packages corresponding to various businesses and publish them continuously-_-!

In the figure, after being delivered to the tunnel gateway at the other end, it extracts the call meta information and the original call byte stream from the http protocol, and finds App2 through the call meta information. Then it replays the byte stream to App2, so that dubbo calls can be made. In fact, the byte stream that App2 sees from the tunnel gateway is exactly the same as the byte stream called from other machines in the cluster. As shown in the following figure:

The return value also goes through the tunnel mechanism

Obviously, our return value also needs to go through the tunnel mechanism. Like Request, it will also go through the tunnel protocol twice, as shown in the following figure:

So what App1 actually receives is actually the Tunnel Response. How can it transparently receive the original response bit stream? This requires the caller to access the lightweight jar package developed by the author (in fact, the tunnel of the initial request also requires such a jar package)

Expanding dubbo

Since Dubbo has an excellent filter mechanism, it can be extended in various places. For this tunnel mechanism, the author extended the invoke call logic. As shown in the following figure:

As long as you introduce the jar package written by the author, you can easily perform automatic expansion. Except for adding two lines to pom.xml, other business codes do not need to be modified at all.

Interface discovery for tunnel gateways

So how does tunnel gateway A know that the interface is in cluster B and deliver it to tunnel gateway B? Obviously, we need a cluster communication mechanism between tunnel gateways.

For example, the tunnel gateway may ask other different tunnel gateways whether there is such an interface and cache it according to a certain strategy.

Discovery of dubbo cluster

The last question is how does the tunnel gateway know other dubbo clusters? Compared with the number of dubbo interfaces, the number of clusters is small and does not change frequently. We just need to find a place to simply record it, such as putting it in a database. Then, since it is an http call, load balancing can be done directly by resolving the domain name through DNS.

performance

Since the serialization and deserialization of this mechanism is completely on the Provider/Consumer side, there is no pressure on the gateway at all, so the CPU consumption of the gateway is very low. In terms of single call latency, due to the extra two hops, there is inevitably some loss, about 2ms more for each interface.

Summarize

From the initial conception to the full operation of this mechanism on the production line with minimal performance loss, the author still spent a lot of effort. Seeing such a result, I still feel very accomplished. In fact, this tunnel mechanism borrows concepts from the Internet in many places. It can be said that different technologies can indeed migrate to each other, they just solve essentially the same problems at different levels!

This article is reprinted from the WeChat public account "Bug Solving Road", which can be followed through the following QR code. To reprint this article, please contact the Bug Solving Road public account.