Build a distributed IM (Instant Messaging) system yourself

I have shared an article "Designing a ***'s message push system" before. Although some pseudo code was posted in the article, some friends hope to share some executable source code directly. It has been so long, it's time to fill the hole.

So I improved some content based on the previous ones. Let’s take a look at the introduction of this project: CIM (CROSS-IM) is an IM (instant messaging) system for developers. It also provides some components to help developers build their own horizontally scalable IM.

With CIM, you can achieve the following requirements:

• IM Instant messaging system.
• Message push middleware for App.
• Message transparent transmission middleware in IoT massive connection scenarios.

The complete source code is hosted on GitHub:

 https://github.com/crossoverJie/cim

This time it mainly involves IM instant messaging, so I specially recorded two video demonstrations (group chat and private chat).

Group Chat

Private Chat

Architecture Design

Let's take a look at the specific architecture design:

• Each component in CIM is built using Spring Boot.
• Netty + Google Protocol Buffer is used to build the underlying communication.
• Redis stores each client's routing information, account information, online status, etc.
• Zookeeper is used for registration and discovery of IM-server services.

The whole system mainly consists of the following modules:

• cim-server, IM server: used to receive client connections, message transparent transmission, message push, etc. Supports cluster deployment.
• cim-forward-route, message routing server: used to handle message routing, message forwarding, user login, user offline, and some operation tools (obtaining the number of online users, etc.).
• cim-client, IM client: a messaging terminal for users, which can be started and communicate with others (group chat, private chat) with one command; some commonly used commands are built-in for easy use.

flow chart

The overall process is relatively simple, the flow chart is as follows:

• The client initiates a login to the Route.
• If the login is successful, the available im-server is selected from Zookeeper and returned to the client, and the login and routing information is saved in Redis.
• The client initiates a long connection to the im-server and keeps the heartbeat after success.
• When the client goes offline, the status information is cleared through Route.

So when we deploy it ourselves, we need to follow these steps:

• Build basic middleware Redis and Zookeeper.
• Deploy cim-server, which is the real IM server. In order to meet performance requirements, it supports horizontal expansion and only needs to be registered with the same Zookeeper.
• Deploy cim-forward-route, which is a routing server. All messages need to pass through it. Since it is stateless, you can also use Nginx proxy to improve availability.
• cim-client is a real user-oriented client; after starting, it will automatically connect to the IM server and you can send and receive messages on the console.

For more usage instructions, please refer to Quick Start.

Detailed Design

Next, we will focus on the specific implementation, such as how group chat and private chat messages flow; IM server load balancing; how services are registered and discovered, etc.

IM Server

Let’s first look at the server; it mainly implements functions such as client online and offline, and message sending.

First, start the service:

Since it is built in Spring Boot, the Netty service needs to be started when the application starts.

From Pipline, we can see that Protobuf encoding and decoding is used (the specific message is analyzed in the client).

Registration Discovery

The horizontal expansion requirements of the IM server need to be met, so cim-server needs to publish its own data to the registration center.

Therefore, after the application is successfully started, its own data needs to be registered in Zookeeper.

The main purpose is to register the current application's ip + cim-server-port + http-port.

The picture above shows two cim-server instances registered in the demonstration environment (since they are on the same server, only the ports are different).

In this way, the client (listening to this Zookeeper node) can know the currently available service information in real time.

Log in

When the client requests the login interface in cim-forward-route (see below for details) and completes the business verification (just like logging into other websites on a daily basis), the client will initiate a long connection to the server, as shown in the previous process:

At this time, the client will send a special message to indicate that it is currently logging in. After receiving it, the server needs to save the client's userID and the current Channel relationship.

It also caches the user's information, namely userID and user name.

Offline

When the client is disconnected, the cached information also needs to be cleared.

At the same time, you also need to call the Route interface to clear related information (see below for the specific interface).

IM Routing

As can be seen from the architecture diagram, the routing layer is a very important part; it provides a series of HTTP services to connect the client and the server. Currently, the main interfaces are as follows:

①Registration interface

Since each client needs to be logged in before it can be used, the first step is naturally to register.

The design here is relatively simple, using Redis directly to store user information; the user information only includes ID and userName.

Just to facilitate the query, the KV in Redis stores a VK in turn, so that both the ID and userName must be unique.

②Login interface

The login here is different from the login in cim-server and has a business nature:

• After a successful login, it is necessary to determine whether it is a repeated login (a user can only run one client).
• After successful login, you need to obtain the service list (cim-server) from Zookeeper and select a service based on a certain algorithm and return it to the client.
• After successful login, you also need to save the routing information, that is, save the service instance assigned to the current user into Redis.

In order to ensure that only one user can log in, the Set in Redis is used to save the login information; using userID as the key, duplicate logins will fail to be written.

Similar to HashSet in Java, it can only save data without duplication.

Getting an available routing instance is also relatively simple:

• First, get all service instances from Zookeeper to make an internal cache.
• Poll to select a server (currently there is only this algorithm, and new ones will be added later).

Of course, before obtaining the service instance in Zookeeper, you naturally need to listen to the node that cim-server previously registered.

The specific code is as follows:

It also monitors the routing nodes in Zookeeper after the application is started, and updates the internal cache once changes occur.

Guava's Cache is used here, which is based on Concurrent HashMap, so the atomicity of clearing and adding cache can be guaranteed.

③Group chat interface

This is a real message sending interface. The effect is that when one client sends a message, all other clients can receive it!

The process is definitely that the client sends a message to the server. After receiving it, the server traverses all Channels in the SessionSocketHolder introduced above and then sends the message.

The server can be a single machine, but now it is a cluster design. So all clients will be assigned to different cim-server instances according to the previous polling algorithm.

Therefore, the routing layer is needed to play a role:

After receiving the message, the routing interface first traverses the relationship between all clients and service instances. The routing relationship is stored in Redis as follows:

Due to the single-threaded nature of Redis, when the amount of data is large; once Keys is used to match all cim-route:* data, Redis will be unable to process other requests.

So here we use the Scan command to traverse all cim-route:*. Then we call the HTTP interface of each client's server one by one to push messages.

The implementation in cim-server is as follows:

After receiving the message, cim-server will query the internal cache for the channel of the userID, and then just send the message.

④Online user interface

This is an auxiliary interface that can query the current online user information.

The implementation is also very simple, that is, just query the one that saved the "user login status" before and reset it.

⑤Private chat interface

The reason why getting online users is an auxiliary interface is that it is actually used to assist private chats.

Generally speaking, before we can use private chat, we must know which users are currently online, and then you will know who you want to chat with privately.

Something like this:

In our scenario, the prerequisite for private chat is to obtain the userID of the online user.

Therefore, after receiving the message, the private chat interface needs to query the cim-server instance information where the recipient is located, and the subsequent steps are the same as group chat. Call the HTTP interface of the instance where the recipient is located to send the message.

The only difference is that group chats traverse all online users, while private chats only send one.

⑥Offline interface

Once the client goes offline, we need to delete some information previously stored in Redis (routing information, login status).

IM Client

Some of the logic in the client has actually been discussed above.

Log in

The first step is to log in. You need to call the login interface of Route at startup, obtain the cim-server information and then create a connection.

During the login process, the Route interface will determine whether it is a repeated login. If it is a repeated login, the program will be exited directly.

The next step is to create a connection using the cim-server instance information (ip+port) returned by the Route interface.

The first step is to send a login message to the server to let it maintain the relationship between the client and the Channel.

Custom Protocol

The login messages and real message messages mentioned above can be distinguished in our custom protocol.

Since Google Protocol Buffer is used for encoding and decoding, let's take a look at the original format first.

In fact, there are currently three fields in this protocol:

• requestId can be understood as userId.
• reqMsg is the actual message.
• type is the message category mentioned above.

There are currently three main types, corresponding to different businesses:

Heartbeat

In order to maintain the connection between the client and the server, a heartbeat needs to be automatically sent every once in a while if no message is sent.

The current strategy is to send a heartbeat packet to the server every minute:

In this way, the server will receive a Ping heartbeat packet every minute if it does not receive any business message:

Built-in commands

The client also has some basic commands built in for ease of use.

For example, entering :q will exit the client and shut down some system resources.

When you enter :olu (short for onlineUser), the Route will be called to obtain all online user interfaces.

Group Chat

The use of group chat is very simple. You only need to enter a message in the console and press Enter. Then the group chat interface of Route will be called.

Private Chat

The same is true for private chats, but the prerequisite is that you need to trigger the keyword; use userId;; message content in this format to send a message to a certain user, so it is generally necessary to use the :olu command to obtain all online users before it is convenient to use.

Message callback

In order to meet some customized requirements, such as the need to save messages, the client will call back an interface after receiving the message, in which the implementation can be customized.

Therefore, a Caller Bean is created first. This Bean contains a CustomMsgHandleListener interface. If you need to handle it yourself, you only need to implement this interface.

Customize the interface

Since I am not very good at writing interfaces, but I am sure that other experts can write them, the group chat, private chat, online user acquisition, message callback and other services (as well as subsequent services) in the client are all provided in the form of interfaces.

It is also convenient for later page integration. You only need to adjust these interfaces; you don’t need to worry about the specific implementation.

Summarize

Cim is currently only the latest version, with many bugs and few functions (only a few group members were invited to do the test); however, it will be improved later. At least this version will bring some ideas to friends who have no relevant experience.

Next steps: