Exploring the core idea of ​​the Reactor network model

In the network programming series, we implemented a network framework based on epoll, and developed a simple HTTP service based on it. In that series, we used two buffers, read and write, to separate network IO and data reading and writing. The switching between them is completely notified by epoll events. If you have studied the source code carefully, you will find that all operations on network IO are triggered by events. This event-triggered network model is usually called the Reactor network model.

Since the code implementation in the network programming series is relatively complex and difficult to explain clearly, I decided to publish a few separate articles to expand on that series, mainly involving network programming ideas and performance testing.

In this article, we implement a simple network framework to illustrate the general idea of ​​Reactor network model implementation. Its essential idea is basically the same as that of the x-net project, but the code has been greatly simplified, which makes it much easier to understand.

First, let's look at a piece of code

 #include <sys/socket.h> #include <errno.h> #include <netinet/in.h> #include <stdio.h> #include <string.h> #include <unistd.h> #include <sys/epoll.h> int main() { int sockfd = socket(AF_INET, SOCK_STREAM, 0); struct sockaddr_in servaddr; memset(&servaddr, 0, sizeof(struct sockaddr_in)); servaddr.sin_family = AF_INET; servaddr.sin_addr.s_addr = htonl(INADDR_ANY); servaddr.sin_port = htons(2048); if (-1 == bind(sockfd, (struct sockaddr*)&servaddr, sizeof(struct sockaddr))) { perror("bind fail"); return -1; } listen(sockfd, 10); printf("sock-fd:%d\n", sockfd); int epfd = epoll_create(1); struct epoll_event ev; ev.events = EPOLLIN; ev.data.fd = sockfd; epoll_ctl(epfd, EPOLL_CTL_ADD, sockfd, &ev); struct epoll_event events[1024] = {0}; while(1) { int nready = epoll_wait(epfd, events, 1024, -1); int i = 0; for (i = 0; i < nready; i++) { int connfd = events[i].data.fd; if (events[i].events & EPOLLIN && sockfd == connfd) { struct sockaddr_in clientaddr; socklen_t len = sizeof(clientaddr); int clientfd = accept(sockfd, (struct sockaddr*)&clientaddr, &len); ev.events = EPOLLIN | EPOLLET; ev.data.fd = clientfd; epoll_ctl(epfd, EPOLL_CTL_ADD, clientfd, &ev); printf("clientfd: %d\n", clientfd); } else if (events[i].events & EPOLLIN) { char buffer[10] = {0}; int count = recv(connfd, buffer, 10, 0); if (count == 0) { printf("discounnect\n"); epoll_ctl(epfd, EPOLL_CTL_DEL, connfd, NULL); close(i); continue; } send(connfd, buffer, count, 0); printf("clientfd: %d, count: %d, buffer: %s\n", connfd, count, buffer); } } } }

People who are familiar with epoll should be familiar with the above code. The core of this code is in the while main loop below. If it is the current Server's Socket, it means that a new connection has come in. Call accept to get the client's fd, put it in the epoll's events, and register the EPOLLIN event, which we generally understand as a readable event.

If it is not sockfd, it means that the client's fd is readable. We read the data and send it back as is.

The main problem with the above code is that we write the accept and read and write operations of the socket directly in the main loop, which will make the logic of the code difficult to understand.

For a socket, the most direct operations are reading and writing. Therefore, the easiest thing to think of is to separate reading and writing. In order to achieve the separation of reading and writing, we encapsulate two callback functions as follows:

 int recv_callback(int fd, char *buffer, int size); int send_callback(int fd, char *buffer, int size);

You can think about how to write these two functions. The following is the code that encapsulates reading and writing in the recv_callback and send_callback functions according to the original logic:

 int recv_callback(int fd, char *buffer, int size) { int count = recv(fd, buffer, size, 0); send_callback(fd, buffer, count, 0); return count; } int send_callback(int fd, char *buffer, int size) { int count = send(fd, buffer, size, 0); return count; }

Then, in the main loop you can use

 int main() { ... while(1) { int nready = epoll_wait(epfd, events, 1024, -1); int i = 0; for (i = 0; i < nready; i++) { int connfd = events[i].data.fd; if (events[i].events & EPOLLIN && sockfd == connfd) { ... } else if (events[i].events & EPOLLIN) { char buffer[10] = {0}; int count = recv_callback(fd, buffer, 10); if (count == 0) { printf("disconnect\\n"); epoll_ctl(epfd, EPOLL_CTL_DEL, connfd, NULL); clise(i); continue; } } } } }

Although we split read and write into two methods, read and write are not separated. In recv_callback, we call send_callback every time we receive data and send the data back as is. Here we hope that recv_callback and send_callback can manage their own things without interfering with each other, such as the following

 int recv_callback(int fd, char *buffer, int size) { int count = recv(fd, buffer, size, 0); return count; } int send_callback(int fd, char *buffer, int size) { int count = send(fd, buffer, size, 0); return count; }

But this is obviously problematic. After reading in recv_callback, how to send data? Here, we can think about what parts are around a socket? Can we design a dictionary-like structure, where the key of the dictionary corresponds to the socket, and the value corresponds to the various components related to the socket.

We put recv_callback and send_callback in a conn_channel structure and designed two buffers, one for reading data and the other for sending data. conn_channel is the value corresponding to this dictionary. The code is as follows:

 #define BUF_LEN 1024 typedef int(*callback)(int fd); struct conn_channel { int fd; callback recv_call; callback send_call; char wbuf[BUF_LEN]; int wlen; char rbuf[BUF_LEN]; int rlen; };

Among them, fd represents the current client socket. Then we define an array to represent the mapping relationship between socket and socket value. The code is as follows:

 struct conn_channel conn_map[1024] = {0};

In this way, we can add the corresponding socket to conn_map in the main loop as follows:

 int main() { ... while(1) { int nready = epoll_wait(epfd, events, 1024, -1); int i = 0; for (i = 0; i < nready; i++) { int connfd = events[i].data.fd; if (events[i].events & EPOLLIN && sockfd == connfd) { struct sockaddr_in clientaddr; socklen_t len = sizeof(clientaddr); int clientfd = accept(sockfd, (struct sockaddr*)&clientaddr, &len); ev.events = EPOLLIN; ev.data.fd = clientaddr; epoll_ctl(epfd, EPOLL_CTL_ADD, clientfd, &ev); conn_map[clientfd].fd = clientfd; conn_map[clientfd].rlen = 0; conn_map[clientfd].wlen = 0; conn_map[clientfd].recv_call = recv_callback; conn_map[clientfd].send_call = send_callback; memset(conn_map[clientfd].rbuf, 0, BUF_LEN); memset(conn_map[clientfd].wbuf, 0, BUF_LEN); printf("clientfd:%d\\n", clientfd); } else if (events[i].events & EPOLLIN) { ... } } } }

In the above code, every time a client socket is accepted, we put it into conn_map and set the read/write buffer and callback function. But if you are careful, you will find that the callback function signatures in recv_callback, send_callback and conn_channel are different. So, we need to adjust the implementation of these two functions. The adjusted code is as follows:

 int recv_callback(int fd) { int count = recv(fd, conn_map[fd].rbuf + conn_map[fd].rlen, BUF_LEN - conn_map[fd].rlen, 0); // do something memcpy(conn_map[fd].wbuf, conn_map[fd].rbuf, conn_map[fd].rlen); conn_map[fd].wlen = conn_map[fd].rlen; conn_map[fd].rlen = 0; return count; } int send_callback(int fd) { int count = send(fd, conn_map[fd].wbuffer, conn_map[fd].wlen, 0); return count; }

Because of conn_map, the buffer and size passed in are no longer needed, as they are already recorded in conn_channel. So only one fd parameter is needed. We simulated the reply message in recv_callback and forced the read data to be written to wbuffer. I would like to add that rbuffer in conn_channel is used to read data from the socket, and wbuffer represents the data to be sent to the socket.

You can try to run the above code, and then you will find that it does not work as expected, and the send in send_callback does not seem to work. This is because we only write the data from rbuffer to wbuffer, and send_callback has no chance to call. You can think about where to put send_callback?

In the above example, it is more appropriate to execute it in the main loop. In epoll, EPOLLOUT indicates a writable event, and we can use this event. After recv_callback is executed, we register an EPOLLOUT event, and then listen to the EPOLLOUT event in the main loop. In this way, after recv_callback copies the data of rbuffer to wbuffer, send_callback can be executed in the main loop through the EPOLLOUT event.

In order to achieve the above effect, we need to modify two places. One is that we need to register the EPOLLOUT event in recv_callback. The code is as follows:

 int recv_callback(int fd) { int count = recv(fd, conn_map[fd].rbuf + conn_map[fd].rlen, BUF_LEN - conn_map[fd].rlen, 0); // do something memcpy(conn_map[fd].wbuf, conn_map[fd].rbuf, conn_map[fd].rlen); conn_map[fd].wlen = conn_map[fd].rlen; conn_map[fd].rlen = 0; struct epoll_event ev; ev.events = EPOLLOUT; ev.data.fd = fd; epoll_ctl(epfd, EPOLL_CTL_MOD, fd, &ev); return count; }

After copying rbuf to wbuf, we register the EPOLLOUT event for the current fd, and then we handle the EPOLLOUT event in the main loop. The code is as follows:

 int main() { ... while(1) { int nready = epoll_wait(epfd, events, 1024, -1); int i = 0; for (i = 0; i < nready; i++) { int connfd = events[i].data.fd; if (events[i].events & EPOLLIN && sockfd == connfd) { struct sockaddr_in clientaddr; socklen_t len = sizeof(clientaddr); int clientfd = accept(sockfd, (struct sockaddr*)&clientaddr, &len); ev.events = EPOLLIN; ev.data.fd = clientaddr; epoll_ctl(epfd, EPOLL_CTL_ADD, clientfd, &ev); conn_map[clientfd].fd = clientfd; conn_map[clientfd].rlen = 0; conn_map[clientfd].wlen = 0; conn_map[clientfd].recv_call = recv_callback; conn_map[clientfd].send_call = send_callback; memset(conn_map[clientfd].rbuf, 0, BUF_LEN); memset(conn_map[clientfd].wbuf, 0, BUF_LEN); printf("clientfd:%d\\n", clientfd); } else if (events[i].events & EPOLLIN) { int count = conn_map[connfd].recv_call(connfd); printf("recv-count:%d\\n", count); } else if (events[i].events & EPOLLOUT) { // 处理EPOLLOUT事件int count = conn_map[connfd].send_call(connfd); printf("send-count:%d\\n", count); } } } }

It should be noted that epfd is defined in the main function, and we use it in recv_callback, so we can temporarily declare epfd as a global variable and put it outside.

There is a problem with the above code. After the EPOLLOUT event is triggered, you will find that there is no response when sending data to the current fd. This is because the epoll event has been modified by us. To solve this problem, we can set it back after send_callback is executed, as follows:

 int send_callback(int fd) { int count = send(fd, conn_map[fd].wbuffer, conn_map[fd].wlen, 0); struct epoll_event ev; ev.events = EPOLLIN; ev.data.fd = fd; epoll_ctl(epfd, EPOLL_CTL_MOD, fd, &ev); return count; }

In this way, we shield the IO operation. In the main loop, we only focus on events. Different events call different callback functions. In the corresponding callback function, we only do what we should do, and after that, register the event to notify other callback functions.

However, the above code is not elegant enough. For accept and read events, they are both EPOLLIN events in epoll. Can these two be processed together? The answer is yes. First, we need to separate the logic related to accept. The code after separation is as follows:

 int accept_callback(int fd) { struct sockaddr_in clientaddr; socklen_t len = sizeof(clientaddr); int clientfd = accept(fd, (struct sockaddr*)&clientaddr, &len); ev.events = EPOLLIN; ev.data.fd = clientaddr; epoll_ctl(epfd, EPOLL_CTL_ADD, clientfd, &ev); conn_map[clientfd].fd = clientfd; conn_map[clientfd].rlen = 0; conn_map[clientfd].wlen = 0; conn_map[clientfd].recv_call = recv_callback; conn_map[clientfd].send_call = send_callback; memset(conn_map[clientfd].rbuf, 0, BUF_LEN); memset(conn_map[clientfd].wbuf, 0, BUF_LEN); return clientfd; }

We found that the signatures of accept_callback, recv_callback and send_callback are the same, so we can use a union in conn_channel and put accept_callback into conn_channel as follows:

 struct conn_channel { int fd; union { callback accept_call; callback recv_call; } call_t; callback send_call; char wbuf[BUF_LEN]; int wlen; char rbuf[BUF_LEN]; int rlen; };

In the main loop, we can first register the accept callback function for sockfd, and then we only need to keep two logics in the main loop. The code is as follows:

 int main() { int sockfd = create_serv(9000); if (sockfd == -1) { perror("create-server-fail"); return -1; } make_nonblocking(sockfd); epfd = epoll_create1(1); struct epoll_event ev; ev.events = EPOLLIN; ev.data.fd = sockfd; epoll_ctl(epfd, EPOLL_CTL_ADD, sockfd, &ev); struct epoll_event events[1024] = {0}; conn_map[sockfd].rlen = 0; conn_map[sockfd].wlen = 0; conn_map[sockfd].fd = sockfd; conn_map[sockfd].call_t.accept_call = accept_callback; conn_map[sodkfd].send_call = send_callback; memset(conn_map[sockfd].rbuf, 0, BUF_LEN); memset(conn_map[sockfd].wbuf, 0, BUF_LEN); while(1) { int nready = epoll_wait(epfd, events, 1024, -1); int i = 0; for (i = 0; i < nready; i++) { int connfd = events[i].data.fd; if (events[i].events & EPOLLIN) { int count = conn_map[connfd].call_t.recv_call(connfd); printf("recv-count:%d\\n", count); } else if (events[i].events & EPOLLOUT) { int count = conn_map[connfd].send_call(connfd); printf("send-count:%d\\n", count); } } } }

You can think about it, we registered call_t.accept_call, but when calling it, it is call_t.recv_call. Why does this work?

In the network programming series, we abstracted an object for accept. You can compare these two implementations to see what the difference is. Why do we abstract an accepter object in the series?

As you can see, the logic in the final main loop has only two branches, which represent two types of events. This event-driven network model is the Reactor network model. This article simplifies the code for easy understanding. In actual projects, we have to consider many situations. For example, the above code only supports epoll. Can we abstract the event-driven related code into a separate component so that it can support other event models?

Although the code in this article is simple, the implementation of the Reactor network model basically cannot escape this routine. It is just that each part may be encapsulated separately on this basis. For example, we abstracted channels and maps in the network programming series of articles to make it adaptable to various scenarios.

Summarize

The reactor network model is a very important programming concept in network programming. This article attempts to explain the core idea of ​​the reactor network programming model through a short example. Of course, the implementation of this article is not perfect yet. For example, when calling the callback function, the fd is still passed in. Can we not need this parameter and completely separate it from IO?