Sina Weibo Hou Qinglong: Weibo LNMP architecture in the new era

【51CTO.com original article】Just last week, the WOTA Global Architecture and Operation Technology Summit hosted by 51CTO was grandly held at the Beijing Renaissance Hotel. This WOTA set up 15 cutting-edge hot technology forums, and 60+ technical experts from first-tier Internet companies at home and abroad such as Google, LinkedIn, Airbnb, Baidu, Alibaba, Tencent, and Kingsoft brought more than 50 practical architecture experiences and successful experience sharing cases, and worked together to create a two-day industry-leading technology event.

At the A conference venue of high-availability architecture on the afternoon of the ***th day, Hou Qinglong, the head of R&D of Sina Weibo main site, delivered a speech entitled "Weibo LNMP architecture in the new era". After the speech, the reporter interviewed Hou Qinglong, who shared with the reporter some of his and Sina Weibo's technical team's deployment experience on the LNMP architecture in the new era, as well as some challenges encountered in the new era. In addition, he also introduced the ideas and gains of Sina Weibo's LNMP platform during development from the perspective of elasticity.

Use new ideas to avoid the shortcomings of traditional architecture

As an important social platform, Sina Weibo often encounters some emergencies, and massive forwarding brings great challenges to the service architecture. Traditional practices have some shortcomings, such as the long application cycle for traditional equipment procurement, cumbersome expansion and contraction, and high equipment operating costs. When facing traffic pressure, the conventional practice is that IT equipment will be partially redundant, but not completely redundant, after all, cost issues need to be considered. Hou Qinglong takes the CPU as an example. Under normal circumstances, the CPU utilization rate may be in the range of 20% to 30%, which is a normal state. Sina has an internal requirement that each server CPU must run to about 40% before it is considered idle. But if the CPU runs to 60%, the technical team may need to consider capacity expansion.

In the face of traffic pressure, another common practice is to downgrade services, shutting down those less important functional modules one by one to ensure that the most important functions run smoothly. However, the disadvantage of doing so is that in the most serious cases, many modules of Weibo will no longer be displayed, and the user experience will be very bad.

Under such circumstances, Sina Weibo's technical team began to think about how to reduce equipment operating costs while enhancing the elastic expansion deployment of the business. Hou Qinglong told reporters that Sina Weibo finally chose the PHP elastic expansion deployment solution based on the hybrid cloud platform and built a DCP platform, which can achieve elastic scheduling of the business and cross-cloud operation of the infrastructure, which effectively solved the problem of sudden traffic.

Previously at the conference, a participant asked a question: If the traffic suddenly surges, there will be a certain time difference in temporary capacity expansion. How to deal with it to ensure lossless service? Hou Qinglong said that it is necessary to make advance judgments at this time. If the CPU is on the verge of danger at 60% of the operating state, then capacity expansion should be done in advance at 50%. If time is tight, downgrade first, because it is more convenient for us to downgrade. Many levels in the background can be completed by just pressing a button. Prioritize a downgrade and restore it immediately after the expansion is completed to shorten the downgrade time as much as possible.

The value of DCP

So, how does Sina's DCP platform solve the problems of elastic expansion and cross-cloud?

Hou Qinglong told reporters that elastic expansion is to package all environmental differences through containerized images, and try to make the underlying host transparent. Why do we do this? Because the previous deployment of PHP servers and Java servers had very different operating system choices and business environment deployments, and they could not be used even in a redundant state. However, through the DevOps Community technology, all these technologies are distributed to the image, and the same Docker engine is deployed. When Sina Weibo needs to deploy a service, it can quickly download the image and start it immediately.

Cross-cloud infrastructure actually eliminates the differences between hosts, allowing users to no longer worry about how many servers are needed for capacity expansion, thus reducing the workload in the infrastructure link in the past.

In Hou Qinglong's view, the value of DCP lies in organizing all behaviors in a systematic way in advance. When it is necessary to deploy a service or package an image, it can be completed quickly based on the system by pressing buttons, and the details are completely automated.

Another obvious benefit of DCP is that it reduces the difficulty of operation and maintenance, because operation and maintenance personnel no longer need to be proficient in technology, they only need to understand the operating system. All technical things have been hidden in the back end, and what is presented on the front end is an automated operation interface.

Why Docker?

Hou Qinglong revealed that they had considered not using Docker technology but using virtual machines instead. When Sina Weibo was working on a development machine or testing phase, it actually already had a set of pure virtual machines that could be used directly. If it was based on Docker, it would mean that all access methods must be accessed in Docker mode.

However, the *** technical team believed that they hoped that the new service architecture would not be perceptible to the development colleagues and did not want to have two systems internally, because the same configuration could actually be deployed on the elastic expansion platform or on the traditional platform without any difference, so they finally chose Docker technology.

In addition, the new service architecture also encountered some difficulties. Because the code can be deleted at any time during elastic expansion, the process is relatively complicated. The safest way is to deploy in full. However, since PHP code is different from Java code, Java code is packaged locally and the packaged file can be pushed directly. PHP files are completely scattered. The traditional way is to push them through rsync, but after Dockerization, they found that each full deployment needs to be pushed again, even if the code only changes one instruction. Later, after some tests, they found that the image deployment is particularly good. A 1G image may be downloaded in tens of seconds. Full deployment is much faster than expected.

Experience sharing: How to make the architecture flexible?

Hou Qinglong introduced that Sina Weibo's business systems are all developed based on PHP. In the past, when the technical team made images and optimized them, they would mostly deploy them separately and make them into separate images. But when it came time to expand the capacity, the system would run very slowly. The reason was that when editing images in the past, the technical staff would package each image, which was equivalent to each image containing an operating system. Assuming that an operating system was 600 megabytes in size, then even if the image was installed with 100 megabytes of PHP, it would add up to 700 megabytes, which was very slow to download.

Later, they packaged this image into an image package, and all components shared the same operating system, which was much more compact and faster to download. In this way, when deploying new system functions, it may have taken half an hour before, but now it only takes a few minutes.

At the end of the interview, the reporter asked Hou Qinglong about his prediction for the future development trend of service architecture. He said that from a macro perspective, when an enterprise is in the early stages of development, the purpose of service architecture is to do whatever is convenient, but when the enterprise has reached a certain scale, the service architecture needs to consider how to use tools to solve all problems and try to solve problems from an automated perspective. This is his experience sharing, and he hopes it can be helpful to everyone.

[51CTO original article, please indicate the original author and source as 51CTO.com when reprinting on partner sites]