HortonworksTed Yu: China's participation in international exchanges is accelerating

[Original article from 51CTO.com] On November 25-26, 2016, the WOT2016 Big Data Summit was held at the JW Marriott Hotel Beijing. More than 50 front-line experts in the big data field and data technology pioneers gathered at the scene to engage in in-depth exchanges and discussions on cutting-edge technology topics such as data intelligence, big data business, blockchain, real-time computing, system architecture, NoSQL, etc., while sharing the best practices and the most popular industry applications in the big data field.

In the morning KeyNote session of the WOT2016 Big Data Technology Summit, Ted Yu, a senior technical member of Hortonworks and a core contributor to HBase, gave a speech. After the meeting, the reporter interviewed him and asked him to share his technical insights with everyone.

[[177125]]

Ted Yu told reporters that the title of his speech was TinyLFU, which uses many methods to reduce the memory application of original data and can achieve relatively high performance while occupying very little memory.

A technology sharer who is not a homebody at all

Ted Yu speaks very fast, with lots of English technical terms. During his speech, a listener sitting next to the reporter was discussing quietly with others, praising his careful thinking and logic. During the interview, the reporter also found that Ted Yu is very good at talking. As a very active technology developer in the big data industry, Ted Yu is not labeled as a technical geek who is not good at speaking.

Ted Yu explained that this was due to historical reasons. He was exposed to HBase relatively early, and it has been six years now. In recent years, he has returned to China to participate in some industry discussion activities. Through communication, he found that China has grown very fast in the field of big data in recent years. After Hadoop technology was introduced to China, it was sought after by technicians. In addition, domestic technicians have a thorough understanding of open source technology and have developed their own technology, including more than one big data platform. This is a very gratifying phenomenon. "I also hope to bring new trends in big data to everyone when I return to China, and learn from my domestic peers."

Necessary prerequisites for giving back to the community

In the open source community, many people admit that Chinese technicians have little feedback to the community. Ted Yu has a different view. He said that open source projects such as Hadoop and HBase initiated abroad are all in English, whether programming or communication. The English level of domestic technicians is uneven, and some people find it difficult to express their ideas clearly in English, which brings certain limitations. However, Ted Yu also believes that this situation is gradually improving in recent years.

Ted Yu also used Apache HBase as an example. He said that Apache HBase management is very democratic. When someone proposes a technical solution, they need to explain in English what problem the solution is going to solve, how the solution is designed, and what its strengths and weaknesses are. When others look at the proposal, they will combine their own practical experience. Everyone also pays more attention to whether the solution can take into account the needs of others while solving the problem.

"If not, everyone will be more worried about whether the introduction of this technical solution will reduce the overall stability. For mature products like Hadoop, HBase, and Spark, a decrease in stability is generally not allowed." Ted Yu said that this requires programmers to spend a lot of time communicating with the community, and there is a time difference for Chinese programmers, so communication may not be so timely.

Of course, Ted Yu also admitted that language and time difference are secondary issues. The key is to have the confidence to describe technical problems clearly in English. In addition, the code must be written in a standardized way so that it can be accepted by mainstream developers and companies in North America.

Hbase this year and next year

When the reporter asked about the dynamics of Ted Yu's research and development field, Ted Yu said that new data keeps coming in as time goes by. In order to reduce the amplification of compaction i/o, it has now entered production. In addition, on the read path, HBase is written in JAVA, and no matter whether CMS is used, it will face GC tuning problems. There will be different parameters for different workloads. The final solution is to store a lot of data under the JAVA heap. Last year, the main work was on the read path, including Xiaomi and Intel developers working on the write path. Offheap on the write path is obviously difficult. "But I think it is still expected to be solved by next year. The advantage is that if you use HBase to support your business in the background, the stability will be significantly improved."

In addition, Ted Yu also introduced that HBase is stacked under JAVA on the read path, and he is involved in the backup layer. In relational databases, disaster recovery is to back up data regularly, and if a disaster occurs, the backed-up data is returned to the remote cluster. He hopes to submit it to the master branch in December this year.

In an interview with ***, Ted Yu shared his understanding of big data from the perspective of the areas he is interested in. "My personal energy is limited, so I mostly focus on HBase and Phoenix, which are projects directly related to HBase. There are four main projects: HBase, Phoenix, Spark, and Flink. I think these four projects have developed very quickly in 2016. In 2017, Spark will be able to quickly narrow the gap with Hive in terms of SQL query performance, and it is likely to replace Hive in many aspects."

[51CTO original article, please indicate the original author and source as 51CTO.com when reprinting on partner sites]