Let's talk about TPC

[[403151]]

In recent days, the PR publicity of OceanBase TPC-H winning the world's first place has been reported by Chinese self-media. In addition to the excitement, the author also thought that based on the contact with the TPC organization and its auditors in recent years, he has gained some understanding of the TPC benchmark and its auditing and release process, and has also heard some anecdotes and interesting stories. So when congratulating OceanBase for winning the championship, today we will change the angle and talk about the TPC benchmark test.

1.Who is doing the TPC-H test?

Let’s start with TPC-H. We can see that mainstream traditional database vendors have stopped publishing TPC-H audit reports in recent years. As a result, TPC-H has increasingly become one of the promotional indicators used by hardware equipment vendors to showcase their server processing capabilities. So today, we can see a very “interesting” scene on the official TPC website [1]. Although it is a benchmark test of database performance, the result list (the so-called “list”) is almost all hardware vendors: the System column is almost all server models, and the test sponsors (Company/Sponsor) are almost all equipment vendors, with the exception of OceanBase and Alibaba Cloud AnalyticDB. Different equipment vendors on the list often choose the same database product for benchmark testing. For example, in TPC-H, the most commonly used product in recent years is Microsoft’s SQL Server Enterprise Edition. Seeing this, we may begin to have some different ideas about the role of TPC-H in the industry.

Why is this the case? First, let's take a look at the TPC-H benchmark. The TPC-H benchmark includes very strict ACID tests and query performance tests, but strictly speaking, it is not a high-concurrency scenario of TP and AP coexisting in HTAP, but a performance test of some relatively simple (compared to TPC-DS) OLAP queries based on ACID verification. Among them, the concurrent queries of AP are generally only 2-11 concurrencies (each concurrency is called a stream in TPC-H), depending on the amount of test data (scale factor), and 22 queries are executed sequentially in each stream. In the entire test, there is only one TP concurrency (stream), and this TP stream only contains about one thousandth of the data written and deleted in the Order and Lineitem tables, which generally only accounts for a short time of each AP stream execution. For example, according to the audit results publicly released by OceanBase this time, the AP stream1 of Performance RUN1 took 1382 seconds, while the TP took less than 7 seconds to complete, which has little impact on most queries in the AP workload. In addition, since the data model of TPC-H is too simple and the data distribution is single, it does not pose a great challenge to traditional database systems. In addition, various database vendors and academia have done a thorough study of the TPC-H benchmark, and have even published papers discussing which queries should be optimized for which tasks many years ago [2]. Therefore, in general, TPC-H does not pose any challenge to traditional database systems, and its promotional effect is also limited. For the "new forces" in the database industry, represented by OceanBase, TPC-H is moderately difficult and comprehensively evaluates the capabilities of database systems, including ACID. Publishing the test results can bring them a certain degree of credibility (we will explain later why it is "certain" rather than "absolute"), and can be used as a part of commercial promotion.

2. How valuable is TPC-H as a reference?

Let's talk about the credibility of the benchmark test results. First of all, TPC has a strict third-party audit system. It is an audit organization that has been cooperating with TPC for many years. The auditors in this organization are all experts with decades of experience in the database field. Some of them are directly involved in the formulation and modification of the benchmark. The entire audit process is very strict. It is said that even the format of the log output required for the audit has strict requirements. It stands to reason that under such a strict audit process, the audit results must be very credible. Yes and No!

There is no problem with the strictness of the audit process, but here TPC requires that most of the TPC benchmark results, including TPC-H, TPC-DS, and TPC-C, be publicly testable and publicly released product versions. In addition, TPC has requirements on which special optimizations are allowed or not allowed. Traditional database vendors have also made many targeted optimizations for these benchmarks, and some optimizations cannot be turned on by default in the system and need to be controlled by switches. Such special optimizations are not recognized and accepted by TPC, so I heard that some traditional database vendors have to withdraw and re-publish a certain TPC result. Back to large-scale distributed database systems, auditors generally do not log in directly to test and verify products in person. They often review test processes and scripts and ask each vendor to provide their own proofs. These proofs include audit logs of test processes, test results, proof of official versions, proof of price of the hardware system used, proof of public sale or provision of services (cloud products), etc. Sometimes these proofs cannot be fully verified, so after each result is released, there will be a three-month public notice period to allow the public (including competitors) to verify. However, few people have the resources to verify the test results of very large clusters, and for cloud products, each manufacturer can also find ways to circumvent restrictions. So in general, the credibility of this result is generally not a problem from a technical perspective, but from the perspective of TPC's requirements for publicly testable products, it is sometimes impossible to fully guarantee it.

Let's take a look at TPC-DS. The first company to be listed in this benchmark test was StarRing, a Chinese company. This was also the first official audit test after the launch of the TPC-DS benchmark. So StarRing actually helped TPC solve many audit process details at that time, which is why StarRing's audit took a long time. Later, Alibaba Cloud's Cloud AnalyticDB and Cloud E-MapReduce products, thanks to the public test scripts and documents of StarRing's audit results, can take fewer detours in the test process. Here's some gossip. I heard that there is internal competition between these two Alibaba Cloud products, so there is a PK on TPC-DS. Multiple audit tests in two years have made the third-party audit company of TPC-DS very prosperous in the past two years. The enthusiasm of Chinese companies for ranking should really move TPC and its audit organization. It is said that the cost of each audit is not cheap. I think this has also made the audit company's business busier and more profitable in the past two years. I even heard that many times it is necessary to make an appointment with the auditor's schedule a long time in advance.

3. Do we still need to compete on the TPC rankings in the cloud era?

So far, we have seen that the TPC-DS test score list is basically occupied by Chinese companies, while there are few foreign companies, especially American companies. There are different reasons for this. First of all, AWS Redshift has excellent TPC-DS default out-of-the-box performance, and its product's market popularity and status do not need to rely on TPC-DS to promote it, just like traditional database vendors rarely conduct audit tests on TPC-H. Emerging cloud data warehouse vendors such as Snowflake have always been very sensitive to being publicly tested (especially by competitors), and their TPC-DS performance has always been relatively "mysterious". I look forward to seeing their standard audit results data in the future. A domestic manufacturer is said to have contacted TPC for an audit in 2019, but because it was on the US Entity Restriction List, TPC had to refund its audit fees and did not continue. So now, in addition to the two products of Alibaba Cloud, StarRing and H3C also conducted audit tests, and their results can be seen on the TPC official website. Among them, StarRing's second submission was still very good, surpassing the first "ranking" results of the two Alibaba Cloud products, but there has been no latest results published by these two Alibaba Cloud products for a year. In an interview article with the person in charge of OceanBase products in a Chinese self-media, the author of the article is also looking forward to the release of OceanBase's TPC-DS audit results. I wonder if it will once again set off a new round of "ranking" enthusiasm for Alibaba Cloud, OceanBase and other domestic database "new forces". Let us wait and see.

Finally, let's look at TPC-C. I personally think that the difficulty and "gold content" of this benchmark is much higher than TPC-H. It is a true measure of the comprehensive capabilities of a TP system. However, like TPC-H, most traditional database vendors have almost stopped auditing and publishing the results of this benchmark for various reasons. Most of them are equipment vendors or integrators. Another reason why traditional vendors no longer "brush up the list" is that the scalability of their architecture has reached a relative bottleneck, which is also an important reason why OceanBase can use its horizontal expansion capabilities to "dominate" the top of the list.

Well, today I simply wrote some random thoughts about TPC. Personally, I think that TPC-H ranking is meaningful, but technically it is of limited significance, and TPC-C and TPC-DS will have higher technical challenges. On the other hand, there are more and more emerging database technologies and products in China. As the basic system software that the country focuses on developing, it is a very good thing for the development of China's database technology. However, we must also see that database technology requires long-term accumulation, precipitation and innovation. We must be careful not to compete for the sake of ranking, which will cause "involution" among domestic manufacturers, but to truly develop new technologies and products that have practical significance and technological breakthroughs for the country and people's livelihood.

References:

[1]TPC, "TPC-H official results," [Online]. Available: http://tpc.org/tpch/results/tpch_last_ten_results5.asp?version=3 .

[2]TN a. OE Peter Boncz, "TPC-H Analyzed: Hidden Messages and Lessons Learned from an Influential Benchmark," [Online]. Available: https://homepages.cwi.nl/~boncz/snb-challenge/chokepoints-tpctc.pdf.

This article is reprinted from the WeChat public account "Yang Jianrong's Study Notes", which can be followed through the following QR code. To reprint this article, please contact Yang Jianrong's Study Notes public account.