How is an excellent testing infrastructure created? eBay's Ru Bingsheng talks about the history of testing evolution

[51CTO.com original article] The Global Software and Operation Technology Summit hosted by 51CTO was held in Beijing on May 18-19, 2018. The summit focused on 12 core hot topics such as artificial intelligence, big data, the Internet of Things, and blockchain, and brought together 60 front-line experts from home and abroad. It was a high-end technology feast and a platform that top IT technical talents should not miss to learn and expand their network.

At the "DevOps Transformation Road" session, Ru Bingsheng, technical director of test infrastructure at eBay China R&D Center, delivered a keynote speech on "The Evolution of Test Infrastructure", sharing his experience and insights in designing test infrastructure for large e-commerce websites. After the meeting, 51CTO reporters compiled Ru Bingsheng's speech at the WOT2018 Global Software and Operation Technology Summit.

[[232477]]

The evolution of GUI automation testing frameworks

Ru Bingsheng introduced that eBay is a large e-commerce platform, in which the test infrastructure is closely related to DevOps and is highly integrated with CI/CD (continuous integration and continuous delivery). In the CI/CD process, all calls to the test are made through a unified test execution service, which is used to initiate all test executions, including API testing, GUI testing, and performance testing. During the entire CI/CD process, the initiator does not need to know where the test is running, where the test execution environment is, or how the test is designed. He is only responsible for initiating a test, getting a result synchronously or asynchronously, and then deciding whether the pipeline can proceed. These behaviors are all built based on the test infrastructure.

GUI (Graphical User Interface) automated testing is one of the earliest automated tests. It is a relatively heavyweight test with a low input-output ratio. Therefore, it is usually used for lightweight smoke testing before large e-commerce websites go online to ensure the correctness of all core functions. At the same time, GUI (Graphical User Interface) automated testing has also undergone a legendary change, from a very simple architecture to a large e-commerce that can adapt to global sites. The same set of test scripts can run on sites in different countries around the world.

In the original test diagram, business requirements will be converted into functional requirements, functional requirements will be converted into test requirements, test requirements will have test cases, and test cases will be run in the local test execution environment. The test team will open the website on the local machine for testing. Then the problem comes. Once full regression testing is required, the original method will definitely be inefficient and must rely on automated testing functions. Recording and playback is the initial automation. Tools such as UFT can repeatedly replay scripts after recording. However, the disadvantage is that once there is any change in the interface, the script needs to be modified from the beginning, which is obviously unacceptable.

Modularization is therefore born, which can abstract some repeatable scripts based on the operation level and parameterize them. However, Ru Bingsheng said that in actual operation, it is difficult to determine which scripts are repeatable and how to control the strength of the scripts. Because everyone has different understandings, the definition of reusable scripts varies greatly between teams.

After further development, Ru Bingsheng and his team further evolved the repeatable script into an abstraction of Page Object (page object model), and the automation script became a sub-packaging of the page, with operations based on page elements. Later, they made a layer of Business Flow (business process) abstraction based on the page, so that testers can directly see the business-driven test scripts, which has taken case maintenance to a higher level in terms of operability and readability.

Later, Ru Bingsheng and his team began to try to use Out-of-box Test Data / Golden Data Set test data, and gradually began to implement Flow Branch control based on the Unified Flow Framework. Ru Bingsheng explained that for large e-commerce websites with sites all over the world, each country will have slightly different website functions, which requires the technical team to be able to implement different functional points in the same business process. In the past, five countries had to write five independent scripts, but now only one script is needed for differentiated testing of sites in different countries, which is very helpful for improving the work efficiency of engineers.

Later, they improved the efficiency of Page Object based on Page Encapsulation Code Generator. When a new page or a page is modified, they can dynamically capture all the elements on the page through a very small program. When needed in the future, as long as the elements on the page are available, they can call them. The entire page generation is completed automatically without manual work.

At this stage, the testing capabilities were already very strong, but eBay's testing team was still not satisfied, so they introduced Test Data Service to provide a unified test data preparation service. They provided a complete and unified interface to help testers reduce the complexity of all test data and make testing work more efficient.

The pain of test data + the platform evolution of response strategies

Ru Bingsheng summarized the pain points of test data into five parts.

The first pain point is the time-consuming preparation of on-the-fly data. What is the concept of on-the-fly? Before the test case is implemented, the tester will dynamically generate data in the test script, but if it is very complex data, it will be very time-consuming.

The second pain point is dirty data in out-of-box test data. In scenarios with a large number of test cases, there may be problems with data interference, which will cause a large number of test cases to fail due to dirty data.

The third pain point is the complexity of the test data combination itself. E-commerce websites need to bind different payment methods and express delivery methods. Different countries have different legal requirements. There are many combinations of various parameters, which brings great trouble to the test data.

The fourth pain point is the environmental dependency of test data preparation. For example, to test a certain function, you need to prepare specific data. However, because of microservices, this data is provided by another server. However, various problems may cause the data to fail to be prepared, and as a result, the functional test cannot be completed.

The fifth pain point is the time spent preparing performance test data. eBay has very good practices in this regard. Through Test Data Service, they have greatly increased the success rate and reduced the problem of test data from the original 30% to less than 5%.

The evolution of API automation testing frameworks

Ru Bingsheng introduced that large e-commerce websites usually have tens of thousands of APIs. Due to rapid iteration and online release, there is very little time left for testing. These API tests can only be run in parallel through a large cluster environment. They will introduce a concurrent access controller to control these clusters and tens of thousands of APIs.

After five or six different stages of development, eBay has now completely migrated its API testing to microservices. The company currently has more than a hundred services. If the original API approach is followed, the number of cases will exceed 100,000, and even a cluster cannot run them all. So they changed their strategy and introduced a verification model based on consumer contracts. For example, when B on the A side calls a script, the test system only needs to know who calls it, how to call it, and then test the API calls involved. Next time, only the scripts that have been called before will be tested, which can ensure the quality of the entire module.

Regarding the construction of the test execution environment, Ru Bingsheng took GUI testing as an example. For example, if a tester requires that the GUI test be run on a certain version of a certain browser in a certain operating system, they will first send a request to the Selenium Grid cluster to ask if there is a node with this browser version of this operating system installed in the cluster. If there is, the test system will send it directly to him. If not, the test system will dynamically create one.

The above content is compiled by 51CTO reporter based on the interview with Ru Bingsheng, technical director of test infrastructure technology at eBay China R&D Center, at the WOT2018 Global Software and Operation Technology Summit. For more information about WOT, please visit .com.

[51CTO original article, please indicate the original author and source as 51CTO.com when reprinting on partner sites]