Morphling: How to maximize cost reduction when deploying AI in cloud native?

With the vigorous development of cloud-native technology and its increasingly mature industry implementation, machine learning on the cloud is rapidly advancing towards large-scale and industrialized development.

Recently, Morphling, as one of the independent sub-projects of Alibaba's open source KubeDL, has become a Cloud Native Computing Foundation (CNCF) Sandbox project. It aims to provide automated deployment configuration tuning, testing, and recommendations for large-scale industrial deployment of machine learning model inference services. In the context of increasingly mature GPU virtualization and reuse technologies, it helps companies fully enjoy the advantages of cloud native, optimize the performance of online machine learning services, reduce service deployment costs, and efficiently solve the performance and cost challenges of machine learning in actual industrial deployment. In addition, the academic paper related to the Morphling project, "Morphling: Fast, Near-Optimal Auto-Configuration for Cloud-Native Model Serving", was accepted by the ACM Symposium on Cloud Computing 2021 (ACM SoCC 2021).

Morphling is a character from the game Dota, which can flexibly change its form to optimize combat performance according to environmental requirements. Through the Morphling project, we hope to achieve flexible and intelligent deployment configuration changes for machine learning inference jobs, optimize service performance, and reduce service deployment costs.

background

The workflow of machine learning on the cloud can be divided into two parts: model training and model serving. After offline training, tuning and testing, the model will be deployed as an online application in the form of a container to provide users with uninterrupted high-quality reasoning services, such as target object recognition in online live videos, online language translation tools, and online image classification. For example, Alibaba's internal Taobao content social platform Machine Vision Application Platform (MVAP) supports Taobao live product highlight recognition, live cover image deduplication, and browsing image and text classification through an online machine learning reasoning engine. According to Intel data, the era of large-scale reasoning ("Inference at Scale") is coming: by 2020, the ratio of reasoning to training cycles will exceed 5:1; Amazon data shows that in 2019, Amazon AWS's infrastructure spending on model reasoning services accounted for more than 90% of its total machine learning task spending. Machine learning reasoning has become the key to the implementation and "monetization" of artificial intelligence.

Inference tasks on the cloud

The inference service itself is a special long-running microservice form. With the increasing deployment volume of inference services on the cloud, its cost and service performance have become critical optimization indicators. This requires the operation and maintenance team to perform reasonable configuration optimization on the inference container before deployment, including hardware resource configuration, service operation parameter configuration, etc. These optimized configurations play a vital role in coordinating service performance (such as response time, throughput) and resource utilization efficiency. In practice, our tests found that different deployment configurations can lead to a difference of up to ten times in throughput/resource utilization.

Relying on Alibaba's extensive experience in AI inference services, we first summarized the inference business. Compared with the configuration of traditional service deployment, it has the following characteristics:

Using expensive graphics card resources, but low video memory usage: The development and maturity of GPU virtualization and time-sharing multiplexing technology give us the opportunity to run multiple inference services on a single GPU at the same time, significantly reducing costs. Unlike training tasks, inference tasks use a well-trained neural network model to process user input information through a neural network to obtain output. The process only involves the forward propagation of the neural network, and the demand for video memory resources is low. In contrast, the training process of the model involves the backward propagation of the neural network, which requires the storage of a large number of intermediate results, and puts much more pressure on the video memory. Our large amount of cluster data shows that allocating an entire graphics card to a single inference task will cause a considerable degree of resource waste. However, how to choose the right GPU resource specifications for inference services, especially incompressible video memory resources, has become a key problem.
Performance resource bottlenecks are diverse: In addition to GPU resources, reasoning tasks also involve complex data pre-processing (processing user input into parameters that meet model input) and result post-processing (generating data formats that meet user cognition). These operations are usually performed using the CPU, and model reasoning is usually performed using the GPU. For different service businesses, GPUs, CPUs, and other hardware resources may become the dominant factors affecting service response time, thus becoming resource bottlenecks.
In addition, the configuration of container operating parameters has also become a dimension that business deployment personnel need to tune: in addition to computing resources, container runtime parameters will also directly affect service RT, QPS and other performance, such as the number of concurrent threads running in the container service, the batch processing size of the inference service, etc.

Optimizing the inference service deployment configuration

Cloud native technologies, with Kubernetes as the mainstream, are being widely used in new application loads in a variety of forms. Building machine learning tasks (including training and reasoning) on ​​Kubernetes and achieving stable, efficient, and low-cost deployment have become the focus and key for major companies to promote AI projects and cloud services. The industry is still constantly exploring and experimenting with the configuration of reasoning containers under the Kubernetes framework.

The most common mode is to manually configure parameters based on human experience, which is simple but inefficient. The actual situation is often that service deployers, from the perspective of cluster managers, tend to configure more resource redundancy to ensure service quality, choosing to sacrifice stability over efficiency, resulting in a large waste of resources; or directly use the default values ​​for operating parameters, losing performance optimization opportunities.
Another option is to further refine and optimize resource allocation based on the historical resource water level profile. However, our observations and practices have found that daily resource water levels cannot reflect the traffic peak during service stress testing and cannot evaluate the upper limit of service capacity. Secondly, for newly launched businesses, there is generally a lack of reliable historical water level information for reference. In addition, due to the characteristics of the machine learning framework, the historical usage of GPU video memory usually cannot correctly reflect the actual demand of the application for video memory. Finally, for the tuning of program running parameters inside the container, there is a lack of sufficient data support from the perspective of historical data.
In general, although the Kubernetes community has some research and products on automated parameter recommendations for more general hyperparameter tuning, the industry lacks a cloud-native parameter configuration system that is directly targeted at machine learning inference services.

Relying on Alibaba's extensive experience in AI inference services, we have found that the pain points of inference service configuration and tuning are:

Lack of framework for automated performance testing and parameter tuning: Iterative manual adjustment of configuration - service stress testing brings huge manual burden to deployment testing, making this direction an impossible option in reality.
Stable and non-intrusive service performance testing process: Directly deploying and testing online services in a production environment will affect user experience.
Requirement for efficient parameter combination tuning algorithm: Considering the increase in the number of parameters that need to be configured, the combined optimization configuration of multi-dimensional parameters is jointly debugged, which puts forward higher efficiency requirements for the tuning algorithm.

Morphling

To address the above challenges, Alibaba's cloud-native cluster management team developed and open-sourced a Kubernetes-based machine learning inference service configuration framework, Morphling, which automates the entire process of parameter combination tuning and combines it with efficient intelligent tuning algorithms to enable the configuration tuning process of the inference business to run efficiently on Kubernetes, solving the performance and cost challenges of machine learning in actual industry deployment.

Morphling abstracts the parameter tuning process at different levels of cloud native abstraction, providing users with a simple and flexible configuration interface, and encapsulating the underlying container operations, data communication, sampling algorithms, and storage management in the controller. Specifically, Morphling's parameter tuning-performance stress testing uses an experiment-trial workflow.

Experiment is the most user-friendly abstraction layer. Through interaction, users can specify the storage location of the machine learning model, the configuration parameters to be tuned, the upper limit of the number of tests, etc., and define a specific parameter tuning job.
For each parameter tuning job experiment, Morphling defines another layer of abstraction: trial. Trial encapsulates a performance test process for a specific parameter combination, covering the underlying Kubernetes container operations: In each trial, Morphling configures and starts the inference service container according to the test parameter combination, detects the availability and health status of the service, and performs stress testing on the service to measure the service performance of the container under the configuration, such as response time delay, service throughput, resource utilization efficiency, etc. The test results will be stored in the database and fed back to the experiment.
Morphling uses an intelligent hyperparameter tuning algorithm to select a small number of configuration combinations for performance testing (trial), and uses the results of each round of testing as feedback to efficiently select the next set of parameters to be tested. In order to avoid exhaustive sampling of specification points, we use Bayesian optimization as the internal core driver of the portrait sampling algorithm. By continuously refining the fitting function, we provide near-optimal container specification recommendation results with a low sampling rate (<20%) stress testing overhead.

Through this iterative sampling and testing, optimized configuration combination recommendations are finally fed back to business deployment personnel.

At the same time, Morphling provides a management and control suite: Morphling-UI, which allows the business deployment team to initiate inference business configuration tuning experiments, monitor the tuning process, and compare the tuning results through simple and easy-to-use operations on the interface.

The practice of Morphling in Taobao content social platform

Alibaba's rich online machine learning reasoning scenarios and a large number of reasoning service instance requirements provide first-hand implementation practice and test feedback for Morphling's implementation verification. Among them, the Alibaba Taobao content social platform Machine Vision Application Platform (MVAP) team supports Taobao live broadcast product highlight recognition, live broadcast cover image deduplication, browsing image and text classification and other services through online machine learning reasoning engines.

During the Double Eleven shopping festival in 2020, we used Morphling to perform specification testing and optimization on the AI ​​inference container to find the optimal solution between performance and cost. At the same time, the algorithm engineering team conducted targeted model quantification and analysis on these resource-intensive inference models, such as the Taobao video viewing service, and optimized them from the perspective of AI model design, supporting the peak traffic of Double Eleven shopping festival with minimal resources while ensuring that the performance of the business did not decline, greatly improving GPU utilization and reducing costs.

Academic Exploration

In order to improve the efficiency of the inference service parameter tuning process, Alibaba Cloud Native Cluster Management Team, based on the characteristics of the inference business, further explored the use of meta-learning and few-shot regression to achieve a more efficient and low-sampling cost configuration tuning algorithm to meet the actual industry's "fast, small sample sampling, low test cost" tuning requirements, as well as a cloud-native and automated tuning framework. The related academic paper "Morphling: Fast, Near-Optimal Auto-Configuration for Cloud-Native Model Serving" was accepted by ACM Symposium on Cloud Computing 2021 (ACM SoCC 2021).

In recent years, topics related to the optimized deployment of AI reasoning tasks on the cloud have been active in major academic journals and conferences related to cloud computing and systems, and have become a hot topic for academic exploration. The topics explored mainly include dynamic selection of AI models, dynamic expansion and contraction of deployment instances, traffic scheduling for user access, and full utilization of GPU resources (such as dynamic model loading and batch size optimization). However, this is the first time that research has been conducted on the issue of optimizing container-level reasoning service deployment based on large-scale industry practices.

In terms of algorithms, performance tuning is a classic hyper-parameter tuning problem. Traditional hyper-parameter tuning methods, such as Bayesian optimization, are difficult to deal with tuning problems with high dimensions (multiple configuration items) and large search spaces. For example, for AI reasoning tasks, we perform "combination optimization" hyper-parameter tuning in four dimensions (configuration items): number of CPU cores, GPU memory size, batch size, and GPU model. Each configuration item has 5 to 8 optional parameters. In this way, the parameter search space in the combination case is as high as more than 700. Based on our accumulated experience in testing production clusters, for an AI reasoning container, each test of a set of parameters takes several minutes from launching the service, stress testing, to data reporting; at the same time, AI reasoning services are of various types, with frequent updates and iterations, limited deployment engineers, and limited test cluster costs. In order to efficiently test the optimal configuration parameters in such a large search space, new challenges have been posed to the hyper-parameter tuning algorithm.

In this paper, our core observation is that for different AI reasoning services, the impact of various configurations that need to be optimized (such as GPU memory and batch size) on the service performance of the container (such as QPS) is "stable and similar", which is reflected in the visualized "configuration-performance" surface. For different AI reasoning instances, the shapes of the "configuration-performance" surfaces are similar, but the degree of impact of the configuration on performance and the key nodes are different in terms of values:

The figure above visualizes the impact of the two-dimensional configuration of <CPU core count, GPU memory size> on the container service throughput RPS of three AI reasoning models. The paper proposes to use Model-Agnostic Meta-Learning (MAML) to learn these commonalities in advance and train a meta-model, so as to quickly find the key nodes in the surface for new AI reasoning performance tests, and make accurate fits under small samples (5%) based on the meta-model.

Summarize

Morphling is based on the Kubernetes machine learning inference service configuration framework, combined with the "fast, small sample sampling, low testing cost" tuning algorithm, to achieve an automated, stable and efficient AI inference deployment tuning process for cloud native, enabling faster optimization and iteration of the deployment process, and accelerating the launch of machine learning business applications. The combination of Morphling and KubeDL will also make the configuration tuning experience of AI from model training to inference deployment smoother.