Dewu CDN domain name convergence and multi-vendor disaster recovery optimization practice

Background

Too many CDN domain names cause request fragmentation, leading to the following problems:

TCP connection establishment is frequent and network request performance is poor

The network connection pool resources used to request CDN static resources are limited. Different domain names will create their own TCP connections, which will compete for TCP connection pool resources, resulting in frequent TCP connection interruptions. Re-initiating a network request requires re-establishing a TCP connection, which increases the time spent on the connection establishment phase (including: DNS resolution, TCP handshake, TLS handshake), resulting in an increase in the total time spent.

Too many domain names, high daily maintenance costs

Too many domain names increase the complexity of domain name management, performance monitoring, performance optimization, and online changes, and increase labor and operation and maintenance costs.

For example, the Dewu IPv6 upgrade project and the TLS1.3 protocol upgrade project both require multiple online change processes to be executed in batches by domain name (including: test regression, change application, change review, change verification, and performance monitoring).

Some domain names are not named in a standardized manner and may be at risk of being taken offline

Due to historical reasons, there are many domain names that do not conform to the existing new domain name specifications (such as: xxx.poizon.com, xxx.dewu.com). Domain names that are not owned by Dewu are at risk of being forced offline. For example, the old domain name offline project has invested a lot of manpower costs for transformation.

DNS resolution of IP addresses is frequent, and Alibaba Cloud HttpDNS service costs are high

Too many domain names increases the frequency of calling Alibaba Cloud HttpDNS to resolve the corresponding IP address when establishing a TCP connection for each domain name. The increased number of resolutions leads to high HttpDNS service costs.

In order to solve the problem caused by too many CDN domain names, we decided to perform CDN domain name convergence optimization on the client side.

2. CDN domain name convergence

2.1 Convergence Thoughts

Let's first think about a question: How can we converge multiple CDN domain names into a unified domain name on the client side without affecting the business?

The following points should be considered:

• The server side does not need to make any changes and can still send multiple CDN domain name URLs in the existing way.
• The client side implements the CDN domain name convergence logic without intruding into the business code, without any awareness of the upper business layer.
• When the client finally makes a network request, the domain name of the request URL is replaced with a unified domain name.
• After receiving the URL request of the unified domain name, the CDN server can return to the source normally and return the requested static resources.

Let's first look at an example of a client requesting static resources through a CDN service before CDN domain name convergence:

The process of a static resource URL request to the origin site can be roughly divided into three stages:

• Client side: business upper layer -> client network library;
• Public network: client network library -> CDN service;
• CDN service side: CDN service -> Alibaba OSS origin site;

The second section of the public network in the red box in the above figure: Client network library -> CDN service. This section is a one-to-one relationship. To achieve the convergence of multiple CDN domain names into a unified domain name, the main thing is to transform this section into a single one-to-one relationship. Therefore, just transform the first section into many-to-one and the third section into one-to-many. In simple terms, it is how the client side converges the domain name and how the CDN service side distributes it back to the source station.

2.2 Client-side convergence domain name

When the client-side network library is initialized, a network request interceptor is inserted in advance to intercept network requests and then perform unified processing at the bottom layer, replacing each domain name with a unified domain name. This avoids intrusion into the business layer code and enables unified domain name convergence at the bottom layer without the perception of the upper business layer.

Taking the Android version of Dewu as an example, the network library uses OkHttp3, which supports adding custom interceptors. We can add interceptors for CDN domain name convergence to all OkHttpClient objects. Inserting interceptors through ASpectJ plug-in does not invade business code, and can also ensure that newly created OkHttpClient objects are automatically inserted with interceptors.

 // ASpectJ instrumentation @Aspectpublic class OkHttpAspect {
 /** * Insert after the okhttp3.OkHttpClient.Builder.new(..) method, * to ensure that OkHttpClient created using the new method or the buidler method is covered*/ @After ( "execution(okhttp3.OkHttpClient.Builder.new(..))" ) public void addInterceptor ( JoinPoint joinPoint ) { OkHttpClient .Builder target = ( OkHttpClient .Builder ) joinPoint .getTarget ( ) ; addMergeHostInterceptor ( target ) ; }
 /** * Add CDN domain name convergence interceptor MergeHostInterceptor */ private void addMergeHostInterceptor ( OkHttpClient .Builder builder ) { // Avoid repeated additions, determine whether the interceptor has been added, and return if it has been added for ( Interceptor interceptor : builder .interceptors ( ) ) { if ( interceptor instanceof MergeHostInterceptor ) { return ; } } builder .addInterceptor ( new MergeHostInterceptor ( ) ) ; } }

When the client-side business upper layer initiates a network request for a static resource URL, the network library interceptor intercepts the request and executes the CDN domain name convergence logic. Each domain name is replaced with a unified domain name, and the original domain name information is carried by inserting a path prefix representing each domain name (which is mapped one-to-one with each domain name).

One-to-one mapping table between original domain name and path prefix

Example of generating a new URL after the original URL of the image.xxx.com domain is intercepted

First replace the image.xxx.com domain name with the unified domain name cdn.xxx.com, and then insert the path prefix /image to generate a new URL.

The comparison between the original URL and the new URL is as follows:

https://image.xxx.com/xxx.jpg is replaced by https://cdn.xxx.com/image/xxx.jpg

The specific implementation code is as follows:

 /** * Converge domain name logic and map domain name to path* * @param urlStr original url * @param sourceHost original domain name* @param targetHost unified domain name* @param pathPrefix path prefix mapped to original domain name* @return concatenated new url */ public static String replaceMergeHostUrl ( String urlStr , String sourceHost , String targetHost , String pathPrefix )
 if ( ! TextUtils .isEmpty ( urlStr ) ) { // Replace domain name urlStr = urlStr .replaceFirst ( sourceHost , targetHost ) ;
 if ( ! TextUtils .isEmpty ( pathPrefix ) ) { // Insert path prefix StringBuilder urlStrBuilder = new StringBuilder ( urlStr ) ; int offset = urlStr .indexOf ( targetHost ) + targetHost .length ( ) ; urlStrBuilder .insert ( offset , pathPrefix ) ; urlStr = urlStrBuilder .toString ( ) ; } }
 return urlStr ; }

The network library uses the new URL to request the CDN service. If the CDN service node does not have a resource cache for the new URL or the cache has expired, the CDN service side distribution source station logic is triggered.

2.3 CDN service side distribution source station

• CDN service-side distribution source station solution selection

Solution 1: CDN edge script redirection

CDN edge script Alibaba Cloud official document: https://help.aliyun.com/document_detail/126588.html

Write CDN edge scripts and deploy them on CDN service nodes to support restoring and forwarding requests to the source OSS by redirecting them.

Solution 2: Alibaba Cloud OSS image back to source

Image back-to-source Alibaba Cloud official document: https://help.aliyun.com/document_detail/409627.html

Configure the image return rule for the unified OSS. When the unified OSS does not have static resources, a 404 error occurs, triggering the image return to the source OSS. After successfully pulling resources from the source station, a copy is stored in the unified OSS. The next time you access it, the unified OSS will directly return the stored resource copy and will no longer trigger the image return.

Schematic diagram of image back-to-source principle (from Alibaba Cloud official documentation)

The two solutions have their own advantages and disadvantages. The specific comparison is shown in the following table:

Taking into account both the transformation cost and performance impact, we finally chose "Alibaba Cloud OSS Mirror Back to Source" as the CDN service-side distribution source station solution.

• Alibaba Cloud OSS image back-to-source implementation

Match the path prefix in the unified OSS image back-to-source configuration rules, and map and restore to the source OSS corresponding to the original domain name to achieve accurate image back-to-source to the source OSS.

The one-to-one mapping table between Path prefixes and source OSS is as follows:

For example, the example diagram of OSS mirror back-to-origin configuration for the domain name community.xxx.com

After achieving client-side convergence and CDN service-side distribution of the origin server, let's take a look at the example diagram of a client requesting static resources through the CDN service:

The red box on the left shows an example of client-side convergence completing many-to-one transformation, and the red box on the right shows an example of service-side OSS image back-to-source completing one-to-many transformation. The architecture has basically achieved the goal of CDN domain name convergence. However, we still need to consider how to ensure the stability of the function launch phase and the flexibility of domain name convergence after launch.

2.4 Stable and flexible expansion during the launch phase

Ensure stability by supporting overall grayscale functionality and monitoring logs

Configure the AB experiment switch as the client function grayscale switch, support percentage-based volume expansion, control the grayscale ratio of the client CDN domain name convergence function, and ensure the stability of function launch.

An example of how the client can grayscale the CDN domain name convergence function through AB experiments:

Logs (sampling is supported) are embedded in the key code logic of the CDN domain name convergence function and reported to Alibaba Cloud Log Service (SLS). Monitoring alarms are configured on the SLS platform to facilitate timely detection and processing of online anomalies.

Definition of code-level monitoring points for CDN domain name convergence function

Ensure flexibility by supporting the separate release of domain names to be converged

The client configuration center sends configuration data of the list of domain names to be converged, and supports dynamic delivery and gradual increase of domain names to be converged based on the single domain name dimension.

 {
 "mergeHostConfigs" : [ { // List of CDN domain names to be merged
 "host" : "image.xxx.com" , // original domain name
 "pathPrefix" : "/image" , // path prefix mapped to the original domain name
 "rate" : 1 // Single domain name convergence grayscale rate
 } , {
 "host" : "product.xxx.com" ,
 "pathPrefix" : "/product" ,
 "rate" : 0.5
 } , {
 "host" : "community.xxx.com" ,
 "pathPrefix" : "/community" ,
 "rate" : 0.1
 } ]
 }

Now, let's look at the flow chart of the client requesting static resources after CDN domain name convergence:

Grayscale release process in the development, testing, and release stages

AB experiment volume expansion process

The process of converging and increasing volume in a single domain name dimension

3. CDN multi-vendor disaster recovery

As an e-commerce platform, Dewu needs to provide users with stable and reliable CDN services whether it is a major promotion event (such as 618, Chinese Valentine's Day, Double Eleven, Double Twelve, etc.) or daily services.

Alibaba Cloud CDN service SLA only supports 99.9%, which means there is a risk of 43 minutes of online service unavailability per month. Alibaba Cloud CDN service SLA official document: http://terms.aliyun.com/legal-agreement/terms/suit_bu1_ali_cloud/suit_bu1_ali_cloud201803050950_21147.html?spm=a2c4g.11186623.0.0.7af85fcey4BKBZ

Therefore, after converging multiple CDN domain names into a unified domain name, we decided to upgrade the unified domain name to CDN disaster recovery for the same vendor and multiple vendors.

3.1 Disaster Recovery Idea

The following points are mainly considered:

• A single primary domain name is at risk of becoming unavailable, so you can dynamically send a domain name list to the client.
• When the primary domain name is unavailable, the client automatically downgrades to a disaster recovery state and selects a backup domain name for network requests.
• When all domain names in the domain name list are unavailable, you can restore and use the original domain name before domain name convergence as a backup.
• For unavailable domain names in the domain name list, availability detection can be used to automatically restore the domain name to availability.

3.2 Dynamically send domain name list

The client supports configurable CDN domain names. The domain name used when loading static resources is selected according to the domain name list issued by the configuration center. The priority strategy for selecting domain names is based on the order of the domain names in the domain name list. By default, the first available domain name in the domain name list is used as the current domain name. Other domain names of the same manufacturer or multiple manufacturers can also be configured in the list as backup domain names. Domain name switching priority: primary domain name -> backup domain name of the same manufacturer -> backup domain name of multiple manufacturers.

 {
 "hostListConfig" : [ { // CDN domain name list
 "host" : "cdn.xxx.com" , // primary domain name
 "rate" : 1
 } , {
 "host" : "cdn-ss.xxx.com" , // Alternative domain name of the same manufacturer
 "rate" : 1 // Single domain grayscale rate
 } , {
 "host" : "cdn-bak.xxx.com" , // Multiple vendor domain names
 "rate" : 1 // Single domain grayscale rate
 } ] ,
 "reviveProbeInterval" : 600000 , // Domain name revival detection interval, unit ms
 "configRequestInterval" : 600000 , // Configure the fallback interface request interval, in ms
 "configVersion" : 1 , // Configuration content version number. Each configuration change requires a version number + 1
 "publicIpList" : [ "223.5.5.5" , "180.76.76.76" ] // Domestic public DNS service IP
 }

3.3 Automatic Disaster Recovery Downgrade of Domain Name

When the client business layer initiates a network request for a static resource URL, the domain name replacement logic is executed in the network library CDN domain name convergence interceptor, the domain name of the original URL is replaced with the current domain name selected from the domain name list, and a network request for loading static resources is initiated.

Monitor the abnormal callback of the current domain name request. If the current domain name is determined to be unavailable, update the availability status field isDisabled of the current CDN domain name and assign it to true (true for unavailable and false for available (default)). The conditions for determining that the current domain name is unavailable are as follows:

The HTTP protocol status code returns 5XX (500<=code and code <600)

The socket connection times out/failed, and the client network status is normal (check the client network reasons).

Method for judging the client network status:

Use the InetAddress.isReachable function (Android) or the ping command (such as ping -c 1 223.5.5.5) to detect that at least one of the domestic public DNS services (configured and sent IP lists, such as 223.5.5.5, 180.76.76.76) can be pinged successfully.

If the ping succeeds, it means the client network is normal; if the ping fails, it means the client network is abnormal.

 /**
 * Determine whether the client network is normal
 */
 public static boolean isNetworkStatusNormal ( ) {

 // If there is a public IP in the IP list that can be reached, the network status is considered normal
 for ( int i = 0 ; i < publicIpList .size ( ) ; i ++ ) {
 String ip = publicIpList .get ( i ) ;
 try {
 if ( ! TextUtils .isEmpty ( ip ) ) {
 InetAddress inetAddress = InetAddress .getByName ( ip ) ;
 boolean result = inetAddress .isReachable ( 3000 ) ;
 if ( result ) {
 return true ;
 }
 }
 } catch ( IOException e ) {
 DuLogger .t ( TAG ) .e ( ip + " test is bad, error e = " + e ) ;
 }
 }
 return false ;
 }

If the current CDN domain name is unavailable, traverse the CDN domain name list and obtain the next available CDN domain name as the new current CDN domain name. Replace the URL domain name with the new current domain name again, initiate a static resource network request, and implement automatic disaster recovery and degradation of the end-side domain name to support disaster recovery of the same manufacturer and multiple manufacturers.

If all domain names in the CDN domain name list are unavailable, the fallback logic is executed to restore the original URL before the CDN domain name convergence and initiate the request again.

Flowchart of automatic domain name disaster recovery and downgrade:

3.4 Automatic recovery of domain availability

For CDN domain names whose isDisabled field in the domain name list status field has been assigned a value of true, after the CDN domain name is unblocked and the CDN server failure is restored, the client side needs to promptly perceive and automatically restore the CDN domain name availability to ensure that the client network request traffic is gradually switched back to the primary domain name from the low-priority backup domain name.

The specific steps for restoring domain name availability are as follows:

• The client implements the domain name availability detection logic and supports detecting whether the CDN domain name has been restored and is available.
• After the network library intercepts the network request for the static resource URL, if the current time is greater than the interval time (configuration sent, such as 10 minutes) from the last detection time, the domain name availability detection logic is executed asynchronously.
• In the child thread, the CDN domain name list is traversed, and the unavailable domain names whose availability status field isDisabled is assigned to true are taken out one by one, and the original domain names of the original URL are replaced to generate a new URL.
• Use the new URL to send an Http HEAD request for detection. If the Http status code returned by the request is 2XX, it means that the CDN domain name has been restored and available, and the status field isDisabled is set to false; otherwise, skip it.
• Continue to traverse and detect the availability of the next unavailable domain name until the CDN domain name list traversal is completed.
• The CDN domain name that has regained availability can be reselected and used according to priority in subsequent static resource network requests.

Flowchart of automatic recovery of domain name availability:

4. Challenges encountered

4.1 Scenarios where a small number of resources are dynamically updated

• Challenges

The static resources (pictures, videos, zip files) requested by the client through the CDN service generally do not update the file content (the URL remains unchanged, and the file is overwritten and updated). However, in some business scenarios, the resources requested are json files, and the json file content will be dynamically updated. For example, when the client configuration center platform publishes configuration data, the json file will be overwritten and updated.

We have already introduced the working principle of OSS image back-to-source above. After pulling resources from the source OSS, the unified OSS will store a copy. The next time you access it, it will directly return the stored resource copy and will no longer trigger image back-to-source. Therefore, such dynamic updates of resources require separate compatibility processing.

• Solution

Sort out all business scenarios where resource content is dynamically updated (enumerable) and promote OSS dual-write compatibility transformation. When resource content needs to be dynamically updated, synchronize the source OSS and the unified OSS to ensure that the file content of the resources on the unified OSS is the latest.

4.2 CDN service console monitoring does not support Path dimension monitoring

• Challenges

After the client completes CDN domain name convergence, multiple original domain names are converged into a unified domain name, and the original domain names are distinguished by the Path prefix. The business side expects that after the domain name convergence, it can support viewing performance monitoring reports by the Path dimension, but the Alibaba Cloud CDN service console currently only supports domain name dimension monitoring. Therefore, it is necessary to consider self-developed Path dimension monitoring reports.

• Workaround

The client network monitoring platform supports Path dimension monitoring indicators, including: number of requests, number of return to the source, traffic, bandwidth, and cost.

Use the Get Path Dimension Monitoring Data API provided by Alibaba Cloud CDN to query data, including the number of requests and traffic.

Use the popular URL back-to-source API provided by Alibaba Cloud CDN to query the number of back-to-source times for popular URLs, and then count the number of back-to-source times in the Path dimension.

Example of a Path dimension monitoring report:

4.3 How to check the newly applied CDN domain name and OSS in the future?

• Challenges

Although domain name convergence and mirroring back to the source have been done for the existing multiple domain names and multiple source OSS, we still need to consider how to ensure that no new CDN domain names are added on the Dewu App side.

• Workaround

After communicating with the operation and maintenance colleagues, an approval step was added to the application process for new domain names and new OSS to implement the checkpoint.

4.4 Configuration data is also sent through the CDN domain name, which may cause unavailability risks.

• Challenges

Since configuration data is also issued through the CDN domain name, when the domain name is unavailable, the client cannot pull the latest configuration data (such as the domain name list newly configured by the configuration platform) through the configuration center SDK, and the client resource request failure cannot be restored in time by adjusting the configuration data.

• Workaround

To ensure the timeliness and reliability of configuration data, we can add a dedicated API interface to obtain configuration data as a backup. When the App is cold started or all domain names in the CDN domain name list are unavailable, the dedicated API interface is sent asynchronously to request configuration data.

 Dedicated API interface for obtaining configuration data Interface definition: / app / config - center / module - config
 Request method: Post

Since the configuration center SDK and the dedicated API interface both obtain configuration data from the same source and are independent of each other, when using configuration data, the client needs to compare the configuration data from the two data sources and use the latest configuration data. How to achieve this? We add a configVersion field to represent the version of the configuration data. Each time the configuration data is updated, the configVersion is incremented by +1. The larger the version number, the newer the data. The client can determine which configuration data is the latest data by judging the size of the configVersion in the configuration data of the two data sources and use it.

 // The client obtains the latest configuration data
 private ConfigModel getConfigData ( ) {
 ConfigModel configModel = DataSource .getConfigCenterData ( ) ;
 ConfigModel apiConfigModel = DataSource .getConfigApiData ( ) ;

 // Use the configuration center to send the latest data from the configuration data and the backup interface to send the configuration data
 if ( configModel != null && apiConfigModel != null ) {
 if ( configModel .configVersion < apiConfigModel .configVersion ) {
 configModel = apiConfigModel ;
 }

 } else if ( configModel == null && apiConfigModel != null ) {
 configModel = apiConfigModel ;
 }

 return configModel ;
 }

5. Effect after going online

Completed the convergence of 8 CDN domain names into a unified primary domain name, and supported disaster recovery capabilities for CDN domain names of the same manufacturer and multiple manufacturers.

Improved network request performance

• Android average time: 354ms -> 277ms, a decrease of 77ms (21%)
• iOS average time: 226ms -> 194ms, a decrease of 32ms (14%)

Reduced network request exception rate

After the CDN domain name convergence function was launched, the TCP connection reuse rate was significantly improved, the number of exceptions caused by DNS resolution failure and TCP connection failure was significantly reduced, and the exception rates on both ends were significantly reduced.

• Android exception rate: 3.42% -> 2.56%, a decrease of 0.86%
• iOS exception rate: 0.61% -> 0.13%, a decrease of 0.48%

Improved stability

• The domain name is unified as dewucdn.com to avoid the risk of the old domain name being taken offline and the entire domain being transformed.
• The disaster recovery capability has been improved. CDN domain names support disaster recovery of the same manufacturer and multiple manufacturers, and the SLA has been increased from 99.9% to 99.99%+.

It supports Alibaba Cloud & Tencent Cloud multi-vendor disaster recovery capabilities. When the user network is normal, if the request for resources to Alibaba Cloud CDN service fails (http code 5xx, or socket failure), it will automatically retry and switch to the backup domain name to request resources through Tencent Cloud CDN, and the SLA is improved from 99.9% for a single vendor to 99.99%+.

HTTPDNS cost reduction

Alibaba Cloud HttpDNS service costs reduced by 24%

6. Pitfalls

During the grayscale mass production phase, internal colleagues reported that some identification images on the identification page were blurred.

Cause: OSS image back-to-source rule configuration problem, "Back-to-source parameters: carry request string" was checked.

This means that when the unified OSS is mirroring back to the source, the request parameter after "?" will be carried to the source OSS, and the source OSS will return the cropped thumbnail to the unified OSS according to the cropping parameters, rather than the original image. When the client requests again, the unified OSS will use the thumbnail as the original image, and then perform a second cropping according to the image cropping parameters, resulting in a very small image size returned to the client, which will be blurred after being stretched by the View.

It should be noted that the closer the cropping parameters carried by the first request of the entire network are to the original image, the smaller the impact on the clarity of the image; the smaller the cropping parameters, the larger the cropping parameters of the second request, the greater the degree of stretching of the image, and the blurrier the image.

The first request sample URL of the whole network: https://cdn.xxx.com/image-cdn/app/xxx/identify/du_android_w1160_h2062.jpeg?x-oss-process=image//resize,m_lfit,w_260,h_470 The second request sample URL:

https://cdn.xxx.com/image-cdn/app/xxx/identify/du_android_w1160_h2062.jpeg?x-oss-process=image//resize,m_lfit,w_760,h_1500

As shown in the example URL, the original image is a picture with a width of 1160 and a height of 2062. Due to the need for display in the client View (width 260, height 470), the Alibaba Cloud image cropping parameters "x-oss-process=image//resize,m_lfit,w_260,h_470" are spliced. After hitting the CDN domain name to converge the grayscale, the entire network uses the new URL replaced with the unified domain name to request the image for the first time. The CDN service does not have this URL cache, and the source is returned to the unified OSS. The unified OSS triggers the image back to the source and carries the request parameters to the source OSS. The source OSS will return a thumbnail with a width of 259 and a height of 473 to the unified OSS based on the image cropping parameters in the request parameters. The unified OSS stores the thumbnail as a copy of the original image.

When the second request is made, the unified OSS crops the image according to the cropping parameters of the second request, width 760 and height 1500. However, the width and height of the original copy are smaller than the cropping parameters (width 260 < 760, height 470 < 1500), and finally returns a thumbnail with a width of 260 and a height of 470 to the client. The client View (width 760, height 1500) pulls the resized thumbnail for display, resulting in a blurry image.

During the testing phase, we only verified whether the images were returned and displayed normally. We did not pay attention to the problem that mirroring with image parameters back to the source would cause blurry images.

Solution:

Close the grayscale volume configuration of image.xxx.com.

Reconfigure the OSS image back-to-source rule and uncheck "Back-to-source parameters: carry request string". Use the new path prefix as the mapping to ensure that the original image is correctly pulled when the image back-to-source is re-triggered.

Carry out test verification, and then increase the volume again after confirming that there is no image blur problem.

Delete the incorrect thumbnail images under /image-cdn to avoid wasting OSS storage costs.

7. Summary

Through CDN domain name convergence, we not only achieved improvements in CDN network performance and stability, but also achieved the unification and standardization of multiple domain names, greatly reducing the complexity of subsequent CDN domain name optimization and maintenance. In addition, it also supports the disaster recovery capability of the CDN main domain name, ensuring the stability of online services. ​