Linkerd Canary Deployment and A/B Testing

[[413903]]

This guide shows you how to use Linkerd and Flagger to automate canary deployments and A/B testing.

Flagger Linkerd Traffic Split

Prerequisites

Flagger requires a Kubernetes cluster v1.16 or later and Linkerd 2.10 or later.

Install Linkerd the Prometheus (part of Linkerd Viz):

 linkerd install | kubectl apply -f -
 linkerd viz install | kubectl apply -f -

Install Flagger in the linkerd namespace:

 kubectl apply -k github.com/fluxcd/flagger//kustomize/linkerd

Bootloader

Flagger takes a Kubernetes deployment and optionally Horizontal Pod Autoscaling (HPA), and then creates a series of objects (Kubernetes Deployment, ClusterIP Service, and SMI Traffic Splitting). These objects expose the application to the mesh and drive Canary analysis and promotion.

Create a test namespace and enable Linkerd proxy injection:

 kubectl create ns test
 kubectl annotate namespace test linkerd.io/inject=enabled

Install a load testing service to generate traffic during canary analysis:

 kubectl apply -k https://github.com/fluxcd/flagger//kustomize/tester?ref=main

Create a deployment and a horizontal pod autoscaler:

 kubectl apply -k https://github.com/fluxcd/flagger//kustomize/podinfo?ref=main

Create a Canary custom resource for the podinfo deployment:

 apiVersion: flagger.app/v1beta1
 kind: Canary
 metadata:
 name : podinfo
 namespace: test
 spec:
 # deployment reference
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name : podinfo
 # HPA reference (optional)
 autoscalerRef:
    apiVersion: autoscaling/v2beta2
    kind: HorizontalPodAutoscaler
 name : podinfo
 # the maximum time   in seconds for the canary deployment
  # to make progress before it is   rollback ( default 600s)
 progressDeadlineSeconds: 60
 service:
 # ClusterIP port number
 port: 9898
    # container port number or   name (optional)
 targetPort: 9898
 analysis:
    # schedule interval ( default 60s)
 interval: 30s
    # max number of failed metric checks before rollback  
 threshold: 5
    # max traffic percentage routed to canary
 # percentage (0-100)
 maxWeight: 50
 # canary increment step
 # percentage (0-100)
 stepWeight: 5
    # Linkerd Prometheus checks
 metrics:
    - name : request-success-rate
      # minimum req success rate (non 5xx responses)
 # percentage (0-100)
 thresholdRange:
 min : 99
 interval: 1m
 - name : request-duration
      # maximum req duration P99
 # milliseconds
 thresholdRange:
 max : 500
 interval: 30s
 # testing (optional)
 webhooks:
      - name : acceptance-test
 type: pre-rollout
        url: http://flagger-loadtester.test/
 timeout: 30s
 metadata:
 type: bash
          cmd: "curl -sd 'test' http://podinfo-canary.test:9898/token | grep token"  
 - name : load -test
 type: rollout
        url: http://flagger-loadtester.test/
 metadata:
          cmd: "hey -z 2m -q 10 -c 2 http://podinfo-canary.test:9898/"  

Save the above resources as podinfo-canary.yaml and apply:

 kubectl apply -f ./podinfo-canary.yaml

When the canary analysis starts, Flagger will call the pre-rollout webhooks before routing traffic to the canary. The canary analysis will run for five minutes while validating HTTP metrics and rollout hooks every half minute.

After a few seconds, Flager will create the canary object:

 # applied
 deployment.apps/podinfo
 horizontalpodautoscaler.autoscaling/podinfo
 ingresses.extensions/podinfo
 canary.flagger.app/podinfo 
 
 # generated
 deployment.apps/podinfo- primary  
 horizontalpodautoscaler.autoscaling/podinfo- primary  
 service/podinfo
 service/podinfo-canary
 service/podinfo- primary  
 trafficsplits.split.smi-spec.io/podinfo

After boostrap, the podinfo deployment will be scaled to zero and traffic to podinfo.test will be routed to the primary pod. During canary analysis, the canary pod can be directly located using the podinfo-canary.test address.

Automatic canary advancement

Flagger implements a control loop that gradually shifts traffic to the canary while measuring key performance indicators such as HTTP request success rate, average request duration, and Pod health. Based on the analysis of the KPIs, the canary is promoted or aborted, and the analysis results are published to Slack.

Flagger Canary Phase

Trigger a canary deployment by updating the container image:

 kubectl -n test set image deployment/podinfo \
 podinfod=stefanprodan/podinfo:3.1.1

Flagger detects that the deployment revision has changed and starts a new deployment:

 kubectl -n test describe canary/podinfo 
 
 Status:
 Canary Weight: 0
 Failed Checks: 0
  Phase: Succeeded
 Events:
 New revision detected! Scaling up podinfo.test
 Waiting for podinfo.test rollout to finish: 0 of 1 updated replicas are available
 Pre-rollout check acceptance-test passed
 Advance podinfo.test canary weight 5
 Advance podinfo.test canary weight 10
 Advance podinfo.test canary weight 15
 Advance podinfo.test canary weight 20
 Advance podinfo.test canary weight 25
 Waiting for podinfo.test rollout to finish: 1 of 2 updated replicas are available
 Advance podinfo.test canary weight 30
 Advance podinfo.test canary weight 35
 Advance podinfo.test canary weight 40
 Advance podinfo.test canary weight 45
 Advance podinfo.test canary weight 50
 Copying podinfo.test template spec to podinfo- primary .test
 Waiting for podinfo- primary .test rollout to finish: 1 of 2 updated replicas are available
 Promotion completed! Scaling down podinfo.test

Note that if you apply new changes to your deployment during canary analysis, Flagger will restart the analysis.

A canary deployment is triggered by a change to any of the following objects:

• Deployment PodSpec (container image, command, ports, environment, resources, etc.)
• ConfigMaps mounted as volumes or mapped to environment variables
• Secrets mounted as volumes or mapped to environment variables

You can monitor all canaries via:

 watch kubectl get canaries --all-namespaces   
 
 NAMESPACE NAME STATUS WEIGHT LASTTRANSITIONTIME
 test podinfo Progressing 15 2019-06-30T14:05:07Z
 prod frontend Succeeded 0 2019-06-30T16:15:07Z
 prod backend Failed 0 2019-06-30T17:05:07Z

Automatic rollback

During canary analysis, you can generate HTTP 500 errors and high latency to test whether Flagger pauses and rolls back the faulty version.

Trigger another canary deployment:

 kubectl -n test set image deployment/podinfo \
 podinfod=stefanprodan/podinfo:3.1.2

Execute the load tester pod using the following command:

 kubectl -n test exec -it flagger-loadtester-xx-xx sh

Generates an HTTP 500 error:

 watch -n 1 curl http://podinfo-canary.test:9898/status/500

Generate Delay:

 watch -n 1 curl http://podinfo-canary.test:9898/delay/1

When the number of failed checks reaches the canary analysis threshold, traffic is routed back to the primary server, the canary is scaled to zero, and the rollout is marked as failed.

 kubectl -n test describe canary/podinfo 
 
 Status:
 Canary Weight: 0
 Failed Checks: 10
  Phase: Failed
 Events:
 Starting canary analysis for podinfo.test
 Pre-rollout check acceptance-test passed
 Advance podinfo.test canary weight 5
 Advance podinfo.test canary weight 10
 Advance podinfo.test canary weight 15
 Halt podinfo.test advancement success rate 69.17% < 99%
 Halt podinfo.test advancement success rate 61.39% < 99%
 Halt podinfo.test advancement success rate 55.06% < 99%
 Halt podinfo.test advancement request duration 1.20s > 0.5s
 Halt podinfo.test advancement request duration 1.45s > 0.5s
 Rolling back podinfo.test failed checks threshold reached 5
 Canary failed! Scaling down podinfo.test

Custom Indicators

Canary analysis can be extended via Prometheus queries.

Let's define a check for not finding errors. Edit canary analysis and add the following indicator:

 analysis:
 metrics:
 - name : "404s percentage"  
 threshold: 3
 query: |
 100 - sum (
 rate(
 response_total{
                  namespace= "test" ,
                  deployment= "podinfo" ,
                  status_code!= "404" ,
                  direction= "inbound"  
 }[1m]
 )
 )
 /
 sum (
 rate(
 response_total{
                  namespace= "test" ,
                  deployment= "podinfo" ,
                  direction= "inbound"  
 }[1m]
 )
 )
 * 100

The above configuration verifies the canary version by checking if the HTTP 404 req/sec percentage is below 3% of the total traffic. If the 404s rate reaches the 3% threshold, the analysis is aborted and the canary is marked as failed.

Trigger a canary deployment by updating the container image:

 kubectl -n test set image deployment/podinfo \
 podinfod=stefanprodan/podinfo:3.1.3

Generates a 404:

 watch -n 1 curl http://podinfo-canary:9898/status/404

Monitor Flagger logs:

 kubectl -n linkerd logs deployment/flagger -f | jq .msg 
 
 Starting canary deployment for podinfo.test
 Pre-rollout check acceptance-test passed
 Advance podinfo.test canary weight 5
 Halt podinfo.test advancement 404s percentage 6.20 > 3
 Halt podinfo.test advancement 404s percentage 6.45 > 3
 Halt podinfo.test advancement 404s percentage 7.22 > 3
 Halt podinfo.test advancement 404s percentage 6.50 > 3
 Halt podinfo.test advancement 404s percentage 6.34 > 3
 Rolling back podinfo.test failed checks threshold reached 5
 Canary failed! Scaling down podinfo.test

If you have Slack configured, Flagger will send a notification stating why the canary failed.

Linkerd Ingress

There are two ingress controllers that are compatible with Flagger and Linkerd: NGINX and Gloo.

Install NGINX:

 helm upgrade -i nginx-ingress stable/nginx-ingress \
 --namespace ingress-nginx  

Create an ingress definition for podinfo that rewrites incoming headers to the internal service name (required for Linkerd):

 apiVersion: extensions/v1beta1
 kind: Ingress
 metadata:
 name : podinfo
 namespace: test
 labels:
 app:podinfo
 annotations:
    kubernetes.io/ingress.class: "nginx"  
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster. local :9898;
      proxy_hide_header l5d-remote-ip;
      proxy_hide_header l5d-server-id;
 spec:
 rules:
 - host: app.example.com
 http:
 paths:
 - backend:
              serviceName:podinfo
              servicePort: 9898

When using an ingress controller, Linkerd traffic splitting does not apply to incoming traffic because NGINX runs outside the mesh. To run canary analysis on the frontend application, Flagger creates a shadow ingress and sets NGINX-specific annotations.

A/B Testing

In addition to weighted routing, Flagger can also be configured to route traffic to canaries based on HTTP match conditions. In an A/B testing scenario, you will use HTTP headers or cookies to target your specific user base. This is particularly useful for front-end applications that require session affinity.

Flagger Linkerd Ingress

Edit the podinfo canary analysis, set the provider to nginx, add the ingress reference, remove the max/step weights and add match conditions and iterations:

 apiVersion: flagger.app/v1beta1
 kind: Canary
 metadata:
 name : podinfo
 namespace: test
 spec:
 # ingress reference
 provider: nginx
 ingressRef:
    apiVersion: extensions/v1beta1
 kind: Ingress
 name : podinfo
 targetRef:
 apiVersion: apps/v1
 kind: Deployment
 name : podinfo
 autoscalerRef:
    apiVersion: autoscaling/v2beta2
    kind: HorizontalPodAutoscaler
 name : podinfo
 service:
 # container port
 port: 9898
 analysis:
 interval: 1m
 threshold: 10
 iterations: 10
 match:
      # curl -H 'X-Canary: always' http://app.example.com
 - headers:
 x-canary:
 exact: "always"  
      # curl -b 'canary=always' http://app.example.com
 - headers:
 Cookies:
 exact: "canary"  
    # Linkerd Prometheus checks
 metrics:
    - name : request-success-rate
 thresholdRange:
 min : 99
 interval: 1m
 - name : request-duration
 thresholdRange:
 max : 500
 interval: 30s
 webhooks:
      - name : acceptance-test
 type: pre-rollout
        url: http://flagger-loadtester.test/
 timeout: 30s
 metadata:
 type: bash
          cmd: "curl -sd 'test' http://podinfo-canary:9898/token | grep token"  
 - name : load -test
 type: rollout
        url: http://flagger-loadtester.test/
 metadata:
          cmd: "hey -z 2m -q 10 -c 2 -H 'Cookie: canary=always' http://app.example.com"  

The above configuration will run a 10 minute analysis targeting users who have the canary cookie set to always or who call the service with the X-Canary: always header.

Note that the load test now targets an external address and uses a canary cookie.

Trigger a canary deployment by updating the container image:

 kubectl -n test set image deployment/podinfo \
 podinfod=stefanprodan/podinfo:3.1.4

Flagger detects that the deployment revision has changed and starts an A/B test:

 kubectl -n test describe canary/podinfo 
 
 Events:
 Starting canary deployment for podinfo.test
 Pre-rollout check acceptance-test passed
 Advance podinfo.test canary iteration 1/10
 Advance podinfo.test canary iteration 2/10
 Advance podinfo.test canary iteration 3/10
 Advance podinfo.test canary iteration 4/10
 Advance podinfo.test canary iteration 5/10
 Advance podinfo.test canary iteration 6/10
 Advance podinfo.test canary iteration 7/10
 Advance podinfo.test canary iteration 8/10
 Advance podinfo.test canary iteration 9/10
 Advance podinfo.test canary iteration 10/10
 Copying podinfo.test template spec to podinfo- primary .test
 Waiting for podinfo- primary .test rollout to finish: 1 of 2 updated replicas are available
 Promotion completed! Scaling down podinfo.test