Linkerd 2.10 (Step by Step)—Ingress Traffic

[[406692]]

The Linkerd 2.10 Chinese manual is being continuously revised and updated:

• https://linkerd.hacker-linner.com

As of Linkerd version 2.9, there are two ways to run a Linkerd proxy with your Ingress Controller.

Default Mode

When an ingress controller injects the linkerd.io/inject: enabled annotation, the Linkerd proxy will obey the load balancing decisions made by the ingress controller, rather than applying its own EWMA load balancing. This also means that the Linkerd proxy will not use Service Profiles for this traffic, and thus will not expose per-route metrics or do traffic splitting.

If your Ingress controller injects no additional Ingress-specific configuration, the Linkerd proxy will run in default mode.

Proxy Ingress Mode

If you need Linkerd features like Service Profiles, Traffic Splits, etc., additional configuration is required to enable the Ingress controller's Linkerd proxy to run in ingress mode. This causes Linkerd to route requests based on their :authority, Host, or l5d-dst-override headers instead of the original destination, which allows Linkerd to do its own load balancing and use Service Profiles to expose per-route metrics and enable traffic splitting.

You can enable the proxy of the Ingress controller deployment to run in ingress mode by adding the following annotation linkerd.io/inject: ingress to the Pod Spec of the Ingress Controller.

The same can be done by using the --ingress flag in the injection command.

 kubectl get deployment <ingress-controller> -n <ingress-namespace> -o yaml | linkerd inject --ingress - | kubectl apply -f -  

This can be verified by checking that the Ingress controller's pod has the relevant annotation set.

 kubectl describe pod/<ingress-pod> | grep "linkerd.io/inject: ingress"  

For ingress, most controllers do not rewrite incoming headers (example.com) to internal service names (example.default.svc.cluster.local) by default. In this case, when Linkerd receives an outgoing request, it thinks the request is destined for example.com instead of example.default.svc.cluster.local. This can create a very frustrating infinite loop!

Fortunately, many ingress controllers allow you to modify the Host header or add custom headers to outgoing requests. Here are some descriptions of common ingress controllers:

• Nginx
• Traefik
• GCE
• Ambassador
• Gloo
• Contour
• Kong

If your ingress controller is terminating HTTPS, Linkerd will only provide TCP statistics for incoming requests, since all traffic the proxy sees is encrypted. It will provide full statistics for outgoing requests from the controller to the backend services, since that's plaintext from the controller to Linkerd.

If requests experience 2-3 seconds of latency after being injected into your ingress controller, this may be because the type: LoadBalancer service is hiding the client source IP. You can resolve this issue by setting externalTrafficPolicy: Local in the ingress service definition.

While the Kubernetes Ingress API definition allows the backend's servicePort to be a string value, Linkerd can only use numeric servicePort values. If a string value is encountered, Linkerd will default to using port 80.

Nginx

Here we take emojivoto as an example

An example ingress definition is:

 apiVersion: extensions/v1beta1
 kind: Ingress
 metadata:
 name : web-ingress
 namespace: emojivoto
 annotations:
    kubernetes.io/ingress.class: "nginx"  
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster. local :$service_port;
      grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster. local :$service_port; 
 
 spec:
 rules:
 - host: example.com
 http:
 paths:
 - backend:
          serviceName: web-svc
 servicePort: 80

The important annotations here are:

 nginx.ingress.kubernetes.io/configuration-snippet: |
    proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster. local :$service_port;
    grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster. local :$service_port;

If you are using auth-url, you will also need to add the following snippet.

 nginx.ingress.kubernetes.io/auth-snippet: |
   proxy_set_header l5d-dst-override authn- name .authn-namespace.svc.cluster. local :authn-port;
   grpc_set_header l5d-dst-override authn- name .authn-namespace.svc.cluster. local :authn-port;

This example combines two directives that NGINX uses to proxy HTTP and gRPC traffic. In practice, depending on the protocol used by the service, only the proxy_set_header or grpc_set_header directive needs to be set, but NGINX will ignore any directives that are not required.

This example ingress definition uses a single ingress for an application with multiple endpoints that use different ports.

 apiVersion: extensions/v1beta1
 kind: Ingress
 metadata:
 name : web-ingress
 namespace: emojivoto
 annotations:
    kubernetes.io/ingress.class: "nginx"  
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster. local :$service_port;
      grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster. local :$service_port;
 spec:
 rules:
 - host: example.com
 http:
 paths:
 - path: /
 backend:
          serviceName: web-svc
 servicePort: 80
      - path: /another-endpoint
 backend:
          serviceName: another-svc
 servicePort: 8080

Nginx will add a l5d-dst-override header to instruct Linkerd what service the request is destined for. You need to include both the Kubernetes service FQDN (web-svc.emojivoto.svc.cluster.local) and the target servicePort.

To test this, you need to get the external IP address of the controller. If you installed nginx-ingress via helm, you can get that IP address by running:

 kubectl get svc --all-namespaces \  
  -l app=nginx-ingress,component=controller \
  -o=custom-columns=EXTERNAL-IP:.status.loadBalancer.ingress[0].ip

You can then use this IP via curl:

 curl -H "Host: example.com" http://external-ip

If you are using the default backend, you will need to create an ingress definition for that backend to ensure the l5d-dst-override header is set. For example:

 apiVersion: extensions/v1beta1
 kind: Ingress
 metadata:
 name : default -ingress
 namespace: backends
 annotations:
    kubernetes.io/ingress.class: "nginx"  
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header l5d-dst-override $service_name.$namespace.svc.cluster. local :$service_port;
      grpc_set_header l5d-dst-override $service_name.$namespace.svc.cluster. local :$service_port;
 spec:
 backend:
    serviceName: default -backend
 servicePort: 80

Traefik

Here we take emojivoto as an example. See getting started to review how to install it.

The simplest way to use Traefik as a Linkerd ingress is to configure the Kubernetes Ingress resource using ingress.kubernetes.io/custom-request-headers as follows:

 apiVersion: extensions/v1beta1
 kind: Ingress
 metadata:
 name : web-ingress
 namespace: emojivoto
 annotations:
    kubernetes.io/ingress.class: "traefik"  
    ingress.kubernetes.io/custom-request-headers: l5d-dst-override:web-svc.emojivoto.svc.cluster. local :80
 spec:
 rules:
 - host: example.com
 http:
 paths:
 - backend:
          serviceName: web-svc
 servicePort: 80

The important annotations here are:

 ingress.kubernetes.io/custom-request-headers: l5d-dst-override:web-svc.emojivoto.svc.cluster. local :80

Traefik will add a l5d-dst-override header to instruct Linkerd what service the request is destined for. You need to include both the Kubernetes service FQDN (web-svc.emojivoto.svc.cluster.local) and the target servicePort. For more information, see the Traefik website.

To test this, you need to get the external IP address of the controller. If you installed Traefik via helm, you can get that IP address by running:

 kubectl get svc --all-namespaces \  
 -l app=traefik \
  -o= 'custom-columns=EXTERNAL-IP:.status.loadBalancer.ingress[0].ip'  

You can then use this IP via curl:

 curl -H "Host: example.com" http://external-ip

This solution won't work if you're using Traefik's service weights, since Linkerd will always send requests to the service name in l5d-dst-override. A workaround is to use traefik.frontend.passHostHeader: "false" instead. Note that if you're using TLS, the connection between Traefik and the backend service will not be encrypted. There is an open issue tracking a solution to this issue.

Traefik 2.x

Traefik 2.x added support for path-based request routing through a Custom Resource Definition (CRD) called IngressRoute.

If you choose to use IngressRoute instead of the default Kubernetes Ingress resource, you will also need to use Traefik's Middleware Custom Resource Definition to add the l5d-dst-override header.

The following YAML produces the same results for the emojivoto application using the Traefik CRD as described above.

 apiVersion: traefik.containo.us/v1alpha1
 kind: Middleware
 metadata:
 name : l5d-header-middleware
 namespace: traefik
 spec:
 headers:
 customRequestHeaders:
      l5d-dst-override: "web-svc.emojivoto.svc.cluster.local:80"  
 ---  
 apiVersion: traefik.containo.us/v1alpha1
 kind: IngressRoute
 metadata:
 annotations:
    kubernetes.io/ingress.class: traefik
 creationTimestamp: null  
 name : emojivoto-web-ingress-route
 namespace: emojivoto
 spec:
 entryPoints: []
 routes:
 - kind: Rule  
 match: PathPrefix(`/`)
 priority: 0
 middlewares:
    - name : l5d-header-middleware
 services:
 - kind: Service
 name : web-svc
 port: 80

GCE

This example is similar to Traefik and also uses emojivoto as an example. See getting started to review how to install it.

In addition to the custom headers found in the Traefik example, it also shows how to use a Google Cloud Static External IP Address and TLS with a Google-managed certificate.

An example ingress definition is:

 apiVersion: extensions/v1beta1
 kind: Ingress
 metadata:
 name : web-ingress
 namespace: emojivoto
 annotations:
    kubernetes.io/ingress.class: "gce"  
    ingress.kubernetes.io/custom-request-headers: "l5d-dst-override: web-svc.emojivoto.svc.cluster.local:80"  
    ingress.gcp.kubernetes.io/pre-shared-cert: "managed-cert-name"  
    kubernetes.io/ingress. global - static -ip- name : "static-ip-name"  
 spec:
 rules:
 - host: example.com
 http:
 paths:
 - backend:
          serviceName: web-svc
 servicePort: 80

To use this example definition, replace managed-cert-name and static-ip-name with the short names defined in your project (nb use the name of the IP address, not the address itself).

The managed certificate will take approximately 30-60 minutes to provision, but the status of the ingress should be healthy within a few minutes. After the managed certificate is provisioned, the ingress should be visible to the Internet.

Ambassador

Here we take emojivoto as an example. See getting started to review how to install it.

Ambassador does not use Ingress resources, but relies on Services. An example service definition is:

 apiVersion: v1
 kind: Service
 metadata:
 name : web-ambassador
 namespace: emojivoto
 annotations:
    getambassador.io/config: |
 ---  
      apiVersion: ambassador/v1
 kind: Mapping
 name : web-ambassador-mapping
      service: http://web-svc.emojivoto.svc.cluster. local :80
 host: example.com
 prefix: /
      add_linkerd_headers: true  
 spec:
 selector:
 app: web-svc
 ports:
 - name : http
 port: 80
 targetPort: http

The important annotations here are:

 add_linkerd_headers: true  

Ambassador will add an l5d-dst-override header to indicate to Linkerd what service the request is for. This will contain the Kubernetes service FQDN (web-svc.emojivoto.svc.cluster.local) and the destination servicePort.

To make it global, add add_linkerd_headers to your Module config.

To test this, you need to get the external IP address of the controller. If you installed Ambassador via helm, you can get that IP address by running:

 kubectl get svc --all-namespaces \  
  -l "app.kubernetes.io/name=ambassador" \
  -o= 'custom-columns=EXTERNAL-IP:.status.loadBalancer.ingress[0].ip'  

If you have installed the management interface, this will return two IPs, one of which is . Just ignore that and use the actual IP address.

You can then use this IP via curl:

 curl -H "Host: example.com" http://external-ip

You can also find a more detailed guide on using Linkerd with Emissary Ingress (aka Ambassador) from the folks at Buoyant here.

Gloo

Here we take books as an example, check Demo: Books to learn how to run it.

If you installed Gloo using the Gateway method (gloo install gateway), then you will need a VirtualService to route traffic to your Books application.

To use Gloo with Linkerd, you can choose one of two options.

Automatic

Starting with Gloo v0.13.20, Gloo has native integration with Linkerd, so the required Linkerd headers are automatically added.

Assuming you installed gloo to the default location, you can enable native integration by running:

 kubectl patch settings -n gloo-system default \
  -p '{"spec":{"linkerd":true}}'   --type=merge  

Gloo now automatically adds the l5d-dst-override header to each kubernetes upstream.

Now we just need to add a route to the upstream books app:

 glooctl add route --path-prefix=/ --dest-name booksapp-webapp-7000  

Manual

As mentioned at the beginning of this document, you need to instruct Gloo to add a header to allow Linkerd to identify where to send traffic.

 apiVersion: gateway.solo.io/v1
 kind: VirtualService
 metadata:
 name : books
 namespace: gloo-system
 spec:
 virtualHost:
 domains:
 - '*'  
 name : gloo-system.books
 routes:
 - matcher:
 prefix: /
 routeAction:
 single:
 upstream:
 name : booksapp-webapp-7000
            namespace: gloo-system
 routePlugins:
 transformations:
          requestTransformation:
            transformationTemplate:
 headers:
                l5d-dst-override:
                  text: webapp.booksapp.svc.cluster. local :7000
                passthrough: {}

The important annotations here are:

 routePlugins:
 transformations:
 requestTransformation:
       transformationTemplate:
 headers:
 l5d-dst-override:
             text: webapp.booksapp.svc.cluster. local :7000
 passthrough: {}

Using the built-in content transformation engine in Gloo, you can instruct it to add the required l5d-dst-override header, which in the example above points to the FDQN and port of the service: webapp.booksapp.svc.cluster.local:7000

Test

To easily test this, you can get the URL of the Gloo proxy by running:

 glooctl proxy URL

This will return something like:

 $ glooctl proxy url
 http://192.168.99.132:30969

For the example VirtualService above, which listens on any domain and path, accessing the proxy URL (http://192.168.99.132:30969) in a browser should open the Books application.

Contour

Contour does not support automatic setting of the l5d-dst-override header. The following example uses Contour getting started to demonstrate how to manually set the required header:

First, inject Linkerd into your Contour installation:

 linkerd inject https://projectcontour.io/quickstart/contour.yaml | kubectl apply -f -

Envoy does not automatically mount the service account token. To fix this, you need to set automountServiceAccountToken: true. You can choose to create a dedicated service account to avoid using the default.

 # create a service account (optional)
 kubectl apply -f - << EOF
 apiVersion: v1
 kind: ServiceAccount
 metadata:
 name : envoy
 namespace: projectcontour
 EOF 
 
 # add service account to envoy (optional)
 kubectl patch daemonset envoy -n projectcontour --type json -p='[{"op": "add", "path": "/spec/template/spec/serviceAccount", "value": "envoy"}]'   
 
 # auto mount the service account token (required)
 kubectl patch daemonset envoy -n projectcontour --type json -p='[{"op": "replace", "path": "/spec/template/spec/automountServiceAccountToken", "value": true}]'  

Verify that your Contour and Envoy installations have a running Linkerd sidecar.

Next we will deploy a demo service:

 linkerd inject https://projectcontour.io/examples/kuard.yaml | kubectl apply -f -

To route external traffic to your service, you need to provide an HTTPProxy:

 apiVersion: projectcontour.io/v1
 kind: HTTPProxy
 metadata:
 name : kuard
 namespace: default  
 spec:
 routes:
 - requestHeadersPolicy:
 set :
      - name : l5d-dst-override
        value: kuard. default .svc.cluster. local :80
 services:
 - name : kuard
 namespace: default  
 port: 80
 virtualhost:
 fqdn: 127.0.0.1.xip.io

Note that the l5d-dst-override header is explicitly set to the destination service.

Finally, you can test your working service mesh:

 kubectl port- forward svc/envoy -n projectcontour 3200:80
 http://127.0.0.1.xip.io:3200

If you use Contour with flagger, the l5d-dst-override request header will be set automatically.

Kong

Kong does not automatically support the header l5d-dst-override. This document will use the following elements:

• Kong
• Emojivoto

Before installing the Emojivoto demo application, install Linkerd and Kong on your cluster. Remember to use the --ingress flag (or annotation) mentioned above when injecting the Kong deployment!

We also need to declare these objects:

• KongPlugin, CRD provided by Kong
• Ingress

 apiVersion: configuration.konghq.com/v1
 kind: KongPlugin
 metadata:
 name : set -l5d-header
 namespace: emojivoto
 plugin: request-transformer
 config:
 add :
 headers:
    - l5d-dst-override:$(headers.host) .svc.cluster.local  
 ---  
 apiVersion: extensions/v1beta1
 kind: Ingress
 metadata:
 name : web-ingress
 namespace: emojivoto
 annotations:
    kubernetes.io/ingress.class: "kong"  
    konghq.com/plugins: set -l5d-header
 spec:
 rules:
 - http:
 paths:
 - path: /api/vote
 backend:
              serviceName: web-svc
              servicePort: http
 - path: /api/list
 backend:
              serviceName: web-svc
              servicePort: http

We explicitly set l5d-dst-override in KongPlugin. Using templates as values, we can use the host header from the request and set the l5d-dst-override value based on that.

Finally, let's install Emojivoto so that its deploy/vote-bot targets the ingress and contains a host header for the web-svc.emojivoto service.

Before applying the injected Emojivoto application, make the following changes to the vote-bot deployment:

 env:
 # Target the Kong ingress instead   of the Emojivoto web service
 - name : WEB_HOST
 value: kong-proxy.kong:80
 # Override the host header on requests so that it can be used to   set the l5d-dst-override header
 - name : HOST_OVERRIDE
 value: web-svc.emojivoto

【Editor's recommendation】

1. Remote desktop developed by Chinese programmers is popular! Available for Mac, only 9MB, supports self-built repeaters
2. 3 minutes to help you understand Kafka thoroughly
3. How big is the bandwidth of Tik Tok's server to accommodate hundreds of millions of people watching at the same time?
4. Github is hot! Can this so-called postmodern editor surpass Vim?
5. The seven most dangerous attack techniques in 2021