After years of working in the infrastructure domain, I’ve realized that migration is an inevitable task. In all companies I’ve worked for, migrations have been a constant presence. Even at tech conferences, it’s very common to see migration stories being shared on stage. These can range from Kubernetes migrations to database migrations, tool migrations, and more.

In this post, I’ll share my experience with Kubernetes cluster migration. Moving from one Kubernetes cluster to another involves many aspects, including CI/CD setup, monitoring tools, load balancers, and more. However, I’ll specifically focus on traffic migration.

Current Setup

In our current Kubernetes cluster, Istio serves as the service mesh solution. The cluster has several ingress controllers, with different types of traffic handled by different controllers. Since Istio is our mesh solution, all traffic coming to the containers, regardless of source, is intercepted by the Istio sidecar. This makes the Istio sidecar the perfect place to implement traffic migration.

Istio and Envoy Features

The Istio sidecar is an Envoy proxy. Istio, as a control plane, modifies Envoy’s configuration based on Istio Custom Resource Definitions (CRDs). Istio provides several CRDs that allow you to configure Envoy, such as VirtualService, DestinationRule, ServiceEntry, and more.

Envoy, as a proxy, has built-in capabilities for splitting traffic across multiple destinations . The next question is how to configure Envoy using Istio.

Envoy Configuration Changes

Here’s the Envoy configuration when routing traffic to a single destination:

route_config:
  '@type': type.googleapis.com/envoy.config.route.v3.RouteConfiguration
  name: inbound|8080||
  validate_clusters: false
  virtual_hosts:
  - domains:
    - '*'
    name: inbound|http|80
    routes:
    - decorator:
        operation: hello-world-service.default.svc.cluster.local:80/*
      match:
        prefix: /
      name: default
      route:
        cluster: inbound|8080||
        max_stream_duration:
          grpc_timeout_header_max: 0s
          max_stream_duration: 0s
        retry_policy:
          num_retries: 2
          retry_on: reset-before-request
        timeout: 0s

To split traffic across multiple destinations, we change cluster to weighted_clusters. The weighted_clusters configuration allows us to distribute traffic across multiple destinations. For example, to split 20% of the traffic to the-new-target-address.example.com and the remaining 80% to the local container, use this configuration:

route_config:
  '@type': type.googleapis.com/envoy.config.route.v3.RouteConfiguration
  name: inbound|8080||
  validate_clusters: false
  virtual_hosts:
  - domains:
    - '*'
    name: inbound|http|80
    routes:
    - decorator:
        operation: hello-world-service.default.svc.cluster.local:80/*
      match: 
        prefix: /
      name: default
      route:
        weighted_clusters:
          clusters:
          - name: inbound|8080||
            weight: 80
          - host_rewrite_literal: the-new-target-address.example.com
            name: outbound|80||the-new-target-address.example.com
            weight: 20
        max_stream_duration:
          grpc_timeout_header_max: 0s
          max_stream_duration: 0s
        retry_policy:
          num_retries: 2
          retry_on: reset-before-request
        timeout: 0s

Here’s a detailed view of the changes:

route_config:
  '@type': type.googleapis.com/envoy.config.route.v3.RouteConfiguration
  name: inbound|8080||
  validate_clusters: false
  virtual_hosts:
  - domains:
    - '*'
    name: inbound|http|80
    routes:
    - decorator:
        operation: hello-world-service.default.svc.cluster.local:80/*
      match: 
        prefix: /
      name: default
      route:
-       cluster: inbound|8080||
+       weighted_clusters:
+         clusters:
+         - name: inbound|8080||
+           weight: 80
+         - host_rewrite_literal: the-new-target-address.example.com
+           name: outbound|80||the-new-target-address.example.com
+           weight: 20
        max_stream_duration:
          grpc_timeout_header_max: 0s
          max_stream_duration: 0s
        retry_policy:
          num_retries: 2
          retry_on: reset-before-request
        timeout: 0s

Istio EnvoyFilter

Istio provides a CRD called EnvoyFilter that allows you to configure Envoy directly. The main purpose of EnvoyFilter is to patch configurations based on matchers. Here’s the EnvoyFilter configuration to apply the changes described above:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: hello-world-service
  namespace: default
spec:
  workloadSelector:
    labels:
      app: hello-world-service
  configPatches:
  - applyTo: HTTP_ROUTE
    match:
      context: SIDECAR_INBOUND
      routeConfiguration:
        vhost:
          name: inbound|http|80
          route:
            action: ROUTE
    patch:
      operation: INSERT_BEFORE
      value:
        match:
          prefix: /
        route:
          weightedClusters:
            clusters:
            - name: inbound|9898||
              weight: 80
            - host_rewrite_literal: the-new-target-address.example.com
              name: outbound|80||the-new-target-address.example.com
              weight: 20

The EnvoyFilter patches the HTTP_ROUTE for inbound traffic. It inserts a new route before the selected route with the weighted_clusters configuration.

This is a simplified version. In a real implementation, you need to ensure that the-new-target-address.example.com is registered to the mesh using ServiceEntry.

Cappy Hoding! 🖖🏾