Documentation
Resiliency
Mirroring

Mirroring

Copy live production traffic to a shadow environment or service so that you can try out, analyze, and monitor new software changes before deploying them to production.

About traffic mirroring

When releasing changes to a service, you want to finely control how those changes get exposed to users. This progressive delivery approach to releasing software allows you to reduce the blast radius, especially when changes introduce unintended behaviors. Traffic mirroring, also referred to as traffic shadowing, is one way to observe the impact of new software releases and test out new changes before you roll them out to production. Other approaches to slowly introduce new software include canary releases, A/B testing, or blue-green deployments.

When you turn on traffic shadowing for an app, kgateway makes a copy of all incoming requests. Kgateway still proxies the request to the backing destination along the request path. It also sends a copy of the request asynchronously to another shadow destination. When a response or failure happens, copies are not generated. This way, you can test how traffic is handled by a new release or version of your app with zero production impact. You can also compare the shadowed results against the expected results. You can use this information to decide how to proceed with a canary release.

To observe and analyze shadowed traffic, you can use a tool like Open Diffy. This tool create diff-compares on the responses. You can use this data to verify that the response is correct and to detect API forward/backward compatibility problems.

Before you begin

  1. Follow the Get started guide to install kgateway.

  2. Follow the Sample app guide to create an API gateway proxy with an HTTP listener and deploy the httpbin sample app.

  3. Get the external address of the gateway and save it in an environment variable. 

    export INGRESS_GW_ADDRESS=$(kubectl get svc -n kgateway-system http -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
echo $INGRESS_GW_ADDRESS  
    kubectl port-forward deployment/http -n kgateway-system 8080:8080

Set up mirroring

  1. Edit the httpbin service that you deployed earlier.

    kubectl edit service httpbin -n httpbin

  2. In the spec.selector field, add version: v1 to make sure that the httpbin service routes requests always to the v1 version of the httpbin app.

    httpbin service snippet
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

    ...
spec:
  clusterIP: 172.20.134.236
  clusterIPs:
  - 172.20.134.236
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    port: 8000
    protocol: TCP
    targetPort: 8080
  - name: tcp
    port: 9000
    protocol: TCP
    targetPort: 9000
  selector:
    app: httpbin
    version: v1

  3. Deploy another version (v2) of httpbin. You use this app to receive the mirrored traffic from httpbin v1.

    kubectl apply -f- <<EOF
apiVersion: v1
kind: Service
metadata:
  name: httpbin2
  namespace: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
    - name: http
      port: 8000
      targetPort: 8080
    - name: tcp
      port: 9000
  selector:
    app: httpbin
    version: v2
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin2
  namespace: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v2
  template:
    metadata:
      labels:
        app: httpbin
        version: v2
    spec:
      serviceAccountName: httpbin
      containers:
        - image: docker.io/mccutchen/go-httpbin:v2.6.0
          imagePullPolicy: IfNotPresent
          name: httpbin
          command: [ go-httpbin ]
          args:
            - "-port"
            - "8080"
            - "-max-duration"
            - "600s" # override default 10s
          ports:
            - containerPort: 8080
        # Include curl container for e2e testing, allows sending traffic mediated by the proxy sidecar
        - name: curl
          image: curlimages/curl:7.83.1
          resources:
            requests:
              cpu: "100m"
            limits:
              cpu: "200m"
          imagePullPolicy: IfNotPresent
          command:
            - "tail"
            - "-f"
            - "/dev/null"
        - name: hey
          image: gcr.io/solo-public/docs/hey:0.1.4
          imagePullPolicy: IfNotPresent
EOF

  4. Verify that the httpbin2 pod is up and running.

    kubectl get pods -n httpbin

  5. Create an HTTPRoute for the httpbin app that mirrors requests along the mirror.example domain from the httpbin app to the httpbin2 app.

    kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin-mirror
  namespace: httpbin
spec:
  parentRefs:
  - name: http
    namespace: kgateway-system
  hostnames:
  - mirror.example
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    filters:
    - type: RequestMirror
      requestMirror:
        backendRef:
          kind: Service
          name: httpbin2
          port: 8000
    backendRefs:
    - name: httpbin
      port: 8000
EOF

  6. Send a few requests to the httpbin app on the mirror.example domain. Verify that you get back a 200 HTTP response code. 

    for i in {1..5}; do curl -vik http://$INGRESS_GW_ADDRESS:8080/headers \
-H "host: mirror.example:8080"; done
    for i in {1..5}; do curl -vik localhost:8080/headers \
-H "host: mirror.example"; done

  7. Get the logs of the httpbin app and verify that you see the requests that you sent.

    kubectl logs -l version=v1 -n httpbin

    Example output:

    time="2025-03-14T19:43:01.1546" status=200 method="GET" uri="/headers" size_bytes=508 duration_ms=0.05 user_agent="curl/8.7.1" client_ip=10.0.8.23
time="2025-03-14T19:43:02.3565" status=200 method="GET" uri="/headers" size_bytes=443 duration_ms=0.06 user_agent="curl/8.7.1" client_ip=10.0.8.23
time="2025-03-14T19:43:03.0178" status=200 method="GET" uri="/headers" size_bytes=508 duration_ms=0.08 user_agent="curl/8.7.1" client_ip=10.0.6.27
time="2025-03-14T19:43:03.3874" status=200 method="GET" uri="/headers" size_bytes=508 duration_ms=0.06 user_agent="curl/8.7.1" client_ip=10.0.8.23
time="2025-03-14T19:43:03.6862" status=200 method="GET" uri="/headers" size_bytes=443 duration_ms=0.07 user_agent="curl/8.7.1" client_ip=10.0.6.27

  8. Get the logs of the httpbin2 app and verify that you see the same requests.

    kubectl logs -l version=v2 -n httpbin

    Example output:

    time="2025-03-14T19:43:01.1548" status=200 method="GET" uri="/headers" size_bytes=443 duration_ms=0.17 user_agent="curl/8.7.1" client_ip=10.0.8.23
time="2025-03-14T19:43:02.3565" status=200 method="GET" uri="/headers" size_bytes=508 duration_ms=0.06 user_agent="curl/8.7.1" client_ip=10.0.8.23
time="2025-03-14T19:43:03.0173" status=200 method="GET" uri="/headers" size_bytes=443 duration_ms=0.15 user_agent="curl/8.7.1" client_ip=10.0.6.27
time="2025-03-14T19:43:03.3869" status=200 method="GET" uri="/headers" size_bytes=443 duration_ms=0.16 user_agent="curl/8.7.1" client_ip=10.0.8.23
time="2025-03-14T19:43:03.6858" status=200 method="GET" uri="/headers" size_bytes=508 duration_ms=0.05 user_agent="curl/8.7.1" client_ip=10.0.6.27

Cleanup

You can remove the resources that you created in this guide. 
kubectl delete service httpbin2 -n httpbin
kubectl delete deployment httpbin2 -n httpbin
kubectl delete httproute httpbin-mirror -n httpbin
Timeouts