For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Custom Envoy bootstrap config

Inject custom Envoy bootstrap configuration into a managed gateway proxy by overriding the bootstrap ConfigMap that the control plane generates with overlays.

Use this technique to set bootstrap-level Envoy options that are not exposed as built-in fields on the GatewayParameters resource, such as the stats_config.histogram_bucket_settings to tune histogram bucket boundaries for your metrics.

How it works

For a managed gateway proxy, the control plane generates a ConfigMap that is named after the Gateway and holds the Envoy bootstrap as an envoy.yaml entry. The proxy pod mounts this ConfigMap on the /etc/envoy path by using a volume that is named envoy-config. The Envoy container is started from the bootstrap configuration.

To inject your own bootstrap configuration while the control plane continues to manage routing through xDS, you can follow these general steps:

1. Get the ConfigMap that is automatically generated to extract the default bootstrap configuration.
2. Customize the bootstrap configuration to your needs and store it in your own ConfigMap.
3. Use a deploymentOverlay in the GatewayParameters resource to point the envoy-config volume at your custom ConfigMap. Volumes merge on the name key under strategic merge patch, so the overlay updates the existing volume rather than adding a new one.

Because you start from a copy of the generated bootstrap, the proxy keeps its control-plane–managed configuration, such as the node identity, xds_cluster, xDS token, and dynamic_resources, and continues to connect to the control plane.

Before you begin

1. Follow the Get started guide to install kgateway.

2. Follow the Sample app guide to create a gateway proxy with an HTTP listener and deploy the httpbin sample app.

3. Get the external address of the gateway and save it in an environment variable.

   Cloud Provider LoadBalancerPort-forward for local testing

   export INGRESS_GW_ADDRESS=$(kubectl get svc -n kgateway-system http -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
   echo $INGRESS_GW_ADDRESS  

   kubectl port-forward deployment/http -n kgateway-system 8080:8080

Override the bootstrap config

1. Deploy a Gateway and a GatewayParameters resource so that the control plane generates the bootstrap ConfigMap.

   kubectl apply -f- <<EOF
   apiVersion: gateway.kgateway.dev/v1alpha1
   kind: GatewayParameters
   metadata:
     name: gw-params
     namespace: kgateway-system
   spec:
     kube:
       deployment:
         replicas: 1
   ---
   kind: Gateway
   apiVersion: gateway.networking.k8s.io/v1
   metadata:
     name: custom
     namespace: kgateway-system
   spec:
     gatewayClassName: kgateway
     infrastructure:
       parametersRef:
         name: gw-params
         group: gateway.kgateway.dev
         kind: GatewayParameters
     listeners:
     - protocol: HTTP
       port: 80
       name: http
       allowedRoutes:
         namespaces:
           from: All
   EOF

2. Get the ConfigMap that contains the Envoy bootstrap configuration and save it in a local file.

   kubectl get configmap custom -n kgateway-system -o yaml > custom-bootstrap.yaml

3. Make the following changes to your ConfigMap.

   1. Change the metadata.name field to a custom name, such as custom-bootstrap. You must create the ConfigMap under a different name to avoid automatic overwrites of the ConfigMap by the managed proxy.

   2. Find the Envoy bootstrap configuration in the envoy.yaml entry and customize it to your needs. The following example adds a stats_config block with custom histogram bucket boundaries (in milliseconds) for cluster stats. Add the block as a new top-level key. Make sure to leave all the existing keys of your Envoy bootstrap configuration unchanged.

      stats_config:
        histogram_bucket_settings:
        - match:
            prefix: "cluster."
          buckets: [1, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000]
      notifications

      If you prefer to extract only the Envoy bootstrap configuration for easier updating, run the following command. Make sure to apply your customizations to the envoy.yaml section of your ConfigMap.

      kubectl get configmap custom -n kgateway-system \
        -o jsonpath='{.data.envoy\.yaml}' > envoy.yaml
      
      open envoy.yaml

   3. Apply the new ConfigMap.

      kubectl apply -f custom-bootstrap.yaml

4. Add a deploymentOverlay to the GatewayParameters resource to point the envoy-config volume at your custom ConfigMap.

   kubectl apply -f- <<EOF
   apiVersion: gateway.kgateway.dev/v1alpha1
   kind: GatewayParameters
   metadata:
     name: gw-params
     namespace: kgateway-system
   spec:
     kube:
       deployment:
         replicas: 1
       deploymentOverlay:
         spec:
           template:
             spec:
               volumes:
               # Merge key is 'name'; this patches the existing envoy-config volume
               - name: envoy-config
                 configMap:
                   name: custom-bootstrap
   EOF

5. Verify that the gateway proxy is restarted.

   kubectl get pods -n kgateway-system | grep custom

6. Port-forward the gateway proxy on port 19000 to access the admin interface.

   kubectl port-forward -n kgateway-system deploy/custom 19000:19000

7. Get the histogram_bucket_settings of your proxy config dump to verify that your custom settings are applied.

   curl -s localhost:19000/config_dump | grep -A25 histogram_bucket_settings

   Example output:

   "histogram_bucket_settings": [
    {
     "match": {
      "prefix": "cluster."
     },
     "buckets": [
      1,
      5,
      10,
      25,
      50,
      100,
      250,
      500,
      1000,
      2500,
      5000,
      10000
     ]
    }
   ]
   

Keep your custom bootstrap up to date

When you upgrade kgateway, the generated bootstrap format can change to add new fields, clusters, or listeners. Because your custom ConfigMap is a copy, it does not pick up these changes automatically. After each upgrade, regenerate the bootstrap from a proxy that uses the default configuration, re-apply your customization, and update your custom ConfigMap.

To avoid this maintenance, use the built-in customization fields where possible, and open a feature request if you need a built-in field for your bootstrap setting.

Cleanup

kubectl delete gateway custom -n kgateway-system
kubectl delete GatewayParameters gw-params -n kgateway-system
kubectl delete configmap custom-bootstrap -n kgateway-system