Set up AI Gateway

Configure your Helm chart installation to use AI Gateway. Then, use a custom GatewayParameters resource to set up AI Gateway.

Before you begin

Get started to install the Kubernetes Gateway API CRDs and kgateway.

Enable the AI extension

Configure your kgateway Helm chart installation to use AI Gateway.

1. Upgrade kgateway with the AI Gateway extension enabled.

   ⚠️

   If you use a different version or extra Helm settings, update the following command accordingly.

   helm upgrade -i -n kgateway-system kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
        --set gateway.aiExtension.enabled=true \
        --version v2.0.0-main

2. Verify that your Helm installation is updated.

   helm get values kgateway -n kgateway-system -o yaml

   Example output:

   gateway:
     aiExtension:
       enabled: true

Create an AI Gateway

1. Create a GatewayParameters resource which enables an AI extension for the Gateway.

   kubectl apply -f- <<EOF
   apiVersion: gateway.kgateway.dev/v1alpha1
   kind: GatewayParameters
   metadata:
     name: ai-gateway
     namespace: kgateway-system
     labels:
       app: ai-kgateway
   spec:
     kube:
       aiExtension:
         enabled: true
         ports:
         - name: ai-monitoring
           containerPort: 9092
       service:
         type: LoadBalancer
   EOF

2. Create a Gateway that uses the default kgateway GatewayClass and the AI-enabled GatewayParameters resource you created in the previous step.

   kubectl apply -f- <<EOF
   kind: Gateway
   apiVersion: gateway.networking.k8s.io/v1
   metadata:
     name: ai-gateway
     namespace: kgateway-system
     labels:
       app: ai-kgateway
   spec:
     gatewayClassName: kgateway
     infrastructure:
       parametersRef:
         name: ai-gateway
         group: gateway.kgateway.dev
         kind: GatewayParameters
     listeners:
     - protocol: HTTP
       port: 8080
       name: http
       allowedRoutes:
         namespaces:
           from: All
   EOF

3. Verify that the AI Gateway is created.

   • Gateway: Note that it might take a few minutes for an address to be assigned.
   • Deployment: The pod has two containers: kgateway-proxy and kgateway-ai-extension.

   kubectl get gateway,pods -l app.kubernetes.io/app=ai-gateway -A

   Example output:

   NAMESPACE         NAME                                           CLASS      ADDRESS       PROGRAMMED   AGE
   kgateway-system   gateway.gateway.networking.k8s.io/ai-gateway   kgateway   xx.xx.xx.xx   True         13s
   
   NAMESPACE         NAME                              READY   STATUS             RESTARTS   AGE
   kgateway-system   pod/ai-gateway-6f4786fcb6-gqhlm   2/2     Running   0          13s

   If you see an error, check the logs of the kgateway-ai-extension container.

   kubectl logs -l app.kubernetes.io/app=ai-gateway -n kgateway-system -c kgateway-ai-extension

Next

Explore how to connect the gateway to an LLM provider and successfully authenticate so that you can use AI services.