Set up AI Gateway

Configure your Helm chart installation to use AI Gateway. Then, use a custom GatewayParameters resource to set up AI Gateway.

Before you begin

Get started to install the Kubernetes Gateway API CRDs and kgateway.

Enable the AI extension

Configure your kgateway Helm chart installation to use AI Gateway.

1. Upgrade kgateway with the AI Gateway extension enabled.

   ⚠️

   If you use a different version or extra Helm settings such as in a -f values.yaml file, update the following command accordingly.

   helm upgrade -i -n kgateway-system kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
        --set gateway.aiExtension.enabled=true \
        --version v$NEW_VERSION

2. Verify that your Helm installation is updated.

   helm get values kgateway -n kgateway-system -o yaml

   Example output:

   gateway:
     aiExtension:
       enabled: true

Create an AI Gateway

1. Create a GatewayParameters resource which enables an AI extension for the Gateway.

   Cloud ProviderLocal Environment

   For AI services in a cloud provider, use a LoadBalancer service. This way, the AI Gateway can be accessed from outside the cluster.

   kubectl apply -f- <<EOF
   apiVersion: gateway.kgateway.dev/v1alpha1
   kind: GatewayParameters
   metadata:
     name: ai-gateway
     namespace: kgateway-system
     labels:
       app: ai-gateway
   spec:
     kube:
       aiExtension:
         enabled: true
         ports:
         - name: ai-monitoring
           containerPort: 9092
         image:
           registry: cr.kgateway.dev/kgateway-dev
           repository: kgateway-ai-extension
           tag: v2.1.0-main
       service:
         type: LoadBalancer
   EOF

   For local environments such as Ollama, use a NodePort service. This way, the AI Gateway can be accessed locally.

   kubectl apply -f- <<EOF
   apiVersion: gateway.kgateway.dev/v1alpha1
   kind: GatewayParameters
   metadata:
     name: ai-gateway
     namespace: kgateway-system
     labels:
       app: ai-gateway
   spec:
     kube:
       aiExtension:
         enabled: true
         ports:
         - name: ai-monitoring
           containerPort: 9092
         image:
           registry: cr.kgateway.dev/kgateway-dev
           repository: kgateway-ai-extension
           tag: v2.1.0-main
       service:
         type: NodePort
   EOF

2. Create a Gateway that uses the default GatewayClass and the AI-enabled GatewayParameters resource you created in the previous step.

   kubectl apply -f- <<EOF
   kind: Gateway
   apiVersion: gateway.networking.k8s.io/v1
   metadata:
     name: ai-gateway
     namespace: kgateway-system
     labels:
       app: ai-gateway
   spec:
     gatewayClassName: kgateway
     infrastructure:
       parametersRef:
         name: ai-gateway
         group: gateway.kgateway.dev
         kind: GatewayParameters
     listeners:
     - protocol: HTTP
       port: 8080
       name: http
       allowedRoutes:
         namespaces:
           from: All
   EOF

3. Verify that the AI Gateway is created.

   • Gateway: Note that it might take a few minutes for an address to be assigned.
   • Deployment: The pod has two containers: kgateway-proxy and kgateway-ai-extension.

   kubectl get gateway,pods -l app.kubernetes.io/name=ai-gateway -A

   Example output:

   NAMESPACE         NAME                                           CLASS      ADDRESS       PROGRAMMED   AGE
   kgateway-system   gateway.gateway.networking.k8s.io/ai-gateway   kgateway   xx.xx.xx.xx   True         13s
   
   NAMESPACE         NAME                              READY   STATUS             RESTARTS   AGE
   kgateway-system   pod/ai-gateway-6f4786fcb6-gqhlm   2/2     Running   0          13s

   If you see an error, check the logs of the kgateway-ai-extension container.

   kubectl logs -l app.kubernetes.io/app=ai-gateway -n kgateway-system -c kgateway-ai-extension

Next

• For OpenAI: Continue with the Authenticate to the LLM guide.
• For other cloud LLM providers such as Gemini: Review the Cloud LLM providers guide for provider-specific setup examples.
• For local LLM providers such as Ollama: Set up Ollama as a local LLM provider.