Azure OpenAI

Docs

Documentation

Agentgateway

LLM consumption

Providers

Azure OpenAI

Configure Azure OpenAI as an LLM provider in agentgateway.

Before you begin

Set up an agentgateway proxy.

Set up access to Azure OpenAI

Create an API key to access the Azure OpenAI API.

Save the API key in an environment variable.

export AZURE_OPENAI_API_KEY=<insert your API key>

Create a Kubernetes secret to store your Azure OpenAI API key.

kubectl apply -f- <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: azure-openai-secret
  namespace: kgateway-system
type: Opaque
stringData:
  Authorization: $AZURE_OPENAI_API_KEY
EOF

Create a Backend resource to configure an LLM provider that references the Azure OpenAI API key secret.

kubectl apply -f- <<EOF
apiVersion: gateway.kgateway.dev/v1alpha1
kind: Backend
metadata:
  name: azure-openai
  namespace: kgateway-system
spec:
  type: AI
  ai:
    llm:
      azureopenai:
        endpoint: my-endpoint.openai.azure.com
        deploymentName: gpt-4o-mini
        apiVersion: 2024-02-15-preview
        authToken:
          kind: SecretRef
          secretRef:
            name: azure-openai-secret
EOF

Review the following table to understand this configuration. For more information, see the API reference.

Setting	Description
`type`	Set to `AI` to configure this Backend for an AI provider.
`ai`	Define the AI backend configuration. The example uses Azure OpenAI (`spec.ai.llm.azureopenai`).
`endpoint`	The endpoint for the Azure OpenAI API to use, such as `my-endpoint.openai.azure.com`.
`deploymentName`	The name of the Azure OpenAI model deployment to use. For more information, see the Azure OpenAI model docs.
`apiVersion`	The version of the Azure OpenAI API to use. For more information, see the Azure OpenAI API version reference.
`authToken`	Configure the authentication token for Azure OpenAI API. The example refers to the secret that you previously created. The token is automatically sent in the `api-key` header.

Create an HTTPRoute resource that routes incoming traffic to the Backend. The following example sets up a route on the /azure-openai path to the Backend backend that you previously created. Note that kgateway automatically rewrites the endpoint to the appropriate chat completion endpoint of the LLM provider for you, based on the LLM provider that you set up in the Backend resource.

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: azure-openai
  namespace: kgateway-system
spec:
  parentRefs:
    - name: agentgateway
      namespace: kgateway-system
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /azure-openai
    backendRefs:
    - name: azure-openai
      namespace: kgateway-system
      group: gateway.kgateway.dev
      kind: Backend
EOF

Send a request to the LLM provider API. Verify that the request succeeds and that you get back a response from the chat completion API.

curl "$INGRESS_GW_ADDRESS:8080/azure-openai" -H content-type:application/json  -d '{
   "model": "",
   "messages": [
     {
       "role": "system",
       "content": "You are a helpful assistant."
     },
     {
       "role": "user",
       "content": "Write a short haiku about cloud computing."
     }
   ]
 }' | jq

curl "localhost:8080/azure-openai" -H content-type:application/json  -d '{
   "model": "",
   "messages": [
     {
       "role": "system",
       "content": "You are a helpful assistant."
     },
     {
       "role": "user",
       "content": "Write a short haiku about cloud computing."
     }
   ]
 }' | jq

Example output:

{
  "id": "chatcmpl-9A8B7C6D5E4F3G2H1",
  "object": "chat.completion",
  "created": 1727967462,
  "model": "gpt-4o-mini",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Floating servers bright,\nData streams through endless sky,\nClouds hold all we need."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 28,
    "completion_tokens": 19,
    "total_tokens": 47
  }
}

Anthropic Gemini