Azure OpenAI Setup
This guide walks you through connecting the TestBench AI Service to an Azure OpenAI resource.
Requirements
- An active Azure subscription
- An Azure OpenAI resource with at least one model deployment
- The TestBench AI Service installed and a
config.tomlpresent - Enough permissions in Azure to configure the resource — see Required permissions if you intend to authenticate with Microsoft Entra ID
1. Create an Azure OpenAI resource
- Open the Azure portal and navigate to Azure OpenAI.
- Click Create and fill in the required details (subscription, resource group, region, name).
- Once the resource is created, open it and go to Keys and Endpoint.
- Copy the Endpoint URL (e.g.
https://your-resource.openai.azure.com) and one of the API keys.
2. Create a model deployment
- In your Azure OpenAI resource, navigate to Azure OpenAI Studio → Deployments → Deploy model.
- Select a base model (e.g.
gpt-4oorgpt-4.1-mini) and give it a deployment name (e.g.my-gpt4o). - Note down the deployment name. This is what you will reference in prompt variants and in
config.toml.
3. Choose an authentication method
The service supports two ways to authenticate against Azure OpenAI:
| Method | auth_method | When to use |
|---|---|---|
| API key | api_key (default) | Direct integration scenarios where a key is explicitly required |
| Microsoft Entra ID | entra_id | Application development, and wherever your security policy mandates Entra ID |
Managed identity is not supported, because the service runs on-premise where no managed identity is available. Entra ID authentication therefore uses a service principal (app registration) with a client secret.
Option A: API key
The service reads the Azure OpenAI API key from the environment variable
AZURE_OPENAI_API_KEY.
Create or update a .env file in the root of your installation directory:
# .env
AZURE_OPENAI_API_KEY=your_azure_openai_api_key
Never commit API keys to version control. Add .env to your .gitignore.
Option B: Microsoft Entra ID
Required permissions
Two different sets of permissions are involved, and they are easy to mix up: the permissions you need in order to set this up, and the permission the service principal needs in order to call the model.
1. Permissions you need to complete the setup
| Task | Required role |
|---|---|
| Create the app registration and a client secret | Application Developer in Microsoft Entra ID — or none at all, if your tenant leaves the default Users can register applications setting enabled |
| Assign a role on the Azure OpenAI resource | Owner, User Access Administrator or Role Based Access Control Administrator on the resource, its resource group or the subscription |
If you lack the second one, Add role assignment is visible but greyed out.
In that case have a subscription owner perform step 4 for you — the app
registration itself can still be created independently.
2. Permission the service principal needs
| Role | Scope | Purpose |
|---|---|---|
| Cognitive Services OpenAI User | The Azure OpenAI resource (or the resource group / subscription containing it) | Data-plane access: send chat completion requests and list deployments |
This is the only role the service requires. Cognitive Services OpenAI Contributor also works but additionally allows creating and deleting model deployments, which the service never does — prefer the User role.
The Azure control-plane roles Reader, Contributor and Cognitive
Services Contributor do not grant inference access. They let you see and
manage the resource in the portal while every model request still returns
401 Unauthorized or 403 Forbidden. The role must be one of the
Cognitive Services OpenAI … roles.
No API permissions or admin consent are needed. The service uses the OAuth 2
client credentials flow and requests the token scope
https://cognitiveservices.azure.com/.default directly. You do not have to add
anything under the app registration's API permissions blade, and no
Microsoft Graph access is involved — the app registration never reads directory
data.
Role assignments can take a few minutes to propagate. If requests still fail
with 403 immediately after step 4, wait and retry before changing anything.
Prepare the app registration in Azure
These steps happen in your own Azure tenant and are your responsibility. The AI service does not perform them.
- In the Azure portal, go to Microsoft Entra ID → App registrations → New registration. Give the application a name and register it.
- On the application's Overview page, note the Application (client) ID and the Directory (tenant) ID.
- Go to Certificates & secrets → New client secret. Note the secret value immediately; it is shown only once.
- Open your Azure OpenAI resource → Access control (IAM) → Add role assignment. Assign the role Cognitive Services OpenAI User to the application you registered.
Step 4 is the one that is most often missed. Without the role assignment the
configuration looks correct in every respect and every request still fails with
401 Unauthorized or 403 Forbidden.
Configure the service
Set auth_method in config.toml:
# config.toml
[testbench-ai-service.llm_config]
provider = "azure_openai"
auth_method = "entra_id"
azure_endpoint = "https://your-resource.openai.azure.com"
api_version = "2025-04-01-preview"
Supply the service principal through environment variables:
# .env
AZURE_TENANT_ID=your_directory_tenant_id
AZURE_CLIENT_ID=your_application_client_id
AZURE_CLIENT_SECRET=your_client_secret
| Variable | Azure portal name |
|---|---|
AZURE_TENANT_ID | Directory (tenant) ID |
AZURE_CLIENT_ID | Application (client) ID |
AZURE_CLIENT_SECRET | Client secret value |
AZURE_OPENAI_API_KEY is not read in this mode and does not need to be set.
Client secrets expire. Note the expiry date from Certificates & secrets and plan the rotation — the service will start failing on the expiry date otherwise.
4. Configure config.toml
Update the [testbench-ai-service.llm_config] section in your config.toml:
# config.toml
[testbench-ai-service.llm_config]
provider = "azure_openai"
azure_endpoint = "https://your-resource.openai.azure.com"
api_version = "2025-04-01-preview"
Both azure_endpoint and api_version are required when provider = "azure_openai".
Choosing an API version
Microsoft releases new API versions regularly. Check the Azure OpenAI REST API reference for the latest stable version. A commonly used recent version is 2025-04-01-preview.
5. Map deployment names to canonical models
In prompt YAML files you reference your Azure deployment name. The service needs to know the underlying canonical model name in order to route requests correctly (for example, reasoning models like o3 require different API parameters than chat models like gpt-4o).
If your deployment name is identical to a known canonical model name no mapping is required. Otherwise, add a deployment_mapping table:
# config.toml
[testbench-ai-service.llm_config]
provider = "azure_openai"
azure_endpoint = "https://your-resource.openai.azure.com"
api_version = "2025-04-01-preview"
[testbench-ai-service.llm_config.deployment_mapping]
"my-gpt4o-deployment" = "gpt-4o"
"my-gpt41-mini" = "gpt-4.1-mini"
"my-o3-deployment" = "o3"
The key is the deployment name you use in prompt variants; the value is the canonical OpenAI model identifier that the deployment is based on.
Known canonical chat models
| Canonical name | Type |
|---|---|
gpt-4o | Chat |
gpt-4o-mini | Chat |
gpt-4o-search-preview | Chat |
gpt-4o-mini-search-preview | Chat |
gpt-4.1 | Chat |
gpt-4.1-mini | Chat |
gpt-4.1-nano | Chat |
gpt-4-turbo | Chat |
gpt-4 | Chat |
gpt-3.5-turbo | Chat |
gpt-3.5-turbo-16k | Chat |
gpt-3.5-turbo-instruct | Chat |
Known canonical reasoning models
| Canonical name | Type |
|---|---|
gpt-5 | Reasoning |
gpt-5-mini | Reasoning |
gpt-5-nano | Reasoning |
gpt-5-codex | Reasoning |
gpt-5-pro | Reasoning |
gpt-5.1 | Reasoning |
gpt-5.1-codex | Reasoning |
gpt-5.1-codex-max | Reasoning |
gpt-5.1-codex-mini | Reasoning |
gpt-5.2 | Reasoning |
gpt-5.2-codex | Reasoning |
gpt-5.2-pro | Reasoning |
gpt-5.3-codex | Reasoning |
gpt-5.4 | Reasoning |
gpt-5.4-mini | Reasoning |
gpt-5.4-nano | Reasoning |
gpt-5.4-pro | Reasoning |
gpt-5.5 | Reasoning |
gpt-5.5-pro | Reasoning |
o1 | Reasoning |
o1-pro | Reasoning |
o3 | Reasoning |
o3-mini | Reasoning |
o4-mini | Reasoning |
6. Reference the deployment in a prompt variant
In your prompt YAML file, set model to the Azure deployment name:
variants:
- name: "Full Description"
model: "my-gpt4o-deployment" # Azure deployment name
messages:
- role: "system"
file: "system.jinja"
- role: "user"
file: "user.jinja"
Minimal complete example
.env
AZURE_OPENAI_API_KEY=abc123yourkey
config.toml
# config.toml
[testbench-ai-service]
tb_server_url = "https://localhost:9443/api/"
host = "127.0.0.1"
port = 8010
language = "en"
[testbench-ai-service.llm_config]
provider = "azure_openai"
azure_endpoint = "https://your-resource.openai.azure.com"
api_version = "2025-04-01-preview"
[testbench-ai-service.llm_config.deployment_mapping]
"gpt-4o-tb" = "gpt-4o"
prompts/en/test_case_set_describer/prompt.yaml (excerpt)
variants:
- name: "Full Description"
model: "gpt-4o-tb"
messages:
- role: "system"
file: "system.jinja"
- role: "user"
file: "full_description_user.jinja"
Project-specific configuration
You can override the Azure OpenAI settings per TestBench project. This is useful when different projects use different Azure resources or deployments.
config.toml
# config.toml
# Global Azure OpenAI configuration
[testbench-ai-service.llm_config]
provider = "azure_openai"
azure_endpoint = "https://shared-resource.openai.azure.com"
api_version = "2025-04-01-preview"
[testbench-ai-service.llm_config.deployment_mapping]
"gpt-4o-tb" = "gpt-4o"
# Override for a specific project
[testbench-ai-service.projects."My Project".llm_config]
provider = "azure_openai"
azure_endpoint = "https://project-specific-resource.openai.azure.com"
api_version = "2025-04-01-preview"
[testbench-ai-service.projects."My Project".llm_config.deployment_mapping]
"gpt-4o-project" = "gpt-4o"
A project-specific API key can also be set via environment variable using the pattern {NORMALIZED_PROJECT_NAME}_AZURE_OPENAI_API_KEY, where the project name is uppercased and all non-alphanumeric characters are replaced with underscores:
# .env
# For a project named "My Project":
MY_PROJECT_AZURE_OPENAI_API_KEY=project-specific-azure-key
If a project-specific key is present the service creates a dedicated client for that project; otherwise the global AZURE_OPENAI_API_KEY is used.
When auth_method = "entra_id" is configured, a project can likewise use its
own service principal. All three variables must be set together:
# .env
# For a project named "My Project":
MY_PROJECT_AZURE_TENANT_ID=project_specific_tenant_id
MY_PROJECT_AZURE_CLIENT_ID=project_specific_client_id
MY_PROJECT_AZURE_CLIENT_SECRET=project_specific_secret
If none of the three are set, the project uses the global service principal. If only some are set, the service reports an error naming the missing variables rather than falling back — a partially configured project principal would otherwise authenticate silently as an unintended identity.
This check does not run at service startup for project overrides. The global service principal is validated at startup, because the global Azure OpenAI client is created then. Project credentials are resolved lazily, the first time a request runs against that project — so the service starts cleanly even with a half-configured project override, and the error only surfaces on that project's first request.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
API key for provider 'azure_openai' not found | AZURE_OPENAI_API_KEY is not set | Set the variable in your .env file or system environment |
'azure_endpoint' must be set for provider 'azure_openai' | Missing endpoint in config.toml | Add azure_endpoint to [testbench-ai-service.llm_config] |
'api_version' must be set for provider 'azure_openai' | Missing API version in config.toml | Add api_version to [testbench-ai-service.llm_config] |
DeploymentNotFound from Azure API | Wrong deployment name in prompt variant | Verify the deployment name in Azure OpenAI Studio and update the prompt YAML |
| Empty or unexpected response | Deployment mapped to wrong canonical model | Check and correct the deployment_mapping in config.toml |
Entra ID authentication ... is incompletely configured | One or two of the three service principal variables are missing | Set all of AZURE_TENANT_ID, AZURE_CLIENT_ID and AZURE_CLIENT_SECRET (or all three project-prefixed variants) |
'auth_method = entra_id' is only supported for provider 'azure_openai' | auth_method set on a non-Azure provider | Remove auth_method, or set provider = "azure_openai" |
The 'azure-identity' package is required | Running from source without the dependency installed | Run pip install azure-identity |
401 Unauthorized / 403 Forbidden with correct credentials | The app registration has no role on the Azure OpenAI resource | Assign the Cognitive Services OpenAI User role (step 4 above) |
403 Forbidden although a role is assigned | A control-plane role such as Contributor or Cognitive Services Contributor was assigned — these grant no inference access | Assign Cognitive Services OpenAI User instead, see Required permissions |
403 Forbidden only in the first minutes after setup | The role assignment has not propagated yet | Wait a few minutes and retry |
| Add role assignment is greyed out in the portal | You are missing Owner / User Access Administrator / Role Based Access Control Administrator on the resource | Ask a subscription owner to assign the role |
ClientAuthenticationError: AADSTS7000215 | Invalid client secret, or the secret value was confused with the secret ID | Copy the secret value from Certificates & secrets |
ClientAuthenticationError after months of working | The client secret expired | Create a new secret and update AZURE_CLIENT_SECRET |