TestBench Integration

This page explains how TestBench communicates with the AI Service and how to configure the connection.

---

Overview

TestBench triggers AI Agents by calling the AI Service's REST API. Unlike the Requirement Service or Defect Service, the AI Service does not use a proxy wrapper. TestBench connects directly via its built-in AI integration.

Authentication is handled via JWT tokens: TestBench passes the current user's JWT token with every request, and the AI Service validates it against the TestBench REST API.

---

Requirements

• TestBench 4.0 or newer is required. Older TestBench versions do not provide the AI integration the service relies on.
• TestBench AI Service is installed and running (see Quickstart).
• The tb_server_url in the AI Service config points to the TestBench REST API.
• The provider API key is set in the environment (for example OPENAI_API_KEY or AZURE_OPENAI_API_KEY).

warning

TestBench 4.0 is the minimum for the integration as a whole — individual agents can require a higher TestBench version. All built-in agents (Test Case Set Reviewer, Test Case Set Describer, Defect Explainer) currently require TestBench 4.1 or newer. Each agent checks the connected server version in its precheck and fails with a 409 Conflict if the server is too old. See the agent pages for the version each agent requires.

---

Configuration in TestBench

Configure the AI Service URL in TestBench to point to the running service:

http://127.0.0.1:8010

If you configured HTTPS, use https:// instead and ensure the TestBench host trusts the certificate.

---

AI service plugins in TestBench

This section explains how to enable and run AI service plugins in the TestBench Client.

Enable plugins for your user

Before using AI features, enable the required plugins for your user profile:

1. Open Report and Report Management.
2. Locate the AI plugins you want to use.
3. Select Used by me for each plugin.

note

If the AI plugins are not visible, contact your TestBench administrator.

Open a plugin

You can start AI plugins in two ways.

Option A: Context menu (standard)

1. Navigate to the item you want to process (for example, a Test Case Set).
2. Right-click the item and select Export.
3. Select the desired AI plugin.

Option B: Plugin Quick Access (recommended)

Add frequently used plugins to Plugin Quick Access in Project Management for faster access.

Run the plugin

After opening the plugin, the configuration window appears.

1. Select a variant in Variant (for example, Detailed Explanation).
2. If required, expand Prompt Variables and provide additional input.
3. Select Start.

Precheck and errors

When you select Start, TestBench AI Service runs a precheck. If it passes, the AI service generates the result. If it fails, an error message explains what must be corrected.

---

Authentication flow

┌─────────────────┐                                                ┌─────────────────┐
│    TestBench    │                                                │    TestBench    │
│       UI        │                                                │     REST API    │
└────────┬────────┘                                                └────────▲────────┘
         │                                                                  │
         │ 1. User triggers AI agent                                        │
         │                                                                  │
┌────────┴────────────────────────────────────────────────────┐             │
│ POST /test-case-set-reviews                                 │             │
│ Header: Authorization: <jwt_token>                          │             │
└────────┬────────────────────────────────────────────────────┘             │
         │                                                                  │
         │                                                                  │
┌────────▼────────────────────────────────────────────────────┐             │
│ TestBench AI Service (FastAPI)                              │             │
│ ┌─────────────────────────────────────────────────────────┐ │ 2. Validate │
│ │ - Extract JWT token from header                         │ │    token    │
│ │ - Call TestBench REST API to verify token               ├─┼─────────────┘
│ └─────────────────────────────────────────────────────────┘ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ 3. Check user's project role for the triggered agent    │ │
│ └─────────────────────────────────────────────────────────┘ │
└────────┬────────────────────────────────────────────────────┘
         │
         │ 4. Return 202 Accepted
         │    (processing in background)
         │
┌────────▼────────┐
│   TestBench UI  │
│    (response)   │
└─────────────────┘

1. The user triggers an AI agent in the TestBench UI. TestBench sends a POST request to the AI Service with the user's JWT token as the Authorization header.
2. The AI Service validates the token by calling the TestBench REST API.
3. If valid, the AI Service checks that the authenticated user has the required project role for the triggered agent.
4. The request is accepted and processed in the background.

No separate username/password configuration is needed. The AI Service uses the JWT token issued by TestBench.

---

Verifying the connection

1. Start the AI Service:

   testbench-ai-service start

2. Open http://127.0.0.1:8010/docs in a browser. The Swagger UI should load.
3. Trigger an AI agent from TestBench. Check the AI Service logs for incoming requests.

---

Troubleshooting

Symptom	Likely Cause	Fix
Connection refused	Service is not running or port mismatch.	Start the service; verifyhost and port in config.
401 Unauthorized	Missing or invalid JWT token.	Re-login to TestBench and retry.
502 Bad Gateway	AI Service cannot reach TestBench REST API.	Verifytb_server_url in config.toml is correct and reachable.
404 Not Found	Agent disabled for the project.	Checkenabled = true in the agent config; check project-specific overrides.
409 Conflict	Precheck failed (e.g., all items locked).	Unlock the test structure elements in TestBench and retry.
409 Conflict with an unsupported-version message	TestBench server is older than the version the triggered agent requires (4.1 for all built-in agents).	Update the TestBench server to the required version or newer.
LLM errors in logs	Missing or invalid API key.	Verify your provider key is set in.env or environment (for example OPENAI_API_KEY or AZURE_OPENAI_API_KEY).

---

Network considerations

• By default the service listens on 127.0.0.1 (loopback only). To accept connections from another machine (e.g., TestBench running on a different host), set host = "0.0.0.0" in config.toml.
• If a firewall is in place, open the configured port (default 8010).
• For production deployments, consider enabling HTTPS — see Configuration.