Defect Explainer

AI-generated explanations for defects found during test execution. The service analyzes failed test cases and writes human-readable explanations back into the TestBench execution overview comments.

Endpoint: POST /defect-explanations

Authorization

The triggering user must hold at least one of the following project roles:

Role	Description
TestManager	Full project management access.
Tester	Execution access.

When authenticating with a JWT token, the token must also grant all of the following API token permissions:

Permission	Description
ReadOwnUserDetails	Read the authenticated user's details.
ReadProjectDetails	Read project metadata.
ReadTovReport	Read test-object-version reports.
ReadCycleReport	Read test cycle reports.
ReadReportingJobDetails	Read reporting job details.
DownloadReportFile	Download report files.
ReadTestThemeTree	Read the test theme tree.
ReadTestCaseSetDetails	Read test case set details.
ImportExecutionResults	Import execution results (required to write back explanations).
ReadExecutionImportingJobDetails	Read execution import job status.

---

How it works

1. The service retrieves all test case sets from the specified test-object-version and cycle (a cycle_key is required).
2. For each test case set, it applies the following precheck filters. A test case set is included only if all conditions are met:
   • The execution tab is not locked by another user.
   • Execution data exists.
   • The execution verdict is ToVerify.
   • The execution activity status is Performed.
3. Items that pass the precheck are processed concurrently in the background:
   • Failed test cases are extracted from the execution comments.
   • For each failed test case, the test data and error message are rendered into the prompt template.
   • Each prompt is sent to the configured LLM.
   • All AI-generated explanations are aggregated and written to the comment section of the execution overview.

note

Unlike the other Agents, defect explanations require a cycle_key in the request. If cycle_key is not provided, the precheck fails with a 409 Conflict response.

Configuration

Default configuration

# config.toml
[testbench-ai-service.agents.defect_explainer]
enabled = true
endpoint_path = "/defect-explanations"
class_path = "testbench_ai_service.agents.defect_explainer.agent.DefectExplainer"

[testbench-ai-service.agents.defect_explainer.prompt]
file = "defect_explainer/prompt.yaml"

Agent data

The following variables are generated by the agent and are accessible in prompt templates as {{ agent.<key> }}:

Variable	Type	Description
agent.failed_test_case	String	Execution trace of the failed test case: set name, each step with its execution verdict (►[Pass]/▼[Fail]), and parameters. Children of compound steps are only included when the compound step failed.
agent.error_message	String	The error message extracted from the failed test case.

User-provided variables

The built-in prompt variants declare the following user-provided variable, accessible as {{ vars.<key> }}:

Variable	Type	Description
vars.validate_output	Boolean	When enabled, the model first outputs a management summary and then validates that the output contains only the summary. Defaults to disabled.

Project-specific override

# config.toml
[testbench-ai-service.projects."My Project".agents.defect_explainer]
enabled = false

[testbench-ai-service.projects."My Project".agents.defect_explainer.prompt]
variant = "Quick Summary"

Prompt variants

The built-in prompt file (defect_explainer/prompt.yaml) ships with the following variants:

Variant	Model	Description
Detailed Explanation (default)	gpt-5.5	Structured failure analysis with a focused step-by-step examination. Produces a business-readable one-sentence summary. Uses system+user message structure for best results with GPT chat models.
Quick Summary	gpt-5.5	Fast one-sentence business explanation of the failure based on the highest-level failing step. Cost-efficient; suitable for high-volume runs where speed matters.

Both variants support the optional validate_output user-provided variable. See Prompts for details on how to customize or create your own variants.