Test Case Set Reviewer

AI-powered quality reviews of test case sets. The service analyzes test case structure, steps, parameters, and naming against best practices (ISTQB, ISO 29119) and writes the review notes back into the TestBench review comment section.

Endpoint: POST /test-case-set-reviews

Authorization

The triggering user must hold at least one of the following project roles:

Role	Description
TestManager	Full project management access.
TestDesigner	Specification authoring access.

When authenticating with a JWT token, the token must also grant all of the following API token permissions:

Permission	Description
ReadOwnUserDetails	Read the authenticated user's details.
ReadProjectDetails	Read project metadata.
ReadTovReport	Read test-object-version reports.
ReadCycleReport	Read test cycle reports.
ReadReportingJobDetails	Read reporting job details.
DownloadReportFile	Download report files.
ReadTestThemeTree	Read the test theme tree.
ReadTestCaseSetDetails	Read test case set details.
ModifySpecifications	Write the review result back to the specification.
ModifySpecManagementInfo	Update specification management metadata.

---

How it works

1. The service retrieves all test case sets from the specified test-object-version (and optional cycle / subtree).
2. For each test case set, it checks that the specification tab is not locked by another user.
3. Items that pass the precheck are processed concurrently in the background:
   • The current review comment is saved as a backup.
   • A "review started" marker is written to the review comment.
   • Test case data (steps, parameters, parameter combinations) is rendered into the prompt template.
   • The prompt is sent to the configured LLM.
   • The AI response is written to the review comment section.
   • If an error occurs, the previous review comment is restored.

Configuration

Default configuration

# config.toml
[testbench-ai-service.agents.test_case_set_reviewer]
enabled = true
endpoint_path = "/test-case-set-reviews"
class_path = "testbench_ai_service.agents.test_case_set_reviewer.agent.TestCaseSetReviewer"

[testbench-ai-service.agents.test_case_set_reviewer.prompt]
file = "test_case_set_reviewer/prompt.yaml"

Agent data

The following variables are generated by the agent and are accessible in prompt templates as {{ agent.<key> }}:

Variable	Type	Description
agent.test_case_set	String	Formatted string representation of the test case set (name, steps, parameters).
agent.parameter_combinations	String	Formatted parameter combination table. May be empty.
agent.test_case_set_description	String	The existing test case set description, if one is present (plain text, stripped of HTML). May be empty.

User-provided variables

The built-in prompt variants declare the following user-provided variable, accessible as {{ vars.<key> }}:

Variable	Type	Description
vars.glossary	String	Optional glossary of project-specific terms. When provided, test steps are checked against it.

You can supply this variable in your config or per-request:

# config.toml
[testbench-ai-service.agents.test_case_set_reviewer.prompt.vars]
glossary = "ABS = Anti-lock Braking System\nESC = Electronic Stability Control"

Project-specific override

# config.toml
[testbench-ai-service.projects."My Project".agents.test_case_set_reviewer]
enabled = false

[testbench-ai-service.projects."My Project".agents.test_case_set_reviewer.prompt]
variant = "Quick Review"

Prompt variants

The built-in prompt file (test_case_set_reviewer/prompt.yaml) ships with the following variants:

Variant	Model	Description
Full Review (default)	gpt-5.5	Comprehensive ISTQB/ISO 29119 quality review. Examines naming, description, step formulation, validation coverage, and consistency. Uses system+user message structure for best results with GPT chat models.
Quick Review	gpt-5.5	Lightweight structural review focused on missing validation steps, generic names, incorrect step form, and glossary mismatches. Fast and cost-efficient; suitable for high-volume batch processing.

Both variants support the optional glossary user-provided variable. See Prompts for details on how to customize or create your own variants.