Skip to main content
Test Suite stores reusable cases for one agent, combines local and shared criteria, and runs those cases against a captured configuration. Cases can come from Inbox conversations or scripted customer turns. See Test Suite for the dashboard workflow.
The previous Test Sets API has been removed. Use the /v2/test-suite/.../public routes below. Old test-set IDs and criterion IDs are not interchangeable with case or criteria-group IDs.

​
Workflow

1

Save cases and criteria

Create a case from a conversation or scenario. Add local criteria or link a criteria group. Use fields context for library criteria and condition metadata.
2

Organize and review setup

Collections include cases directly or through linked groups. Review captured conversation input, or use Action fixtures to script mock responses.
3

Check readiness and run

Preview readiness before starting a run. Choose draft configuration from run configuration. Poll Get run for results.
Runs use recorded or mock Action responses and never execute live Actions. Draft Behavior, rules and articles affect the run without being published. A missing Action response can produce an execution error; save a missing mock into the original conversation case and start another run.

​
Shared objects

​
object
A named conversation or scripted scenario for one botId. Detail responses expose messages, setup, artifacts and history rather than the stored raw input. criteria contains effective checks; localCriteria contains case-specific checks. criteriaGroups and criterionOrigins identify shared checks.
​
object
Shared checks for one agent. Detail responses include linkedCases, caseCount, revision and impactToken. Updating the group or removing members requires the latest impact token. A stale token returns 409; reload and review the changed impact before retrying.
​
object
A named grouping of cases and criteria groups. The API calls these folders. Effective case membership includes direct cases and cases from linked groups, without duplicates. Removing a direct case link does not remove membership through a group.
​
object
A configuration snapshot and case-run items. Item statuses are queued, running, completed and failed. A completed evaluation returns a pass/fail verdict; an execution failure can return diagnostics and an error. Saved snapshots remain readable after a case is deleted.

​
Criterion definitions

A case can have up to 50 effective criteria across its local checks and all shared groups. Cases without criteria can be saved, but cannot run. Groups require at least one criterion. Use one of these definition shapes:
  • Library criterion: defaultCriterionId from fields context, with an optional blocking override. Do not include a name, type, condition or judge prompts with a library reference.
  • Deterministic criterion: nonblank name, type: "deterministic" and a valid condition. Judge prompts must be absent or null.
  • AI judgement: nonblank name, type: "basic_judge" or "complex_judge", and all three nonblank fields: judgePrompt, passPrompt, failPrompt. condition must be absent or null.
For a new custom criterion, blocking defaults to true. A library reference inherits the saved value unless overridden. Only blocking failures make the overall case fail. Create library criterion saves a new custom definition; it does not accept a library reference. A Test Suite condition uses one predicate object, rather than the predicate array used by Rulebook flows:
Allowed context roots are calledActions, escalated, knowledgeSearchUsed, usedArticles, usedKnowledgeFolders, intentRules, tagGroups and replyTypes. Array conditions also use array and a quantifier of ANY, ALL or NONE. Get supported field types, operators and reply-type values from fields context.

​
Case input

For sourceType: "conversation", provide conversationId and omit scenario. The conversation must belong to the same agent, contain customer messages, have at most 200 events, and come from Inbox rather than a replay or Test Suite run. Original agent replies are preview-only; execution generates fresh replies. For sourceType: "scenario", provide scenario and omit conversationId. A scenario has one to 50 turns and optional userAttributes. Each turn has a nonblank message of at most 10,000 characters and up to 50 actions. Each Action includes an assigned ruleId, a nonblank nodeId of at most 100 characters, a toolId and JSON-object output; success defaults to true. Use Action fixtures for valid node identifiers. Both source types support attributeOverrides, shared criteriaGroupIds and manual artifacts. Captured input is limited to 1 MB. Criterion definitions and attribute overrides are each limited to 100 KB. Updating a case always requires name; omit optional fields to preserve them.

​
Recorded Action overrides

Captured-conversation cases expose actionResponses. To change a recorded response, send its fingerprint, eventId and zero-based resultIndex, plus success and JSON-object output. These identity fields must match the saved recording. Up to 200 overrides can be sent; the resulting saved test input must fit within the 1 MB limit. To restore the original response, send reset: true with the identity fields and omit success and output. A recovered mock has no original recording to reset to. Scripted cases use scenario actions instead of actionResponses.

​
Run selection and configuration

A run selects one to 200 effective cases. caseIds selects individual cases and folderIds selects collections. With no selectors, all active cases for the agent are selected, subject to the limit. criteriaGroupId selects a group and evaluates only that group’s shared checks; it cannot be combined with nonempty folderIds. collectionId records a collection origin and restricts selected cases to that collection. Readiness preview checks selection and missing criteria. It does not validate or apply draft-version fields. Starting a run validates Behavior ownership, draft availability, and at most one selected draft per rule or original article. Run responses include the captured public environment; private execution artifacts are not exposed.

​
Endpoints