Skip to main content
Test Suite turns conversations into repeatable regression checks. Each test case captures a conversation, its test setup, and the criteria that define a passing result. Fini replays the customer’s messages and generates new agent replies, then evaluates those replies against your criteria. Select the agent you want to test, then open Test Suite. The workspace has Test cases, Criteria groups, and Runs tabs. Use All tests or a collection to narrow the workspace.
Test Suite uses recorded or mock responses for external Actions. It does not offer live Action execution. Running a test with draft Behavior, rules, or articles does not publish those drafts.

​
Create a test case

1

Choose a conversation

Click Create test case, then Select conversation. Use Paste a link to load an Inbox conversation and preview it before selecting it, or click Browse Inbox with filters to find a conversation for the selected agent. If you need a new example, use Create new conversation to chat with the agent in Inbox, then click Use this conversation to return to the test case.You can also start from the flask icon in Inbox: select one eligible conversation and click Select.
2

Define what should pass

Name the case and open Criteria. Use Add criterion for case-specific checks and Link group for shared criteria groups, or combine both. Criteria are optional when saving a new case, but at least one must be present before it can run.Add criterion opens a dialog with three tabs: Suggested checks for reusable criteria, Exact checks for deterministic checks, and Custom expectation for an AI judgement of a written expectation.In Suggested checks, search and select a check, review its preview, then click Add criterion. In Exact checks, use What do you want to check? to choose an Action, rule, article, reply, handoff, or Advanced condition. Mark checks Required to pass when they should determine the case verdict; this option also applies to suggested checks.
3

Review the test setup

Open Test setup to inspect recorded Action responses and conversation attributes. Override responses or attributes when you want to test a different outcome. These changes belong to the test case, not the source conversation.
4

Save the case

Click Create test case. The saved case has Conversation, Criteria, Test setup, and Run history tabs. Save later edits with Save changes before running.
Customer messages come from the captured conversation. Test Suite does not generate a new customer scenario from a CSV, knowledge article, or judge prompt. To test another customer path, create or select another conversation.
Creating a new conversation in Inbox may call live integrations. Test Suite’s recorded/mock Action behavior applies to test runs, not to the Inbox conversation you create as an input.

​
Criteria and shared groups

Use an AI judgement for an outcome that needs interpretation, such as whether the reply explains a refund policy without inventing an exception. Use exact checks for observable events, such as whether the agent called a particular Action or handed off to a human. All required checks must pass for the case to pass. Non-required checks still appear in the results but do not turn the case verdict into a failure. Criteria groups share checks across linked cases. A case can belong to multiple groups and also have its own criteria. Editing a group’s criteria changes what its linked cases use on their next run; saved run results do not change. Review the affected cases before confirming a group update. A run started from a criteria group evaluates that group’s criteria only, using each member’s individual conversation setup. A normal case run includes its case-specific and linked group criteria. Removing a case from a group keeps the case, its setup, other group links, and run history. If no criteria remain, add a criterion or link another group before running it again.

​
Organize cases into collections

Collections organize cases and criteria groups for a particular feature or release. Select a collection to see its cases, groups, and run history, then use Run collection to test that scope. Removing a case from a collection or deleting the collection does not delete the case. Use Duplicate test case from a saved case’s menu to try a different setup. The copy’s setup and case-specific criteria are independent; linked group criteria remain shared.

​
Delete test cases

Use Delete test case from a case’s menu, or select cases in the workspace and click Delete selected. Review the confirmation before deleting.
Deletion is permanent. It removes the case, its setup, and its links to all collections and criteria groups. This is different from removing a case from one collection or group.
Original Inbox conversations and past run results are kept. Queued or running tests finish using their saved setup. A historical result marks a deleted case as deleted; you cannot save a missing Action mock back to that case. If a bulk deletion only partly succeeds, the dialog keeps the failed cases selected for retry. Retrying deletes only those remaining cases.

​
Run tests without publishing

Use Run test case for one case, Run all from the workspace, or Run collection for a collection. Runs support up to 200 cases; select a smaller batch if the workspace exceeds that limit. The run dialog checks that cases have criteria. If some are incomplete, add criteria first or explicitly choose Run only the ready test cases using the count shown in the dialog. If none are ready, the run cannot start. Under Behaviour, use Latest published behaviour or a saved version. You can also select available rule and article drafts. Selected drafts replace their published versions for this run; new drafts are added for the run. Nothing is published. For recorded Inbox conversations with linked agent replies, the run preserves those reply boundaries instead of generating a reply after every customer message. Consecutive customer messages remain context, and an unanswered final message still gets a reply. External Actions use recorded or mock responses. If a required response is missing, the case reports an execution error instead of calling the live service. This differs from the separate Replay workflow, which has its own execution modes.

​
Read run results

Open Runs, or a case’s Run history, then click the run name to open its dedicated results page. Copy link gives you a URL to share with teammates who have workspace access. Distinguish Failed, where a required criterion did not pass, from Execution error, where the run could not complete normally. Expand a result to inspect criterion-level reasoning, the generated conversation, and Action responses. Compare the recorded run configuration before attributing a changed verdict to an agent regression: the criteria, setup, or selected draft versions may also have changed. For a completed result marked Failed, start with What happened and Main issue. A suggested cause is labeled Suggested · Not confirmed: review its evidence before changing the agent. Needs investigation means the analysis could not establish a supported cause; it does not change the original verdict. Open More details to read the explanation and Criteria results. Failed results keep this section collapsed initially. Under Supporting evidence, message and Action links jump to the corresponding saved reply or Action result when available. Other references identify the supporting context or version. Open saved execution trace opens the replay in Inbox when that link is available. Older saved runs may show Not analysed and retain their original check results without a failure analysis. If analysis is unavailable for a new failed result, the check verdict is still preserved. AI judgement separates customer-facing replies from private context such as Action results, attributes, and internal notes. When available, it uses the historical article version referenced by the run, not today’s article text. A disclosure check should identify content in a customer-facing reply, not merely a value present in private context. For a missing Action response, click Add mock response in the execution error. Choose Success or Failure, enter the Mock response JSON, and click Save mock response. The response is saved to this case’s Test setup → Action responses; nothing is sent to the external service. Use Rerun test case to try again.

​
Troubleshooting

Add at least one case-specific criterion or link a criteria group. Finish saving or canceling pending edits, then reopen the run dialog.
Select a conversation belonging to the current agent with customer messages to replay. The Inbox picker excludes Replay and Test Suite conversations, as well as conversations without a non-empty customer message. Other eligibility checks still apply when selecting a conversation. Create a new example in Inbox if needed.
Inspect the requested inputs in the execution error, save an appropriate mock response, and rerun. The suite does not fall back to a live external Action call.
Reopen the run configuration and choose an available version. Review the warning before rerunning; a deleted or unavailable draft cannot be reused automatically.
Read the failed criterion’s reasoning. Refine the expectation if it tests the wrong behavior, or fix the agent’s prompt, knowledge, or rule if the failure is valid. Then run the case again.