A/B tests on real conversations
A/B tests on real conversations
Try a different prompt, first message, model or voice on part of your real conversations, then keep the better one.
- Open the agent and go to A/B test.
- Choose what variant B changes: the prompt, first message, AI model or voice.
- Choose how many new conversations use B, then start the test. A caller stays on the same side for the whole conversation.
- Compare A and B side by side, including how often conversations were resolved, how often callers said it helped, and credits used.
- Click Use B from now on to keep B, or Stop, keep A to end the test.
Wait for at least 30 conversations on each side before deciding: with fewer, the differences are mostly chance. Switch on Sentiment and topic and Ask if the call helped in Behaviour for the quality numbers.
Only one test runs per agent at a time. To check a change before real callers hear it, use Tests and Compare.