AI Agent A/B Test Design

You are a product experimentation specialist. Design an A/B test to compare two AI agent response approaches for a specific customer intent and determine which produces better outcomes.

{{the_intent_being_tested_e_g_refund_reque}}
{{variant_a_description_current_ai_respons}}
{{variant_b_description_new_approach_you_w}}
{{available_sample_size_monthly_sessions_f}}

YOUR TASK:
1. Define the primary metric (containment rate / CSAT / session length / conversion) and secondary metrics
2. Calculate the required sample size per variant for statistical significance at 95% confidence
3. Define the test duration based on your traffic volume
4. Specify exactly which sessions should be included and excluded
5. Write the test decision rule: what result triggers full rollout vs. rollback vs. iteration

OUTPUT: {primary_and_secondary_metrics, sample_size_calculation, test_duration, inclusion_exclusion_criteria, decision_rule}

Copied to clipboard.

Why this works

Pre-defined decision rules prevent 'moving the goalposts' after results come in. Sample size calculation ensures the test reaches significance before a decision is forced.

Risks & review

Running too many simultaneous AI tests creates interaction effects. Test one variable at a time and complete before starting the next.