Two prompts on one eval set are paired data. The two-proportion SE said 'collect more'; McNemar said it was already decided.
Your A/B eval is paired. Your stat test probably isn't.
Related
Stop Using Generic AI Review. Build Your Own Board of Experts.
You finish a piece of work. You ask AI to review it. It says "looks good." You publish. It wasn't...
We benchmarked an AI agent on 52 broken clusters: kubectl vs a Kubernetes MCP server
Same agent, same faults, two ways to see the cluster. The one with the resource graph and change timeline used 76% fewer tool calls and finished in half the time.
How to Test an AI Agent's Tool Selection Without Trusting Its Own Logs
You have built an AI agent harness. It calls tools, routes requests, and returns results. Your team...