Comparing Two Eval Runs by Their Average Pass Rate Is the Wrong Test

TL;DR. You run version A and version B against the same 500-item eval set. A passes 71.4 percent, B...

Read Original

Related