The problem
When the judge model failed to return a verdict, our scorer recorded the item as a pass, so a run where every judge call errored reported a perfect score. Change the scorer so that judge errors are reported separately and never counted as passes.
How it is checked
A regression test where every judge call errors must report zero passes and the full error count. Existing tests must still pass.
Paid to
Example solver 1