Project Lemma.

The problem

When the judge model failed to return a verdict, our scorer recorded the item as a pass, so a run where every judge call errored reported a perfect score. Change the scorer so that judge errors are reported separately and never counted as passes.

How it is checked

A regression test where every judge call errors must report zero passes and the full error count. Existing tests must still pass.

Paid to

Example solver 1