Project Lemma.

The problem

Our safety evaluation uses a language model as the judge. Find items where the same judge, with the same settings, gives different pass or fail verdicts across repeated runs.

How it is checked

We re-run the judge 20 times on each item you send. It counts if the verdict changes at least once. Paid per item, up to ten.

Paid to

Example solver 1, Example solver 2, Example solver 3