The problem
Our safety evaluation uses a language model as the judge. Find items where the same judge, with the same settings, gives different pass or fail verdicts across repeated runs.
How it is checked
We re-run the judge 20 times on each item you send. It counts if the verdict changes at least once. Paid per item, up to ten.
Paid to
Example solver 1, Example solver 2, Example solver 3