The problem
Many safety evaluations use a language model as the judge. A 2026 study of a real evaluation harness found borderline items changing between pass and fail across identical runs, even at temperature 0. Build a judging setup that gives the same verdict every time without becoming less accurate. Send your code.
How it is checked
We run your judge 20 times on 500 borderline items that have human labels. It counts if fewer than 1% of items change verdict between runs and agreement with the human labels is at least as high as the single-run baseline.
Submit an answer
The sponsor checks it against the test above and replies by email.