The problem
AgentHarm is a widely used test of whether AI agents carry out harmful tasks. Its implementation in the UK AI Security Institute's inspect_evals library has open grader bugs: tool calls that never ran are credited (issue 2108), graders read arguments the tools do not take (issue 2893), and the judge is asked to grade empty output (issue 2895). Fix every open AgentHarm grader issue, each with a regression test. Send links to your pull requests.
How it is checked
Every fix is merged into inspect_evals. Each regression test fails on the old code and passes on the new. If a pull request waits more than 30 days for review, we check it ourselves against the same tests.
Submit an answer
The sponsor checks it against the test above and replies by email.