Judging Agent Runs

Direct Probabilities versus Reasoned Verdicts

AI agents increasingly carry out multi-step tasks on their own, from navigating websites to using software tools. Every benchmark run of such an agent ends with the same question: did the agent succeed at the task?

For my master's seminar at THA, I'm looking at how to answer this question reliably and at low cost. Hand-written rules miss many real successes. LLM judges that reason through the whole run perform better, but they are still often wrong and become expensive when benchmarks repeat each task thousands of times.

Jev takes a different approach. Released by TypeSafe in September 2026 and named after Kahneman's fast, intuitive "System One", it neither reasons nor generates text. It returns a probability directly. This makes it fast and inexpensive, but it is an open question whether such a model can judge long, multi-step agent runs reliably.

As a test case, I'm evaluating this on 1,302 published agent runs from web benchmarks, labelled by experts, comparing Jev with GPT-6 Luna in three modes: reasoning before a verdict, a single "Yes"/"No" token read as a probability, and OpenAI's new Decisions API (in public beta since October 2026).

Alone versus in a Cascade

I'm particularly interested in the cascade setup: the fast judge decides the cases it is confident about and passes only the uncertain ones to the slower reasoning judge. This only pays off if the two judges make different mistakes. Earlier studies disagree on whether they do; this study adds a new data point for web agent runs.

Results will be published after evaluation.